InfiniBand High Performance Multi-Purpose Network Architecture

Super computer cluster
Super computer cluster. Ohio Supercomputer Center / Wikimedia Commons / CC BY-SA 3.0

InfiniBand is a high-performance, multi-purpose network architecture based on a switch design often called a "switched fabric." InfiniBand ("IB" for short) was designed for use in I/O networks such as storage area networks (SAN) or in cluster networks. It has become a leading standard in high-performance computing. Over 200 of the world's fastest 500 supercomputers use InfiniBand, more than are using Gigabit Ethernet.

History of InfiniBand

Work on InfiniBand began in the 1990s under different names by two separate industry groups designing technical standards for system interconnects. After the two groups merged in 1999, "InfiniBand" eventually emerged as the name of the new architecture. Version 1.0 of the InfiniBand Architecture standard was published in 2000.

How InfiniBand Works

Specifications for InfiniBand Architecture span layers 1 through 4 of the OSI model. It covers physical and data-link layer hardware requirements and also features connection-oriented and connectionless transport protocols analogous to TCP and UDP. InfiniBand uses IPv6 for addressing at the network layer.

InfinBand implements a messaging service for applications called Channel I/O that bypasses network operating systems in order to achieve high performance in specialized environments. It provides the ability for two Infiniband-enabled applications to create a direct communication channel having send and receive queues called Queue Pairs. The queues map to memory spaces accessible to each application for data sharing (called Remote Direct Memory Access or RDMA).

An InfiniBand network consists of four primary components:

  • Gateway
  • Host Channel Adapters
  • Subnet Managers
  • Switches

Like other network gateways, an InfiniBand Gateway interfaces an IB network to outside local networks.

Host Channel Adapters connect InfiniBand devices to the IB fabric, like more traditional kinds of network adapters.

Subnet Manager software manages traffic flow on an InfiniBand network. Each IB device runs a Subnet Manager Agent to communicate with the central Manager.

InfiniBand Switches are a required element of the network, to enable a collection of devices to pair up with each other in different combinations. Unlike Ethernet and Wi-Fi, IB networks typically do not use routers.

How Fast is InfiniBand?

InfiniBand supports multi-gigabit network speeds, up to 56 Gbps and higher depending on its configuration. The technology roadmap includes support for 100 Gbps and faster speeds in future versions.

Limitations of InfiniBand

Applications of InfiniBand have largely been limited to cluster supercomputers and other specialized network systems. Marketing claims aside, InfiniBand was not designed for general-purpose application data networking in a way that could replace either Ethernet or Fibre Channel in internet datacenters. It does not use traditional network protocol stacks like TCP/IP because of the performance limitations of these protocols, but in so doing does not support mainstream applications. 

It hasn't yet become a mainstream technology in part because standard network software libraries like WinSock cannot be made to work with InfiniBand without sacrificing the performance benefits of the architecture.