Understanding InfiniBand to PCI Impedance

Published on Thursday, 11 March 2010 06:27
Written by Douglas Eadline
Hits: 6201

From the why didn't I think of that department

Recently, Richard Walsh put together a table with both PCI and InfiniBand (IB) specifications. He published the spreadsheet on the Beowulf Mailing List. Cluster Monkey contacted Richard and asked permission to reproduce the table in HTML so that it was more accessible. We broke it in half resulting in two tables so it would fit on the page, but the information is the same.

While the information is not new, collecting it in these tables was a great idea. The issue is the potential "bandwidth impedance" mismatch between and IB HCA's (Host Channel Adapters or cards) and the PCI bus. (Note: In this sense the impedance mismatch is not considered electrical, but bandwidth.) As an example, the current fastest usable IB speed is QDR (Quad Data Rate, 4.0 GigaBytes/sec) this must be matched with PCI 2.0 (also called Gen2) 8x interface (4.0 GigaBytes/sec) for maximum performance. Most new QDR IB cards support up to 8x PCI Gen2 speeds and will auto-negotiate to lower speeds if the PCI bus is slower resulting in lower performance.

The following table is the current and future PCI Express transfer modes. The list of abbreviations and terms used in both tables are as follows (Giga=10^9).

PCI Express
  # Lanes Transfer Rate Raw
Data Rate
(Per Direction)
Effective
Data Rate
(Per Direction)
Bandwidth
Data Rate
(Per Direction)
1.0 Gen1 20% drop
(8B/10B loss)
 
1x 1 2.5 GT/sec 2.5 Gbits/sec 2.0 Gbits/sec 250 Mbytes/sec
4x 4 2.5 GT/sec 10.0 Gbits/sec 8.0 Gbits/sec 1.0 Gbytes/sec
8x 8 2.5 GT/sec 20.0 Gbits/sec 16.0 Gbits/sec 2.0 Gbytes/sec
16x 16 2.5 GT/sec 40.0 Gbits/sec 32.0 Gbits/sec 4.0 Gbytes/sec
32x 32 2.5 GT/sec 80.0 Gbits/sec 64.0 Gbits/sec 8.0 Gbytes/sec
2.0 Gen2 20% drop
(8B/10B loss)
 
1x 1 5.0 GT/sec 5.0 Gbits/sec 4.0 Gbits/sec 500 Mbytes/sec
4x 4 5.0 GT/sec 20.0 Gbits/sec 16.0 Gbits/sec 2.0 Gbytes/sec
8x 8 5.0 GT/sec 40.0 Gbits/sec 32.0 Gbits/sec 4.0 Gbytes/sec
16x 16 5.0 GT/sec 80.0 Gbits/sec 64.0 Gbits/sec 8.0 Gbytes/sec
32x 32 5.0 GT/sec 160.0 Gbits/sec 128.0 Gbits/sec 16.0 Gbytes/sec
3.0 Gen3 1.5% drop
(128B/130B loss)
 
1x 1 8.0 GT/sec 8.0 Gbits/sec 8.0 Gbits/sec 1.0 Gbytes/sec
4x 4 8.0 GT/sec 32.0 Gbits/sec 32.0 Gbits/sec 4.0 Gbytes/sec
8x 8 8.0 GT/sec 64.0 Gbits/sec 64.0 Gbits/sec 8.0 Gbytes/sec
16x 16 8.0 GT/sec 128.0 Gbits/sec 128.0 Gbits/sec 16.0 Gbytes/sec
32x 32 8.0 GT/sec 256.0 Gbits/sec 256.0 Gbits/sec 32.0 Gbytes/sec

The following table is the current and future InfiniBand transfer modes.

InfiniBand
  # Wires # Pairs LVDS
Signal Rate
Raw
Data Rate
(Per Direction)
Effective Bandwidth
Data Rate
(Per Direction)
SDR (Single Data Rate) 20% drop
(8B/10B loss)
 
1X 4 2 2.5 GHz 2.5 Gbits/sec 2.0 Gbits/sec 250 Mbytes/sec
4X 16 8 2.5 GHz 10.0 Gbits/sec 8.0 Gbits/sec 1.0 Gbytes/sec
12x 48 24 2.5 GHz 30.0 Gbits/sec 24.0 Gbits/sec 3.0 Gbytes/sec
DDR (Double Data Rate) 20% drop
(8B/10B loss)
 
1X 4 2 5.0 GHz 5.0 Gbits/sec 4.0 Gbits/sec 500 Mbytes/sec
4X 16 8 5.0 GHz 20.0 Gbits/sec 16.0 Gbits/sec 2.0 Gbytes/sec
12x 48 24 5.0 GHz 60.0 Gbits/sec 48.0 Gbits/sec 6.0 Gbytes/sec
QDR (Quad Data Rate) 20% drop
(8B/10B loss)
 
1X 4 2 10.0 GHz 10.0 Gbits/sec 8.0 Gbits/sec 1.0 Gbytes/sec
4X 16 8 10.0 GHz 40.0 Gbits/sec 32.0 Gbits/sec 4.0 Gbytes/sec
12x 48 24 10.0 GHz 120.0 Gbits/sec 96.0 Gbits/sec 12.0 Gbytes/sec
EDR (Eight Data Rate) 3% drop
(64B/66B loss)
 
1X 4 2 25.78 GHz 25.78 Gbits/sec 25.0 Gbits/sec 3.125 Gbytes/sec
4X 16 8 25.78 GHz 103.12 Gbits/sec 100.0 Gbits/sec 12.50 Gbytes/sec
12x 48 24 25.78 GHz 309.36 Gbits/sec 300.0 Gbits/sec 37.50 Gbytes/sec

Be sure to double check all the specifications for your IB HCAs and servers to make sure you have a good match.

Unfortunately you have Javascript disabled, please enable Javascript in order to experience the comments correctly