I have a HP dual 10Gb NIC (model number) and also a cisco SFP+ DAC 5m, both from eBay.
So they are actually CNA’s, useful to know. these are in the FCoE mode.
I tested these with a switch but as I only had one 10Gb nic so I went ahead and brought 2 more. Once they arrived I put them in the old xw4600 based workstation and gave them a go with iPerf. The one in the top x16 slot got 8.84 gb/s and in the x8 slot below that got 4.33 gb/s.
So why is this? Well firstly lets look at the hardware – the x8 slot is PCIe Gen1…and wired only x4. And then x16 slots are Gen2. So the theory says that the x16 slot should be able to run a 10Gbit nic just fine, in x8 mode.
The 8x slot is actually only x4 electrically, and only Gen1, which means a max of 8Gb/s.
|PCIe version||Year||Transfers||x1 speed||x4 speed||x8 speed||x16 speed|
Here’s the output showing the PCI link status’. Top card is running x8 at 5GT/s so Gen2, and bottom card is running x4 at 2.5GT/s so Gen1.
You might notice there are in the iSCSI mode rather than FCoE. The 10Gb nics are actually only Gen 2 anyway. (PCIe x8 Gen2.) See https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=c03572683
I can also tell you that in an iPerf loopback the Dell 7010 gets 19.8Gb/s, and the xw4600 gets 14.8Gb/s.
So i’m not 100% sure where the bottleneck is. In subsequent testing I managed to get a 9.22Gb/s rate which I think is probably the limit of transfer given the TCP overheads – i’ll try with Jumbo packets.
With an MTU of 9000 I managed to get bursts of 9.90Gb/s. While running htop on both the dell and the xw4600 I noticed that the puney dual core E6550 at 2.33GHz in the xw4600 was pegged at 100%, While the i3-2120 at 3.3GHz in the dell didn’t go above 60%. I’m pretty sure the bottleneck is in the CPU/PCIe workings, after all using a dual core from 2007 probably wasn’t designed to handle 10Gb networking.