While fibre can move more data over longer distances, we've used MetaWatch to show that direct attach copper cables have the edge over both SMF and MMF fibre (which were essentially equivalent to each other).
Our testing setup looks a little bit like this:
We connected two machines to a MetaMux 32 device running MetaWatch using 10G Ethernet. We then created a loopback through two ports on the device's front panel, first using direct-attach copper cables, and then using fibre cables. For each cable that we tested, we passed around 1,000,000 "ping" packets through the device, pinging backward and forward between the two test servers. By measuring the time that each packet passed through the MetaWatch device, and then correlating those measurements and differencing them, we were able to calculate the time taken for each packet to pass through the device. Each result is pretty darn accurate: +/- 2 ns, but we averaged the million results we had to get even more accurate numbers.
We used mellanox direct-attach copper cables, and Finisar SFPs for the purpose. You can see the part numbers for these in the table below. We used generic MMF and SMF fibres -- it's possible that some will be faster than others (including hollow-core fibres, which should be *much* faster).
Note also that these are passive direct-attach cables, not Cat 5 1GBase-T or 10GBase-T cables with the corresponding SFPs -- we've measured these types of SFPs to have a very high latency of round 300 ns round-trip, which dwarfs any potential savings from the propagation delay of the copper.
Here are the raw results:
|Cable Type||SFPs (1 and 2)||Cable||Distance (m)||Latency (ns)||Description|
|1.008 m direct-attach copper|
|DAC||N/A||MC3309130-002||2.02||12.3344||2.02 m direct-attach copper|
|DAC||N/A||MC3309130-003||2.998||16.7289||2.998 m direct-attach copper|
|DAC||N/A||MC3309124-007||6.992||34.9989||6.992 m direct-attach copper|
|MMF||FTLX8571D3BCV||1 m MMF OM1 fibre||1.094||
|1.094 m MMF fibre|
|MMF||FTLX8571D3BCV||2 m MMF OM1 fibre||2.190||14.9769||2.190 m MMF fibre|
|MMF||FTLX8571D3BCV||3 m MMF OM1 fibre||3.286||20.4918||3.286 m MMF fibre|
|MMF||FTLX8571D3BCV||4 m MMF OM1 fibre||4.381||25.7473||4.381 m MMF fibre|
|MMF||FTLX8571D3BCV||5 m MMF OM1 fibre||5.477||31.2473||5.477 m MMF fibre|
|MMF||FTLX8571D3BCV||6 m MMF OM1 fibre||6.572||36.7727||6.572 m MMF fibre|
|MMF||FTLX8571D3BCV||7 m MMF OM1 fibre||7.667||42.2407||7.667 m MMF fibre|
|SMF||FTLX1471D3BCV||2 m SMF Fibre||2.100||14.4991||2.1 m SMF fibre|
|SMF||FTLX1471D3BCV||4m SMF Fibre||4.200||24.7130||4.2 m SMF fibre|
|SMF||FTLX1471D3BCV||6m SMF Fibre||6.300||35.2940||6.3 m SMF fibre|
For each test we took 1,000,000 samples. To check that our test setup was accurate, we looked at the distribution of the timestamps captured by MetaWatch and see that it is well formed, with a spread of +/- 4 ns.
|Measured Latency (ns)||Sample %|
So, if we plot those results and fit a trendline, we get slopes and offsets:
|Cable Length (m)||SMF Latency (ns)||MMF Latency (ns)||Twinax Latency (ns)|
The trendlines above show that single-mode and multi-mode fibre have near identical latencies of 4.96 ns per metre. This is close to the oft-quoted 5 ns per metre for fibre. The latency for the twinax copper cables shown is 4.60 ns per metre -- faster by about 400 ps per metre. What's also interesting is the zero-offset. When we extrapolate down to a hypothetical 0 m cable, the copper cables have a lower fixed offset compared with the fibre. We consider the copper direct-attach cable to have zero latnecy in the SFPs, since the twinax cable is soldered directly to the pins on the SFP module itself. The difference between the offsets is about 1 ns (copper shows an offset of 3 ns, which we believe is a mis-calibration in the MetaWatch device). This is due to the latency in the SFPs themselves. Note that the latency is nearly identical for both the FTLX8571 and FTLX1471 SFPs: around 500 ps.
It's important to use the right medium for the job. Direct-attach copper cables have a maximum reach of around 7-10 m, depending on the devices being used, compared with a 300 m range for MMF fibre used above, and 10 km range for the SMF -- three orders of magnitude difference. This really only applies within a small footprint.
Direct attach copper cables have the edge over both SMF and MMF fibre when it comes to latency, and this is even more prominent for short cable runs because of the latency introduced by the SFP modules which drive the fibre.
If you really care about latency, (twinax) copper is faster than fibre.