retransmission in iperf3

I'm seeing this in iperf3 (yes i know i should be using nttcp)

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.03 sec 830 MBytes 695 Mbits/sec 2305 sender
[ 5] 0.00-10.00 sec 828 MBytes 695 Mbits/sec receiver
[ 7] 0.00-10.03 sec 825 MBytes 690 Mbits/sec 2849 sender
[ 7] 0.00-10.00 sec 823 MBytes 690 Mbits/sec receiver
[ 9] 0.00-10.03 sec 587 MBytes 491 Mbits/sec 3340 sender
[ 9] 0.00-10.00 sec 586 MBytes 492 Mbits/sec receiver
[ 11] 0.00-10.03 sec 687 MBytes 574 Mbits/sec 3043 sender
[ 11] 0.00-10.00 sec 684 MBytes 573 Mbits/sec receiver
[ 13] 0.00-10.03 sec 761 MBytes 636 Mbits/sec 2602 sender
[ 13] 0.00-10.00 sec 758 MBytes 636 Mbits/sec receiver
[ 15] 0.00-10.03 sec 779 MBytes 652 Mbits/sec 2797 sender
[ 15] 0.00-10.00 sec 778 MBytes 652 Mbits/sec receiver
[ 17] 0.00-10.03 sec 1008 MBytes 843 Mbits/sec 2252 sender
[ 17] 0.00-10.00 sec 1006 MBytes 844 Mbits/sec receiver
[ 19] 0.00-10.03 sec 1.12 GBytes 957 Mbits/sec 2078 sender
[ 19] 0.00-10.00 sec 1.11 GBytes 956 Mbits/sec receiver
[ 21] 0.00-10.03 sec 866 MBytes 725 Mbits/sec 2368 sender
[ 21] 0.00-10.00 sec 865 MBytes 725 Mbits/sec receiver
[SUM] 0.00-10.03 sec 7.31 GBytes 6.26 Gbits/sec 23634 sender
[SUM] 0.00-10.00 sec 7.29 GBytes 6.26 Gbits/sec receiver

the cables are lc type, i know i may have accidentality touched the tip on the edge of the transceiver 🙈😂.

is it time for a cleaning pen ?

L3 tests don't really determine an L1 problem either way.

With managed switches we prefer to use SFP / SFP+ modules that can report light RX levels to the switch
(DDM and optionally the more advanced DOM).

I would not worry unless you can identify some measurable effect on the optical characteristics.

Beyond a clean, dry, lint-free cloth you don't generally need special tools or solvents in domestic applications if you are mainly concerned with minimising dust and oily residues.

I take it that as far as you know, your fibre run is not anywhere close to the upper distance limit of the rating?

Edited by prlzx (Wed 20-Mar-24 22:04:36)

Retransmissions are normal. Packet loss is how TCP senses that the link is full (congestion) and reduces its sending rate to compensate.

Otherwise the TCP window would continue to grow and grow, and buffers in routers would become fuller and fuller. It's when those buffers become full and are forced to drop packets, that TCP congestion control kicks in to reduce the sending rate.

https://en.wikipedia.org/wiki/TCP_congestion_control

Now, retransmissions *can* also be caused by transmission errors (like the dirty fibre connectors you mention). In that case, TCP will also respond to them as if the network were congested, and reduce the rate. However in your case, you seem to be achieving a pretty good rate of over 6Gbps, using what appears to be a multi-threaded test.

If you want to test for transmission errors, then you should use a single-threaded test. Then there's a chance that transmission errors will dominate over buffer-overflow errors.

TCP throughput in the face of transmission errors is also very sensitive to the round-trip time, with the formula given here: https://en.wikipedia.org/wiki/TCP_tuning#Packet_loss

Taking one of your concurrent TCP streams: 83MB/sec is about 57,000 packets per second at 1500 byte. 230 packets/sec lost implies 0.4% packet loss.

A single TCP stream with 1ms RTT (like over a LAN) and 0.4% packet loss is limited to:

1460/0.001/sqrt(0.004) = 23MB/sec = 185Mbps. But in fact you are getting much more than that from each stream.

If the test endpoint were 10ms RTT away, and there were 0.4% packet loss, each TCP stream would be limited to

1460/0.01/sqrt(0.004) = 2.3MB/sec = 18.5Mbps

In short: I don't yet see any evidence of packet loss other than what would be expected from filling a link to capacity. However if you do a single-threaded test and specify the RTT, it will be much easier to determine.

Thank you candlerb, (well both of you) for the replies. I think moving from rj45/copper and 1gbe ethernet brought up a whole new world were i'd rather ask questions.

the linux test was done with a usb ssd and that may have caused issues. I'm not seeing windows to windows direct at 4gbits ..... Struggling to get beyond that in w10.

This is on a 10G LAN presumably?

At this point, performance of the operating system and drivers becomes a big issue, and indeed, a USB-attached NIC can also be a limiting factor. If your NIC drivers can't keep up with the incoming traffic rate then they'll drop packets too, slowing the transmission down to a speed they can cope with. And cheapo USB NICs may only be USB 3.0 @ 5Gbps.

If you're doing stuff with Linux then 10G with PCI or Thunderbolt NICs is likely to work out-of-the-box. Trying to get Windows work well with 10G may be more painful.

If you boot both ends from Ubuntu Live USB sticks and run an iperf3 between those, you will see what the hardware is capable of when given a decent operating system and drivers. Then you can see whether the USB NIC is at fault, or the software.

You'll have to think about how much you *need* 10G performance, and whether you want to pay for upgrades to PCI / Thunderbolt, or even move to Linux or macOS.

Edited by candlerb (Fri 22-Mar-24 15:27:29)

In reply to a post by candlerb:

This is on a 10G LAN presumably?

yes. atm pcie to pcie - so that i can test before going further down the bunny hole.

In reply to a post by candlerb:

At this point, performance of the operating system and drivers becomes a big issue, and indeed, a USB-attached NIC can also be a limiting factor. If your NIC drivers can't keep up with theincoming traffic rate then they'll drop packets too, slowing the transmission down to a speed they can cope with. And cheapo USB NICs may only be USB 3.0 @ 5Gbps.

sorry for the confusion, the usb part was the ssd.... the sfp+ nics are pcie.

In reply to a post by candlerb:

If you're doing stuff with Linux then 10G with PCI or Thunderbolt NICs is likely to work out-of-the-box. Trying to get Windows work well with 10G may be more painful.

yup i've noticed the pain in windows.

In reply to a post by candlerb:

If you boot both ends from Ubuntu Live USB sticks and run an iperf3 between those, you will see what the hardware is capable of when given a decent operating system and drivers. Then you can see whether the USB NIC is at fault, or the software.

yup and this is where i was seeing the retransmission, from a usb ssd booted linux install with iperf3 sending from the linux box to the windows box. The linux test was to make sure the hardware was ok as i've seen 1.6gbits initially. As it was a pure hardware test to check the sfp+ modules and cards were ok. Especially more so because one wouldn't play ball in w10 - in a bizarre issue.

In reply to a post by candlerb:

You'll have to think about how much you *need* 10G performance, and whether you want to pay for upgrades to PCI / Thunderbolt, or even move to Linux or macOS.

the use case is mainly for large cache files for 3d sims - which can exceed 6GB per file per frame. If you want to do network rendering, then those rather big files need to be moved around the lan.

Because i'm using standard am4/5 mobos with 2 gpus in there, becoming pcie bound is a creative issue to resolve as well.

so its a case of test and test again.

Edited by Taras (Fri 22-Mar-24 13:51:41)

In reply to a post by Taras:

yup and this is where i was seeing the retransmission, from a usb ssd booted linux install with iperf3 sending from the linux box to the windows box.

You want to try Linux both ends too. Otherwise it could well be that Linux can send faster than Windows can receive - which will show itself as transmission errors when Windows is dropping packets. (But as I say, that's a normal response, and TCP adjusts itself accordingly)

In reply to a post by Taras:

the use case is mainly for large cache files for 3d sims - which can exceed 6GB per file per frame.

If you're building a local storage network, then I'd definitely set MTU 9000 at both ends - as long as your switch(es) can also be configured to carry jumbo frames. This reduces the number of packets which need to be processed by a factor of 6, and makes a big difference to CPU and interrupt load on both sender and receiver.

In reply to a post by Taras:

is it time for a cleaning pen ?

The error counters and power levels on the optical transceivers would seem a better place to look for data on physical layer problems than an iPerf.

In reply to a post by XGS_Is_On:

In reply to a post by Taras:

is it time for a cleaning pen ?

The error counters and power levels on the optical transceivers would seem a better place to look for data on physical layer problems than an iPerf.

I've only had sfp modules for a month and a bit, and its a learning curve - noob incidents are allowed 😂🙈

In reply to a post by candlerb:

In reply to a post by Taras:

the use case is mainly for large cache files for 3d sims - which can exceed 6GB per file per frame.

If you're building a local storage network, then I'd definitely set MTU 9000 at both ends - as long as your switch(es) can also be configured to carry jumbo frames. This reduces the number of packets which need to be processed by a factor of 6, and makes a big difference to CPU and interrupt load on both sender and receiver.

https://ibb.co/7X1qv2J

Thats only 2gb, easily done 4 to 6gb data files for each frame, they have to be used in renders, if i want to do a network render you can see why i've started to explore this route. Its something i've put off for an age.

moving large files from windows file server, to windows client has its issues, i've not hit beyond 600MB/s. With Linux i've now finally hit 7.5Gbits plus (without tuning).I've built basic linux file servers before but thats for multi machine access and not about speed.

The stupid thing is that atm i'd don't need currently need this type of speed regularly. But i do now, and that means rethinking how i was going do file storage.