Reproduceable - HTTP Previous segment not captured followed by TCP Dup Ack and TCP Out-Of-Order

[raszuk]

Noticed using zabbix over ZT that every graph or screen refresh is slower then native or IPSec ... Looked deeper and see in wireshark regular every refresh request bunch of TCP Dup Acks and TCP Out-of-Order followed each time by HTTP reporting "TCP Previous segment not captured".

My first guess was this was Win10 driver issue ... Zabbix is working on Ubuntu. So I tried to setup another ubuntu locally and test it as NAT router to eliminate any Windows - well the issue is even worse going via extra hop.

I also setup IPSec VPN between Win10 and Zabbix network and works clean ... zero errors.

I still like ZT hence trying to see if anyone else seen this or have some hints on the cause. Captured issue attached.

Thx !

[raszuk]

Anyone ? I tested this on a new ISP ... Identical issue.

[laduke]

Thanks for posting this. It's being looked at. No update yet.

[rhetr]

i saw similar logs running iperf3 through zerotier: https://discuss.zerotier.com/t/questions-about-bandwidth-and-ipv6/6335/2 . the logs i captured are between macos<->linux but zerotier introduces the same bandwidth issues between windows<->linux and linux<->linux.

[rhetr]

after some more research and testing I think this might be a linux-specific issue. I ran across this stackexchange question that also said hamachi has a similar issue on linux, which gave me the idea to test zerotier performance between mac<->windows.

surely enough, my iperf3 tests on LAN vs zerotier were pretty much identical, in fact for some reason zerotier seems to even outperform my LAN which is a bit funny.

[glimberg]

@rhetr thanks for digging in on that. Hadn't seen that Stack Exchange link before. Could definitely be an issue in Linux if it's also appearing on other things like Hamachi.

in fact for some reason zerotier seems to even outperform my LAN which is a bit funny.

ZeroTier compresses data before sending so yes, if you're sending easily compressible data and have enough CPU cycles to throw at it, you can definitely see speeds faster than your physical LAN speed

[rhetr]

ZeroTier compresses data before sending so yes, if you're sending easily compressible data and have enough CPU cycles to throw at it, you can definitely see speeds faster than your physical LAN speed

interesting!

I haven't found anything else yet but I was skimming through some articles on OpenVPN tuning, some asserted in regards to TCP tunneling:

Aside from the usual TCP tuning and the socket-flags TCP_NODELAY option, probably the best optimization is to get rid of TCP tunneling as a whole and use UDP instead.

that inspired me to do some UDP iperf3 tests:

mac->zt->linux (iperf3 -u -c linux.zt -b 2G)

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  7]   0.00-10.00  sec  2.33 GBytes  2.00 Gbits/sec  0.000 ms  0/1726472 (0%)  sender
[  7]   0.00-10.10  sec   506 MBytes   420 Mbits/sec  0.023 ms  1359617/1726294 (79%)  receiver

mac->linux (iperf3 -u -c linux.lan -b 2G)

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  7]   0.00-10.00  sec  2.32 GBytes  1.99 Gbits/sec  0.000 ms  0/1720195 (0%)  sender
[  7]   0.00-10.01  sec  1.11 GBytes   955 Mbits/sec  0.008 ms  892916/1717754 (52%)  receiver

linux->zt->mac(iperf3 -u -c linux.zt -b 2G -R)

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  7]   0.00-10.00  sec  1.21 GBytes  1.04 Gbits/sec  0.000 ms  0/898504 (0%)  sender
[  7]   0.00-10.00  sec   707 MBytes   593 Mbits/sec  0.029 ms  385358/897176 (43%)  receiver

linux->mac (iperf3 -u -c linux.lan -b 2G -R)

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  7]   0.00-10.00  sec  1.26 GBytes  1.08 Gbits/sec  0.000 ms  0/930724 (0%)  sender
[  7]   0.00-10.00  sec  1.11 GBytes   951 Mbits/sec  0.018 ms  99569/920417 (11%)  receiver

so zerotier on linux seems to be receiving UDP packets like a champ but is getting throttled sending them out. I will do some tests between mac and windows soon too for comparison.

update:

mac->zt->windows

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  7]   0.00-10.00  sec  2.33 GBytes  2.00 Gbits/sec  0.000 ms  0/905733 (0%)  sender
[  7]   0.00-10.00  sec  1017 MBytes   853 Mbits/sec  0.023 ms  519744/905719 (57%)  receiver

mac->windows

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  7]   0.00-10.00  sec  2.33 GBytes  2.00 Gbits/sec  0.000 ms  0/1711565 (0%)  sender
[  7]   0.00-10.00  sec   976 MBytes   819 Mbits/sec  0.013 ms  1007337/1708178 (59%)  receiver

windows->zt->mac

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  7]   0.00-10.00  sec  1.51 GBytes  1.30 Gbits/sec  0.000 ms  0/588391 (0%)  sender
[  7]   0.00-10.00  sec  1.03 GBytes   887 Mbits/sec  0.030 ms  186868/588391 (32%)  receiver

windows->mac

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  7]   0.00-10.00  sec   830 MBytes   697 Mbits/sec  0.000 ms  0/596447 (0%)  sender
[  7]   0.00-10.00  sec   827 MBytes   693 Mbits/sec  0.026 ms  2709/596447 (0.45%)  receiver

not really sure what to make of these results.