OpenVPN
Connection delay after openvpn server stop and start
We are seeing connection delay of around 15 minutes between the server and the client when we stop the server and start it again. The server and client are both AIX machines. The openvpn used is 2.6.10 version. The client gets the IP immediately after the server starts, but the pings are working only after around 15 minutes. Any inputs are really helpful.
server.conf ######## port 443 proto tcp dev tap
ca /openvpn-keys/ca.crt cert /openvpn-keys/server.crt dh /openvpn-keys/dh2048.pem
server 10.8.0.0 255.255.255.0 ifconfig-pool-persist ipp.txt client-to-client keepalive 10 120 cipher AES-256-CBC status openvpn-status.log verb 3 explicit-exit-notify 1
######## client.conf ######## client dev tap proto tcp
remote X.X.X.X 443 resolv-retry infinite
nobind
ca /openvpn-keys/ca.crt cert /openvpn-keys/client1.crt key /openvpn-keys/client1.key remote-cert-tls server tls-auth /openvpn-keys/ta.key 1 cipher AES-256-CBC verb 3
Enabled the verbosity to level 9 , and I see the below ( I xxxx'ed the hostname and Ip address in the below log) .
2025-02-13 04:19:38 us=875832 XXXXX/XXXXX:63778 GET INST BY VIRT: 00:bd:28:9c:bb:08@0 [failed] 2025-02-13 04:19:38 us=875839 STREAM: SET NEXT, buf=[136,0] next=[136,1800] len=-1 maxlen=1800 2025-02-13 04:19:38 us=875844 MULTI TCP: multi_tcp_post TA_SOCKET_READ -> TA_TUN_WRITE 2025-02-13 04:19:38 us=875850 MULTI TCP: multi_tcp_action a=TA_TUN_WRITE p=1 2025-02-13 04:19:38 us=875855 MULTI TCP: multi_tcp_wait_lite a=TA_TUN_WRITE mi=0x110077f50 2025-02-13 04:19:38 us=875861 PO_CTL rwflags=0x0000 ev=9 arg=0x110003870 2025-02-13 04:19:38 us=875866 PO_CTL rwflags=0x0002 ev=7 arg=0x1100013a4 2025-02-13 04:19:38 us=875873 I/O WAIT Tr|TW|Sr|Sw [1/0] 2025-02-13 04:19:38 us=875884 PO_WAIT[1,0] fd=7 rev=0x00000002 rwflags=0x0002 arg=0x1100013a4 2025-02-13 04:19:38 us=875890 event_wait returned 1 2025-02-13 04:19:38 us=875895 I/O WAIT status=0x0008 2025-02-13 04:19:38 us=875900 MULTI TCP: multi_tcp_dispatch a=TA_TUN_WRITE mi=0x110077f50 2025-02-13 04:19:38 us=875907 XXXXX/XXXXX:63778 TUN WRITE [98] 2025-02-13 04:19:38 us=875915 XXXXX/XXXXX:63778 write to TUN/TAP returned 98 2025-02-13 04:19:38 us=875921 STREAM: SET NEXT, buf=[136,0] next=[136,1800] len=-1 maxlen=1800 2025-02-13 04:19:38 us=875927 MULTI TCP: multi_tcp_post TA_TUN_WRITE -> TA_UNDEF
verb 9 was not asked for and is not helpful either - it includes way too much information which is not relevant when diagnosing problems that are not related to internal key handling etc.
A verb 3 logfile from client and server, showing the initial connection and the reconnect after server restart would be more useful.
That said, I guess it's likely that you are hitting an arp issue - given that AIX only supports TAP mode, TAP presents itself as a virtual ethernet, and ethernet has ARP tables for mapping IP to Ethernet addresses, which are cached(!)... - when the client reconnects both sides might assign itself a new (random) MAC address, which might need to be aged-out on the other side.
So, if you can reproduce the issue, please try if using arp -d $ip on both sides (clearing the ARP entry for the IP address of the other end) will help. arp -an will show you the tables, and netstat -in should show the MAC address currently assigned to the TAP interface.
On non AIX systems, one could fix the TAP interface MAC using lladdr m:a:c:a:dd:rr in the OpenVPN config, but this has not (yet) been implemented for AIX. So you could script the arp -d call using an --up script in your config... ugly? yes...
There is a "tun mode" emulation for AIX in the works, but that has been stalled because lack of time (and nobody seems to be using OpenVPN on AIX)
@cron2 Thanks for the detailed explanation. I was not getting any logs in the server when pinging from the client side ( when there was connection issue ), hence increased the verbosity to 9.
You are right. It's indeed a arp cache problem. Running arp -d $ip fixed the issue.
You mentioned that the "tun mode" emulation in AIX is stalled. We would like to understand more about that and how we can help here ( like how much more work is required and what technical skills are required). We are from AIX Toolbox team. We provide lot of open source softwares to AIX users as rpm packages through this platform (openvpn is also one of them) --> https://www.ibm.com/support/pages/aix-toolbox-open-source-software-downloads-alpha
Hi. Glad to hear that it was indeed an ARP problem and there is a manual solution, at least.
Since you mentioned you're from the Toolbox team - maybe you have better documentation that I have :-) - I'd like to implement lladdr support in OpenVPN, so we could set the TAP adapter's MAC address to a configured value, avoiding the ARP cache problem. On other OSes, there is usually "something ifconfig" to set the hardware / ether address, but my AIX's "man ifconfig" does not mention anything. Is there a way?
On the TUN emulation layer - what I have works for IPv4, but IPv6 support is missing. Also it needs rebasing to the latest master, last time I worked on it was in early 2023... - so it mostly needs time. But since you voiced interest, I'll see if I can find a bit of time and polish it for inclusion in 2.7... (not promising anything). WRT "what skills are needed" - well, understanding IPv6 neighbour discovery, and understanding OpenVPN C code...
Thanks again @cron2 .
In AIX, chdev command can be used to set/change the hardware address. Something like "chdev -l entx -a alt_addr <new Mac>". https://www.ibm.com/docs/en/aix/7.2?topic=cards-adapter-management-configuration
Thanks for the link. I have tried it, but either I'm doing something wrong, or it is plainly not supported on tap interfaces...
m-gd@aix$ sudo /usr/sbin/chdev -l tap1 -a alt_addr=0X10005A4F1B7F <
Method error (/usr/lib/methods/chgif):
0514-018 The values specified for the following attributes
are not valid:
0821-228 chgif: Bad attribute(s) or attribute value(s): alt_addr
m-gd@aix$ sudo /usr/sbin/chdev -l tap1 -a use_alt_addr=yes
Method error (/usr/lib/methods/chgif):
0514-018 The values specified for the following attributes
are not valid:
0821-228 chgif: Bad attribute(s) or attribute value(s): use_alt_addr
this is on a 7.3 machine, and the 7.3 documentation still explains this (but only for ethernet and token ring) :-(
alt_addr is part of the underlying adapter interface , in this case tapent1.
lsdev should list all the devices and lsattr -El <dev> will list the attributes of the device.
Okay, so lsattr -El tapent1 indeed shows these attributes
m-gd@aix$ lsattr -El tapent1
alt_addr 0x000000000000 Alternate EtherChannel Address True
jumbo_frames no Enable Gigabit Ethernet Jumbo Frames True
use_alt_addr no Enable Alternate EtherChannel Address True
it won't permit me to actually use them, though...
m-gd@aix$ sudo /usr/sbin/chdev -l tapent1 -a alt_addr=0X10005A4F1B7F
Method error (/usr/lib/methods/chgent):
0514-062 Cannot perform the requested function because the
specified device is busy.
(this is with a tap1 interface which was created with ifconfig tap1 create and not configured further. UP or ifconfig tap1 down does not make a difference, it's always busy...
Do you have a working and complete example of creating and changing the MAC address for a given tap interface? I can then integrate that into openvpn lladdr.c - but at this point, I do not really see how this is supposed to work...
@cron2 Here is a working example.,
# ifconfig tap0 create
tap0
# netstat -I tap0
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
tap0 1500 link#3 0.bd.33.8a.57.0 0 0 0 0 0
# ifconfig tap0 down detach
# chdev -l tapent0 -a use_alt_addr=yes -a alt_addr=0X10005A4F1B7F
tapent0 changed
# ifconfig tap0 up
# netstat -I tap0
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
tap0 1500 link#3 10.0.5a.4f.1b.7f 0 0 0 0 0
Another way is to assign mac address during the creation of the adapter interface itself.,
# mkdev -c adapter -s pseudo -t tapent -a use_alt_addr=yes -a alt_addr=0X10005A4F1B7F
tapent1 Available
# ifconfig tap create
tap1
# netstat -I tap1
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
tap1 1500 link#4 10.0.5a.4f.1b.7f 0 0 0 0 0