opnsense
Mellanox ConnectX-4 Lx CARP Multicast Not Working
Important notices
Before you add a new report, we ask you kindly to acknowledge the following:
- [X] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
- [X] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue
Describe the bug
We have two GoWin GW-BS-1UR2-25G: https://www.gowinfanless.com/products/network-device/1u-2u-server/gw-bs-1ur2-25g https://www.servethehome.com/gowin-1u-25gbe-appliance-review-this-has-everything-including-poe-intel-nvidia/
It seems there is a bug in the ConnectX-4 Lx driver for FreeBSD. Broadcasts are looped back to themselves so CARP won't work in multicast.
Symptoms: Both devices will stay in CARP Backup when using multicast and neither will become Master. Switching to unicast allows CARP to work.
Constant Duplicate Address Detected (DAD) message in system log:
2025-04-13T09:37:45-07:00 <hostname goes here) kernel - - [meta sequenceId="1585"] <3>mce0: a looped back NS message is detected during DAD for fe80:a::526b:4bff:fef0:8058. Another DAD probes are being sent.
<13>1 2025-04-13T09:37:47-07:00 <hostname goes here) kernel - - [meta sequenceId="1586"] <3>mce1: a looped back NS message is detected during DAD for fe80:b::526b:4bff:fef0:8059. Another DAD probes are being sent.
Links I found with problems: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=270859 (Disabling SRV-IO did not work for me) https://network.nvidia.com/pdf/prod_software/Mellanox_FreeBSD_Release_Notes_3.0.0.pdf
Are there any known workarounds for the broadcast problem with the Mellanox ConnectX-4?
To Reproduce
Steps to reproduce the behavior:
- Set up CARP using the Mellanox NIC using multicast
Expected behavior CARP to work using multicast
Describe alternatives you considered Switching to unicast.
Environment OPNsense 25.1.5_4-amd64 Intel i3-N305 CPU Network: Two Intel i226-V (Intel SRKTU) NICs, two Intel i210-AT 1GbE NICs, One i210-AT, and two SFP28 25Gbe ports powered by a (Mellanox) NVIDIA ConnectX-4 Lx.
