opnsense
OFED kernel modules - whether to enable or revert all OFED-related
Hi again.... Last time for now, I promise.
After setting up a new OPNsense device, turns out that OFEDs kernel modules that manage Infiniband aren't included in the current kernel. Somehow I must have added them to my old device, and thought they were native.
After getting a kernel build to work correctly, it seems that a options flag for OFED needs to be enabled in the kernel build options (which I assume is SMP, but I don't know for sure). OFED as an option and ipoib as a device is enough to make everything 'just work'.
So the question becomes if it's worth enabling this to be built in kernel, or if it's better to just revert all these changes since they were based on my misguided assumption that OFED kernel support was already enabled.
I can create a PR either way. Obviously, I'd prefer support be added, but there's no point doing so if the workload for a pretty niche product is much for the team.
If it’s in GENERIC we can consider it, but I thought we do include that kernel config automatically. That leaves the question of is it really included by default in FreeBSD or not.
Well that's what I'm wondering too now.
I know OFED, the userland at least, is enabled upstream according to the FreeBSD wiki. It's also there in a fresh install of 13.2. The opensm component wasn't, which we resolved.
What I'm not sure on is how the kernel modules work. I'm new to BSD development, so I'm still learning, so please bear with me.
Obviously the modules don't need to be included in the kernel, but just there so they can be enabled via a tunable (which is how it's recommended to do it).
https://wiki.freebsd.org/InfiniBand is where I'm basing a lot of my information on.
My build machine (FreeBSD 13.2) allows me to kldload ibcore, one of the modules in question. It appears to be located in /boot/kernel/ibcore.ko on said machine. I did check GENERIC from upstream and cannot find OFED, but if I understand correctly, adding it as an option statically links it to the kernel? Which is obviously not what I want to do.
I think it comes down to a kernel option "MK_OFED", which, when not explicitly set to no, enables OFED modules to be built as per https://cgit.freebsd.org/src/commit/?id=fe0f6950391cf10bbfcd1672d8e111a41063a4c8 where upstream included OFED as default for both kernel and userland support. The oddity is, I can't find MK_OFED being set to no anywhere in OPNsense config files.
The kernel modules are there judging by the wiki entry so I'll let everything as it is and hope you can report back for feedback on 24.7.
Cheers, Franci
In the end, I just spun up a Linux VM with the Mellanox OFED modules on a shared internal hyper-v network to opnsense as its own subnet, passed the IB NIC through to Linux, and once I set up a subnet manager and configured the packet forwarding, it all works fine. This is just what I get for playing with stupid technology in my homelab stack.
It seems I had all sorts of legacy ofed stuff on the working OPNsense setup I had from last time I had tried to configure it. After virtualising and importing config it all broke.
The problem is all the hardware that gets used for IB at this level is EOL, so support is non-existent and tools are even broken. The effort to make it work is too much versus another small VM on my stack.
I may have a look again in future, but it's resolved for now.
On Fri, 19 July 2024, 4:48 pm Franco Fichtner, @.***> wrote:
The kernel modules are there judging by the wiki entry so I'll let everything as it is and hope you can report back for feedback on 24.7.
Cheers, Franci
— Reply to this email directly, view it on GitHub https://github.com/opnsense/tools/issues/409#issuecomment-2238480172, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACSLB4ZP3O67VEYSAEG3Z53ZNCZETAVCNFSM6AAAAABHR6NYZGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZYGQ4DAMJXGI . You are receiving this because you authored the thread.Message ID: @.***>
Yeah go ahead, revert any changes we made
On Fri, 19 July 2024, 11:06 pm Franco Fichtner, @.***> wrote:
@rileyg98 https://github.com/rileyg98 ok so you don't mind me backing out the src.conf switch again?
— Reply to this email directly, view it on GitHub https://github.com/opnsense/tools/issues/409#issuecomment-2239091590, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACSLB47IWUQ5Z6KWOTHXH6TZNEFNZAVCNFSM6AAAAABHR6NYZGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZZGA4TCNJZGA . You are receiving this because you were mentioned.Message ID: @.***>