cloudfoundry
noble switch from iptables to nftables
as in noble it is easier to create proper firewall rules in nftables see https://github.com/cloudfoundry/bosh-linux-stemcell-builder/commit/e1245f41aa5db4acc6d7e3bc81acab0ebdd8509f
i'm also wondering why the bosh-agent feels responsible for setting up iptables for monit. https://github.com/cloudfoundry/bosh-agent/blob/main/platform/net/firewall_provider_linux.go#L77-L101 as this was already done for example in the jammy stemcell https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-jammy/master/stemcell_builder/stages/bosh_monit/assets/restrict-monit-api-access
i think the agent should not be responsible for this.
for setting up the nats rules we can use nftables
in https://github.com/cloudfoundry/bosh-agent/blob/main/platform/net/firewall_provider_linux.go#L115-L130
example" nft add rule inet filter output skuid 0 ip daddr 10.0.0.1 tcp dport 4222 log prefix "Matched skuid director nats rule:" accept
@nouseforaname investigated https://pkg.go.dev/github.com/google/[email protected] but this seems way to complicated for what we are actually trying to achieve for example adding the rule above would look something like
conn.AddRule(&nftables.Rule{
Table: table,
Chain: chain,
Position: 0,
Handle: 0,
Flags: 0,
Exprs: []expr.Any{
&expr.Meta{Key: 10, SourceRegister: false, Register: 1},
&expr.Cmp{Op: 0, Register: 1, Data: []byte{0, 0, 0, 0}},
&expr.Meta{Key: 15, SourceRegister: false, Register: 1},
// skuid 0
&expr.Cmp{Op: 0, Register: 1, Data: []byte{2}},
&expr.Payload{OperationType: 0, DestRegister: 1, SourceRegister: 0, Base: 1, Offset: 16, Len: 4, CsumType: 0, CsumOffset: 0, CsumFlags: 0},
// ip daddr 10.0.0.1
&expr.Cmp{Op: 0, Register: 1, Data: []byte{10, 0, 0, 1}},
&expr.Meta{Key: 16, SourceRegister: false, Register: 1},
// dport
&expr.Cmp{Op: 0, Register: 1, Data: []byte{6}},
&expr.Payload{OperationType: 0, DestRegister: 1, SourceRegister: 0, Base: 2, Offset: 2, Len: 2, CsumType: 0, CsumOffset: 0, CsumFlags: 0},
// port 5000 is caluclated
// 5000 / 256 = 19 * 256 = 4864
// 500 - 4864 = 136
&expr.Cmp{Op: 0, Register: 1, Data: []byte{19, 136}},
// log prefix "Mathced skuid director nats rule:"
&expr.Log{Level: 4, Flags: 0, Key: 36, Snaplen: 0, Group: 0, QThreshold: 0, Data: []byte("Mathced skuid director nats rule:")},
// accept
&expr.Verdict{
Kind: expr.VerdictAccept,
},
},
more investigation is probably needed but do not that the iptable rules do not work anymore in noble as we are using cgroupv2 so action is needed
i'm also wondering why the bosh-agent feels responsible for setting up iptables for monit. https://github.com/cloudfoundry/bosh-agent/blob/main/platform/net/firewall_provider_linux.go#L77-L101
it doesn't it sets up a rule for itself to be able to access monit. But with nftables that wouldn't even be needed anymore, since the statement we ended up using is not relying on cgroups, but rather the uid that is associated with the request socket. Going this way also removes the need to have the monit access helper because being root is enough to not be blocked.
As far as the nftables package goes, it was promising until trying to add a rule. The readability of the Expression structs is horrible. While it works, it seems that having this done via ex/exec is more readable ( even though checking for existing rules is going to end up being a bit if stdout parsing.)
As far as testing goes, it would also be easier to test expected os/exec commands with binmock ( or another thing )..
While there is a way to setup a TestDialer with the package, checking expectations involves matching bytearrays 💥: https://github.com/google/nftables/blob/main/nftables_test.go#L234-L260
I think all of this could be solved by a unit that is baked into the stemcell that waits for /var/vcap/bosh/agent_settings.json and then sets up nftable rules via a script ( after parsing out mbus IP:PORT))
