Controvercial reachability check results

[blacks1ne]

BlossomSub reachability checks are giving controversial results. Here's an example for a standalone node behind NAT without proper port forwarding implemented for worker 3. Thus, the reachibility check should be negative. What we see is positive, negative, positive, negative and this cycle seems to continue.

{"level":"info","ts":1762326422.754412,"logger":"reachability","caller":"p2p/blossomsub.go:602","msg":"worker is externally reachable","process":"worker 3","process":"worker 3","core_id":3}
{"level":"info","ts":1762326425.5577712,"caller":"app/data_worker_node.go:75","msg":"data worker node started","process":"worker 3","process":"worker 3","core_id":3}
{"level":"error","ts":1762326435.2096095,"logger":"reachability","caller":"p2p/blossomsub.go:607","msg":"worker is not externally reachable","process":"worker 3","process":"worker 3","core_id":3,"stacktrace":"source.quilibrium.com/quilibrium/monorepo/node/p2p.NewBlossomSub.func1\n\t/opt/ceremonyclient/node/p2p/blossomsub.go:607"}
{"level":"info","ts":1762326512.7541459,"logger":"reachability","caller":"p2p/blossomsub.go:602","msg":"worker is externally reachable","process":"worker 3","process":"worker 3","core_id":3}
...
{"level":"error","ts":1762326797.2767367,"logger":"reachability","caller":"p2p/blossomsub.go:607","msg":"worker is not externally reachable","process":"worker 3","process":"worker 3","core_id":3,"stacktrace":"source.quilibrium.com/quilibrium/monorepo/node/p2p.NewBlossomSub.func1\n\t/opt/ceremonyclient/node/p2p/blossomsub.go:607"}

The issue seems easily reproducible.

[blacks1ne]

@CassOnMars insights: For some insight on why it stopped working, reachability uses NAT tests to determine whether its observed address is NAT. it's a bit buggy, in that if NAT addresses are seen alongside public addresses, it sometimes picks the NAT address and thus fails reachability testing. With black hole detection, for some reason, this results in NAT addresses not reaching the reachability test. I’m not fully certain on the details of this otherwise I would have already knocked it out. But it's in the libp2p package