Missing Proofs for some Commits

[botnumberseven]

I'm looking into OTP mainnet onchain data and something does not look right. Specifically, I'm trying to see if every submitCommit transaction has a corresponding submitProof transaction. Of course not all of them should have it, but there are cases which i can't explain. I can see ~6581k submitCommit transactions and out of them: 3152k commits do not have associated proofs yet, as the epoch is not over yet 2737k commits have corresponding proofs 582k commits will not have proofs, since they were pushed out of top3 110k commits do not have proofs, while they should

This 110k group is what caught my attention. These are cases when Commit was submitted, it's in top3, but there is no associated Proof for it, while it should be there (it's been more than 90 days since Commit was sent). Out of 110k I can exclude ~43k txs, as looks like these nodes are not active anymore, or they had noticeable issues in the past.

Now i have 67k commits, ~38k of them are from TL nodes and the rest ~29k are from Community nodes. So this is not about specific nodes, the majority of nodes are in this bucket. And this is how these 657k Commits are grouped by date, date here is submitCommit date. So it looks like this issue (no following Proof for a Commit) is more of a thing of the past, since there are not that many cases after April. And some small number cases is expected to be there, since we do not 100% uptime for 100% of nodes. So I don't think any actions are needed here, but I wanted to highlight this example of system misbehavior.