Integrate observer for System Observability

[gabizou]

Starts integrating some of the metrics that we can easily add to aid in observability and debugging.

• [ ] Histogram for Timing tick updates
• [ ] Counter for Missed Transactions (PhaseTracker specific)
• [ ] Ideas??

Supersedes #3296

@pschichtel let me know what you think of this

[pschichtel]

Looks good as far as I can tell. At cube island I toyed with some per-chunk metrics (e.g. entities per chunk per world) in order to find hotspots in a world, but I guess these would still better be added by a plugin since these values can easily be observed through the API.

[gabizou]

per-chunk metrics

Can definitely add per-world gauges by labels, I was considering some counter based things with how many entities, how many block entities, etc. are being done. Would be an interesting metric to observe the number of updates vs the tps. The hard part about per-chunk metrics is: What does the gauge look like? Is it one gauge with labels for chunk coordinates? The cardinality of the coordinates is rather high normally, so it's somewhat difficult to consider prometheus would be happy with having that level of cardinality, compared to say the variations of counts per world. I'll admit it is far easier for sponge to add them than a plugin (since the mixins are fairly simple to add).

[pschichtel]

cardinality of the coordinates

that's the point I was also unsure about. If users use a dedicated prometheus instance for the server (and possibly push aggregated metrics up to their central one) then I think it would be fine. prometheus can handle millions of series without trouble from what I read. Haven't actually tested it myself with that amount of data though.

We could:

• provide the high "resolution metrics" behind an option (or as a plugin) for those that know they can handle it
• always provide them and expect admins to configure drop rules in prometheus if they can't handle them

[ImMorpheus]

Hey! You seem to be running an unsupported version (API 8) of SpongeAPI. Support for API 8 ended in October 2023.

API-8 is now end-of-life. We will not be issuing any more API updates for version 8 of our API^1

As part of our legacy cleanup we are closing issues relating to API8/1.16

---

If you wish to move this forward, please rebase this PR on the current branch and reopen it.