make stderr log filter dynamically configurable

[pH14]

It'd be very helpful to be able to toggle our log levels dynamically when debugging actively-running applications, so we can selectively dig deeper into app behavior without needing a deploy / to bounce a running pod with some filter override / introduce general log spam.

This PR follows @guswynn's work to make opentelemetry tracing dynamically toggleable, and allows a user to pass in a log filter at runtime through the internal HTTP server. The cost of this flexibility is the RwLock inside the reload layer. I'm not sure how big of a performance impact this will make, nor am I quite sure how to measure it.

I've verified this changes works as expected through local calls such as:

> curl -X PUT -d '{"targets": "mz_persist_client=debug"}' -H 'Content-Type: application/json' localhost:6878/api/stderr/config
> curl -X PUT -d '{"targets": "info"}' -H 'Content-Type: application/json' localhost:6878/api/stderr/config

It'd also be helpful to make our tracing filter similarly configurable, but I think that deserves a separate PR if we pursue it.

Motivation

• This PR adds a feature that has not yet been specified.

Tips for reviewer

Checklist

• [ ] This PR has adequate test coverage / QA involvement has been duly considered.

• [ ] This PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way) and therefore is tagged with a T-protobuf label.

• [ ] This PR includes the following user-facing behavior changes:

[benesch]

Huge +1 on this, @pH14! We've been saying we should do this for years. Thanks for actually doing it!

[guswynn]

@pH14 other than my suggestion to make the otel endpoint work the same (which can be done in a later pr if you want!), this still looks good to merge!

[pH14]

other than my suggestion to make the otel endpoint work the same (which can be done in a later pr if you want!), this still looks good to merge!

Sounds good! I'd love to have this change in the next release, so I think I'm going to merge this as-is (assuming it all builds), and then start on a separate PR to work through your suggestions of updating the otel endpoint to have the same behavior / use the same types

[guswynn]

@pH14 sounds good! I retried the failing test but dont know if you like squash or merge or rebase so ill let you do that!