Daniel Harrison

I'm not surprised. At the end of the day yesterday investigating https://github.com/MaterializeInc/materialize/issues/13861, I thought I saw neither storaged (with 1 pubnub source) nor computed (with 1 materialized view on that...

> Is it possible that most of the work that remains to be done here falls under compaction? This is accurate. The short answer: I think (2) has always been...

fwiw, I view #12734 as separate but related. one step is having "bounded memory usage" at all, which (like I said, untested but) I think we already have for (2)....

cc @benesch you mentioned in our 1:1 that this seemed okay to do. is there anyone on cloud we should give a heads up to, given that this may cause...

Some initial questions: - Can we confirm what version they were running? This used to be exposed as part of a prometheus "mz_server_metadata_seconds" counter but this seems to be missing...

More questions: - what is the since of each shard? - what is the seqno and seqno capability of each shard (more interestingly, what time was each of those created)?

cc @aljoscha @pH14 nothing to do here for y'all, just fyi of my debuggings

Nikhil mentioned that this issue seemed to occur both for prod and for staging. I can confirm via staging prometheus metrics that gc is happening and that blob deletions calls...

Oh that all starts very recently. If I go back 2 days, then of the 3 clusters I seem to see in metrics, only one seems to have existed over...

nikhil and I poked around at the devex cluster (`01d730c7-*-0`) and saw at least one shard that seemed to have a bunch of equally sized blobs all written on the...