Filtering doesn't seem to work

[akasprzok]

Summary:

I can't get metrics filtering to work. My goal is to filter on multiple different user metadata labels, but right now I can't get a single one to work.

Version: v0.12.0

Config (abridged, in a k8s deployment manifest):

 spec:
      containers:
      - args:
        - --monitoring.metrics-type-prefixes=cloudsql.googleapis.com/database
        - --monitoring.filters='cloudsql.googleapis.com/database:metadata.user_labels."my-label"="my-value"'

Given that config I'd expect the filter to be included in the debug log here but that doesn't seem to be the case. The log statements generally only include the metric.type filter:

ts=2023-01-24T06:45:40.496Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"cloudsql.googleapis.com/database/postgresql/insights/aggregate/execution_time\""

and the metrics scraped include metrics from databases not tagged with my-label=my-value.

I'm new to GCP, and assume this is just something silly stemming from my lack of familiarity with Stackdriver. Would love any pointers.

[kgeckhart]

That's very odd I can't reproduce this locally on v0.12.0 or on v0.13.0. I'm running,

make build && ./stackdriver_exporter --google.project-id=<redacted> --monitoring.metrics-type-prefixes='cloudsql.googleapis.com/database' --log.level=debug --monitoring.filters='cloudsql.googleapis.com/database:metadata.user_labels."my-label"="my-value"'

and all of the logged metrics filters look like

ts=2023-01-31T17:48:11.711Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"cloudsql.googleapis.com/database/replication/network_lag\" AND (metadata.user_labels.\"my-label\"=\"my-value\")"

You posted an abridged config, are you by chance specifying a --monitoring.filters with multiple values that are comma delimited? If so, that's not supported ATM (it's supported with -monitoring.metrics-type-prefixes) but you can specify multiple --monitoring.filters params which will all be used. I was planning to take a pass at the docs + code to make the two params consistent in how they behave.

[akasprzok]

Yeah I'm specifying multiple --monitoring.filters.

It drove me crazy for a while, but I never found the problem.

For me it's however no longer an issue, we decided to look for a different solution due to lack of caching and no support for exporting user labels.

[kgeckhart]

For me it's however no longer an issue, we decided to look for a different solution due to lack of caching and no support for exporting user labels.

I completely understand the labels aspect. Just wondering, what type of caching would be useful?