[BUG] Unable to filter data using the component label

[dotdc]

Describe the bug

Using parca-agent, we didn't managed to filter data based on the pod's component label.

We tried several approaches like you can see in the configuration examples below, but nothing worked.

Using __meta_kubernetes_pod_label_component:

relabel_configs:
  - source_labels: [ __meta_kubernetes_pod_label_component ]
    regex: platform-temporal.*
    action: keep
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [ __meta_kubernetes_pod_name ]
    action: replace
    target_label: kubernetes_pod_name

Using component:

relabel_configs:
  - source_labels: [ component ]
    regex: platform-temporal.*
    action: keep
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [ __meta_kubernetes_pod_name ]
    action: replace
    target_label: kubernetes_pod_name

Using harcoded values:

relabel_configs:
  - source_labels: [ component ]
    regex: platform-temporal-client
    action: keep
  - source_labels: [ component ]
    regex: platform-temporal-middleware-backend
    action: keep
  - source_labels: [ component ]
    regex: platform-temporal-middleware-features
    action: keep
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [ __meta_kubernetes_pod_name ]
    action: replace
    target_label: kubernetes_pod_name

Pod labels:

❯ k get po -l release=platform-temporal --show-labels=true
NAME                                                     READY   STATUS    RESTARTS   AGE   LABELS
platform-temporal-client-84bc4f8c67-mrhdq                2/2     Running   0          23m   component=platform-temporal-client,pod-template-hash=84bc4f8c67,release=platform-temporal
platform-temporal-middleware-backend-866889d6d-scqsb     2/2     Running   0          95m   component=platform-temporal-middleware-backend,pod-template-hash=866889d6d,release=platform-temporal
platform-temporal-middleware-features-7c775898c5-8szt6   2/2     Running   0          37m   component=platform-temporal-middleware-features,pod-template-hash=7c775898c5,release=platform-temporal

Each time, Parca-agent's claimed that No running processes matching configuration, which wasn't what we expected.

To Reproduce

Try to using our configuration to filter on the component label.

Works when using the release label.

Working configuration:

relabel_configs:
  - source_labels: [ release ]
    regex: platform-temporal
    action: keep
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [ __meta_kubernetes_pod_name ]
    action: replace
    target_label: kubernetes_pod_name

Expected behavior

I would have expected the parca-agent to be able to filter on the component label.

Screenshots or Profiles (using https://pprof.me)

N/A

Logs

N/A

Software (please complete the following information):

• Parca Agent Version: main-1694099744-4f8cb663
• Parca Server Version (if applicable): Polar Signals (12/09/2023)

Workload (please complete the following information):

• Runtime (if applicable): N/A
• Compiler (if applicable): N/A

Environment (please complete the following information):

• Linux Distribution: N/A
• Linux Version: N/A
• Arch: N/A
• Kubernetes Version (if applicable): v1.27.3-gke.100
• Container Runtime (if applicable): N/A

Additional context

N/A

[dotdc]

cc @brancz