additional scrape configs not working with my configuration

[dasari-sai]

Dear All,

I've been struggling with additional scrape config from past few weeks. I've deployed prometheus-operator in our kubernetes cluster using kustomize from bundle.yaml file. I've added the additional scrape config in the prometheus-additional file and created a secret like mentioned in the document and mentioned the entries in the deployment files. But it is not picking it up. These are my manifests. Can anyone please help me ?

Thanks.

prometheus-additional.yaml:

`- job_name: "grafana_metrics"
  scrape_interval: 15s
  scrape_timeout: 5s
  static_configs:
    - targets: ["grafana-service.grafana-operator.svc.cluster.local:3000"]`

additional-scrape-configs:

apiVersion: v1
data:
  prometheus-additional.yaml: 
<key>
kind: Secret
metadata:
  name: additional-scrape-configs
  namespace: prometheus-operator
type: Opaque

prometheus-deploy.yaml:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"
  name: prometheus
  labels:
    prometheus: prometheus
spec:
  serviceAccountName: prometheus
  serviceMonitorSelector: {}
  serviceMonitorNamespaceSelector:
    matchLabels:
      prometheus-scrape: "true"
  podMonitorSelector: {}
  podMonitorNamespaceSelector:
    matchLabels:
      prometheus-scrape: "true"
      control-plane: "controller-manager"
  resources:
    requests:
      memory: 400Mi
  enableAdminAPI: false
  additionalScrapeConfigs:
    name: additional-scrape-configs
    key: prometheus-additional.yaml
```
`

[slashpai]

Did you check prometheus-operator logs?

[DewaldDeJager]

It looks like your secret is in the prometheus-operator namespace and the Prometheus deployment is in the default namespace. Since secrets are namespaced, the secret won't be found and therefore the static config won't be applied.

[dasari-sai]

I've checked the logs nothing is reflecting regarding anything about the additional scrape configs. @DewaldDeJager Let me try it out and update. Thank you for the reply.

[dasari-sai]

I tried to setup for prometheus-operator namespace itself, Just gave it a default job to monitor prometheus instance. But still it's not working or reflecting.

[dasari-sai]

If possible could you please share with me on how you apply to a default namespace.

[AndrStnz]

If possible could you please share with me on how you apply to a default namespace.

you specify namespace: prometheus-operator in your additional-scrape-configs.yaml but you dont do that in your prometheus-deploy.yaml

So add namespace: prometheus-operator to prometheus-deploy.yaml and your secret and your deployment are in the same namespace.

[dasari-sai]

I am doing this deployment using argocd . It doesn't require to specify as the namespace will be specified in kustomization.yaml file

[beasync]

Hi all,

I've found the same problem @dasari-sai describes (prometheus-operator v0.59.2). Operator and secret in the same namespace.

The operator log was as follows:

neither serviceMonitorSelector nor podMonitorSelector, nor probeSelector specified. Custom configuration is deprecated, use additionalScrapeConfigs instead

Adding an unneeded serviceMonitorSelector the operator no longer returns the console warning and theadditionalScrapeConfigs are loaded.

This does not work:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  namespace: monitoring
spec:
  image: quay.io/prometheus/prometheus:v2.38.0
  serviceAccountName: prometheus
  version: v2.38.0
  additionalScrapeConfigs:
    name: additional-scrape-configs
    key: prometheus-additional.yaml

This works:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  namespace: monitoring
spec:
  image: quay.io/prometheus/prometheus:v2.38.0
  serviceAccountName: prometheus
  version: v2.38.0
  serviceMonitorSelector:
    matchLabels:
      foo: bar
  additionalScrapeConfigs:
    name: additional-scrape-configs
    key: prometheus-additional.yaml

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the last 60 days. Thank you for your contributions.

[github-actions[bot]]

This issue was closed because it has not had any activity in the last 120 days. Please reopen if you feel this is still valid.

[bodji]

I still have the same problem.

The fix proposed by @beasync is working.

[tomasbedrich]

Why is this closed? The issue is still valid.

[prm-2024]

I am also trying to add additional scrape configurations in the prometheus.yaml file, but prometheus-additional.yaml is not being picked up from the structure below. Can anyone guide me, please?

additionalScrapeConfigs: name: additional-scrape-configs key: prometheus-additional.yaml

[slashpai]

Hello 👋🏻

Did you check https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/additional-scrape-config.md?

Also we do have ScrapeConfig CRD now (still in alpha) which lets you specify config Kubernetes native way see https://prometheus-operator.dev/docs/user-guides/scrapeconfig/