Hydra doesn't like Grafana Cloud OTLP endpoint

Open Moosemorals opened this issue 2 years ago • 0 comments

Preflight checklist

[X] I could not find a solution in the existing issues, docs, nor discussions.
[X] I agree to follow this project's Code of Conduct.
[X] I have read and am following this repository's Contribution Guidelines.
[ ] I have joined the Ory Community Slack.
[ ] I am signed up to the Ory Security Patch Newsletter.

Ory Network Project

No response

Describe the bug

When I set tracing.providers.otlp.server_url to the URL given by Grafana cloud (https://otlp-gateway-prod-gb-south-0.grafana.net/otlp), Hydra logs show "The configuration contains values or keys which are invalid" and three regexes that look like they should match an ipv4 address and host, ipv6 address and host, or domain name and host.

Digging into the code SetupOTLP in otelx/otlp.go, you're calling otlptracehttp.WithEndpoint, which does only accept a host/address and port number, but that should also be called with WithURLpath since not all providers (e.g., Grafana Cloud) use the standard path.

What I'd really like you to do is not call any of the With* methods at all, and let otlptracehttp pick up it's config from the environment, in line with other otlp collectors.

Reproducing the bug

Configure hydra with a url under tracing.providers.otlp.server_url

tracing:
  providers:
    otlp:
      server_url: otlp-gateway-prod-gb-south-0.grafana.net/otlp

Relevant log output

Jan 12 23:29:15 example.com hydra-server[56720]: The configuration contains values or keys which are invalid:
Jan 12 23:29:15 example.com hydra-server[56720]: tracing: map[provider:otel providers:map[otlp:map[server_url:https://otlp-gateway-prod-gb-south-0.grafana.net/otlp]]]
Jan 12 23:29:15 example.com hydra-server[56720]:          ^-- doesn't validate with "ory://tracing-config#"
Jan 12 23:29:15 example.com hydra-server[56720]:
Jan 12 23:29:15 example.com hydra-server[56720]: The configuration contains values or keys which are invalid:
Jan 12 23:29:15 example.com hydra-server[56720]: tracing.providers.otlp.server_url: https://otlp-gateway-prod-gb-south-0.grafana.net/otlp
Jan 12 23:29:15 example.com hydra-server[56720]:                                    ^-- anyOf failed
Jan 12 23:29:15 example.com hydra-server[56720]:
Jan 12 23:29:15 example.com hydra-server[56720]: The configuration contains values or keys which are invalid:
Jan 12 23:29:15 example.com hydra-server[56720]: tracing.providers.otlp.server_url: https://otlp-gateway-prod-gb-south-0.grafana.net/otlp
Jan 12 23:29:15 example.com hydra-server[56720]:                                    ^-- does not match pattern "^\\[(([0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]))]:([0-9]*)$"
Jan 12 23:29:15 example.com hydra-server[56720]:
Jan 12 23:29:15 example.com hydra-server[56720]: The configuration contains values or keys which are invalid:
Jan 12 23:29:15 example.com hydra-server[56720]: tracing.providers.otlp.server_url: https://otlp-gateway-prod-gb-south-0.grafana.net/otlp
Jan 12 23:29:15 example.com hydra-server[56720]:                                    ^-- does not match pattern "^([0-9]{1,3}\\.){3}[0-9]{1,3}:([0-9]*)$"
Jan 12 23:29:15 example.com hydra-server[56720]:
Jan 12 23:29:15 example.com hydra-server[56720]: The configuration contains values or keys which are invalid:
Jan 12 23:29:15 example.com hydra-server[56720]: tracing.providers.otlp.server_url: https://otlp-gateway-prod-gb-south-0.grafana.net/otlp
Jan 12 23:29:15 example.com hydra-server[56720]:                                    ^-- does not match pattern "^(([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\\-]*[a-zA-Z0-9])\\.)*([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\\-]*[A-Za-z0-9]):([0-9]*)$"
Jan 12 23:29:15 example.com hydra-server[56720]:
Jan 12 23:29:15 example.com hydra-server[56720]: {"audience":"application","event":"&jsonschema.ValidationError{Message:\"doesn't validate with \\\"ory://tracing-config#\\\"\", InstancePtr:\"#/tracing\", SchemaURL:\"https://github.com/ory/hydra/spec/config.json\", SchemaPtr:\"#/properties/tracing/$ref\", Context:jsonschema.ValidationErrorContext(nil), Causes:[]*jsonschema.ValidationError{(*jsonschema.ValidationError)(0xc00095a000)}}","level":"error","msg":"The changed configuration is invalid and could not be loaded. Rolling back to the last working configuration revision. Please address the validation errors before restarting the process.","service_name":"Ory Hydra","service_version":"v2.2.0-rc.3","time":"2024-01-12T23:29:15.154613904Z"}

Relevant configuration

tracing:
  providers:
    otlp:
      server_url: otlp-gateway-prod-gb-south-0.grafana.net/otlp

Version

2.2.0-rc.3

On which operating system are you observing this issue?

Linux

In which environment are you deploying?

Other

Additional Context

I'm running the container under podman on a linux host. Otherwise my install is fine (it's great!).

Jan 12 '24 23:01 Moosemorals