Missing support for Unix Domain Socket configuration via `DD_TRACE_AGENT_URL`

[ivoanjo]

Spotted that we're missing support for specifying a UDS path via DD_TRACE_AGENT_URL, which other Datadog libraries support.

[milosivanovic]

@ivoanjo Is that the cause of this?

W, [2022-05-19T02:18:34.186530 #38]  WARN -- ddtrace: [ddtrace] Invalid URI scheme 'unix' for DD_TRACE_AGENT_URL environment variable ('unix:///var/run/datadog/apm.socket'). Ignoring the contents of DD_TRACE_AGENT_URL.

Not sure how it can be invalid when the docs https://ddtrace.readthedocs.io/en/stable/configuration.html show that unix:///var/run/datadog/apm.socket is actually the default value. The rails logger also outputs the following:

I, [2022-05-19T02:42:35.685422 #38]  INFO -- ddtrace: [ddtrace] DATADOG CONFIGURATION - {"date":"2022-05-19T02:42:35+00:00","os_name":"x86_64-pc-linux-gnu","version":"1.0.0","lang":"ruby","lang_version":"3.1.0","enabled":true,"service":"puma","agent_url":"http+unix:///var/run/datadog/apm.socket?timeout=1","debug":false
...

which is showing the value ending up being http+unix (due to the value of DD_TRACE_AGENT_URL being ignored), but I don't see http+unix:// in the docs anywhere, so it's a bit confusing.

[ivoanjo]

Hey @milosivanovic, thanks for reaching out!

You are correct about this issue being the source of your log message:

W, [2022-05-19T02:18:34.186530 #38] WARN -- ddtrace: [ddtrace] Invalid URI scheme 'unix' for DD_TRACE_AGENT_URL environment variable ('unix:///var/run/datadog/apm.socket'). Ignoring the contents of DD_TRACE_AGENT_URL.

What happens is that dd-trace-rb CAN actually use a unix domain socket, but right now you CAN NOT configure it via DD_TRACE_AGENT_URL.

So how to use a unix domain socket? There's two ways, actually:

1. If you specify no other configuration (no DD_AGENT_HOST/DD_TRACE_AGENT_PORT nor a DD_TRACE_AGENT_URL with an http url) and a unix domain socket exists in /var/run/datadog/apm.socket, then it gets used by default

2. Configuring a path via c.tracing.transport_options, as documented in https://docs.datadoghq.com/tracing/setup_overview/setup/ruby/#using-the-unix-domain-socket-uds-adapter

From your log...

I, [2022-05-19T02:42:35.685422 #38] INFO -- ddtrace: [ddtrace] DATADOG CONFIGURATION - {"date":"2022-05-19T02:42:35+00:00","os_name":"x86_64-pc-linux-gnu","version":"1.0.0","lang":"ruby","lang_version":"3.1.0","enabled":true,"service":"puma","agent_url":"http+unix:///var/run/datadog/apm.socket?timeout=1","debug":false

...I think you're in case 1) -- the DD_TRACE_AGENT_URL you have configured is ignored, and the default gets used; In this case, this matches what you had in DD_TRACE_AGENT_URL but just because you were passing in the same value as a default.

So TL;DR is that the warning is harmless in your case. It'll disappear once we fix this ticket, or you could just unset the DD_TRACE_AGENT_URL and dd-trace-rb will continue to use the default. Hope this clarifies what's up!

[jcogilvie]

Any word on fixing this? I think we're running into the same.

Invalid URI scheme 'unix' for DD_TRACE_AGENT_URL environment variable ('unix:///var/run/datadog/apm.socket'). Ignoring the contents of DD_TRACE_AGENT_URL.

When trying to use a domain socket, we're getting the above error, and additionally

DATADOG DIAGNOSTIC - Agent Error: Datadog::Transport::InternalErrorResponse ok?: unsupported?:, not_found?:, client_error?:, server_error?:, internal_error?:true, payload:, error_type:Errno::ECONNREFUSED error:Failed to open TCP connection to 127.0.0.1:8126 (Connection refused - connect(2) for 127.0.0.1:8126)

Here's my config:

DATADOG CONFIGURATION - {"date":"2022-12-06T21:14:12+00:00","os_name":"x86_64-pc-linux-gnu","version":"1.2.0","lang":"ruby","lang_version":"3.1.2","env":"myenv","enabled":true,"service":"my-service-api-v1","dd_version":"4c4a61fca23eccb2e7cfaf00c476bf5086b0b289","agent_url":"http://127.0.0.1:8126?timeout=30","agent_error":"Datadog::Transport::InternalErrorResponse ok?: unsupported?:, not_found?:, client_error?:, server_error?:, internal_error?:true, payload:, error_type:Errno::ECONNREFUSED error:Failed to open TCP connection to 127.0.0.1:8126 (Connection refused - connect(2) for 127.0.0.1:8126)","debug":false,"analytics_enabled":false,"tags":"env:ninja,version:4c4a61fca23eccb2e7cfaf00c476bf5086b0b289","runtime_metrics_enabled":false,"integrations_loaded":"[email protected],[email protected],[email protected],[email protected]","vm":"ruby-3.1.2","partial_flushing_enabled":true,"priority_sampling_enabled":false,"health_metrics_enabled":false,"integration_grpc_analytics_enabled":"false","integration_grpc_analytics_sample_rate":"1.0","integration_grpc_enabled":"true","integration_grpc_service_name":"my-service-grpc","integration_grpc_distributed_tracing":"true","integration_grpc_error_handler":"#\u003cProc:0x00007f6ecda0abe8 /var/bundle/ruby/3.1.0/gems/ddtrace-1.2.0/lib/datadog/tracing/span_operation.rb:340\u003e","integration_aws_analytics_enabled":"false","integration_aws_analytics_sample_rate":"1.0","integration_aws_enabled":"true","integration_aws_service_name":"my-service-aws","integration_http_analytics_enabled":"false","integration_http_analytics_sample_rate":"1.0","integration_http_enabled":"true","integration_http_service_name":"my-service-net-http","integration_http_distributed_tracing":"true","integration_http_split_by_domain":"false","integration_pg_analytics_enabled":"false","integration_pg_analytics_sample_rate":"1.0","integration_pg_enabled":"true","integration_pg_service_name":"my-service-pg"}

So something in here is still trying to default to 8126 even with the domain socket present.

[ivoanjo]

Hey @jcogilvie, thanks for reaching out!

Unfortunately this hasn't yet been fixed -- having a DD_TRACE_AGENT_URL with a domain socket is not yet recognized (as you've probably spotted 😅).

It's weird that in your case it does not get picked up automatically. One solution I suggest is manually setting it via configuration, as documented in https://docs.datadoghq.com/tracing/trace_collection/dd_libraries/ruby/#using-the-unix-domain-socket-uds-adapter.

Alternatively, double-check if you have any of DD_AGENT_HOST or DD_TRACE_AGENT_PORT set, or any conflicting configuration via code (agent.host, agent.port, tracing.transport_options) as the presence of any of be above takes priority over the auto-detection of the unix domain socket.

Hopefully this gets you unblocked, do let us know if it's still not working for you :)

[relu]

Unfortunately this hasn't yet been fixed -- having a DD_TRACE_AGENT_URL with a domain socket is not yet recognized (as you've probably spotted sweat_smile).

@ivoanjo are there any plans to address this so that the dd-trace-rb becomes consistent with the other flavors?

[ivoanjo]

Thanks for the ping. I may have a bit of time in the next weeks, I'll try to fit this feature in :+1: (and feel free to keep us honest by pinging us!)

[vanHoesel]

I rather not have this fixed the way that DataDog suggest to use it. I think they should first of all fix their own shit.

unix: is just not a valid URI scheme. and thus will break so many other cases.

It should just be file:, because that is what it is... with some interesting characteristics.

[relu]

@vanHoesel correct, according to IANA.

However, such logic exists in the other language implementations of dd-trace, which implies that even if they will switch to the standard file: scheme, they will still have to support unix: in a deprecated state for some time to retain backward compatibility until consumers will be able to transition away from it.

I think the acceptable tradeoff would be to introduce the same logic in dd-trace-rb together with a plan to deprecate unix: scheme across all implementations and most importantly also add support for the standard file: scheme. This way the different language implementations will be consistent.

[vanHoesel]

The reason I responded is that I recently ended up with this issue in my Perl Implementation for OpenTracing, using DataDog as a back-end.

When I wanted to do a type-check on the given environment variable, and check is_URI, it would accept it and treated it as a URI::_Foreign object – which is a pretty useless URI. I could enhance the check with is_URI_http, but there was no is_URI_unix, because the scheme simply does not exist.

One way of dealing with it, is using a substitute s/unix:/file:/. But was making it more complicated. And after all , the internal client wouldn't support file: either.

[ivoanjo]

PR implementing this up on https://github.com/DataDog/dd-trace-rb/pull/2806 :)

I realize unix: is a kinda annoying prefix BUT there's already so many tweaks, knobs and alternative ways of configuring the library that for now I ask for your understanding on not touching that one :)