dd-trace-java icon indicating copy to clipboard operation
dd-trace-java copied to clipboard
verified publisher icon DataDog

Unix Domain sockets (UDS) error when starting profiler

Open amuraco opened this issue 5 months ago • 3 comments

Tracer Version(s)

Starting in 1.52.1

Java Version(s)

24.0.2

JVM Vendor

Amazon Corretto

Bug Report

When starting a Java app with the profiler enabled and Unix domain sockets are configured -Ddd.trace.agent.url=unix:///var/run/datadog/apm.socket, the following error occurs:

[dd.trace 2025-08-21 03:20:49:154 +0000] [dd-task-scheduler] INFO datadog.trace.agent.core.StatusLogger - DATADOG TRACER CONFIGURATION {"version":"1.52.1~6b6db17410","os_name":"Linux","os_version":"5.14.0-503.21.1.el9_5.x86_64","architecture":"amd64","lang":"jvm","lang_version":"24.0.2","jvm_vendor":"Amazon.com Inc.","jvm_version":"24.0.2+12-FR","java_class_version":"68.0","http_nonProxyHosts":"null","http_proxyHost":"null","enabled":true,"service":"adserver-external-production-4969","agent_url":"unix:///var/run/datadog/apm.socket","agent_unix_domain_socket":"/var/run/datadog/apm.socket","agent_error":true,"debug":false,"trace_propagation_style_extract":["datadog","tracecontext","baggage"],"trace_propagation_style_inject":["datadog","tracecontext","baggage"],"analytics_enabled":false,"priority_sampling_enabled":true,"logs_correlation_enabled":true,"profiling_enabled":true,"remote_config_enabled":true,"debugger_enabled":false,"debugger_exception_enabled":false,"debugger_span_origin_enabled":false,"debugger_distributed_debugger_enabled":false,"appsec_enabled":"ENABLED_INACTIVE","rasp_enabled":true,"telemetry_enabled":true,"telemetry_dependency_collection_enabled":true,"telemetry_log_collection_enabled":true,"dd_version":"","health_checks_enabled":true,"configuration_file":"no config file present","runtime_id":"b88ebcb9-2eee-4d72-857c-041082502a62","logging_settings":{"levelInBrackets":false,"dateTimeFormat":"'[dd.trace 'yyyy-MM-dd HH:mm:ss:SSS Z']'","logFile":"System.err","configurationFile":"simplelogger.properties","showShortLogName":false,"showDateTime":true,"showLogName":true,"jsonEnabled":false,"showThreadName":true,"defaultLogLevel":"INFO","warnLevelString":"WARN","embedException":false},"cws_enabled":false,"cws_tls_refresh":5000,"datadog_profiler_enabled":true,"datadog_profiler_safe":true,"datadog_profiler_enabled_overridden":false,"data_streams_enabled":false} at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:544) at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:556) Caused by: java.lang.IllegalArgumentException: Expected URL scheme 'http' or 'https' but was 'unix' at datadog.okhttp3.HttpUrl$Builder.parse(HttpUrl.java:1328) at datadog.okhttp3.HttpUrl.get(HttpUrl.java:916) at com.datadog.profiling.uploader.ProfileUploader.(ProfileUploader.java:151) at com.datadog.profiling.uploader.ProfileUploader.(ProfileUploader.java:134) at com.datadog.profiling.agent.ProfilingAgent.run(ProfilingAgent.java:131) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ... 16 more [dd.trace 2025-08-21 03:20:48:794 +0000] [main] SEND_TELEMETRY datadog.trace.bootstrap.Agent - Throwable thrown while starting profiling agent java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:119) at java.base/java.lang.reflect.Method.invoke(Method.java:565) at datadog.trace.bootstrap.Agent.startProfilingAgent(Agent.java:1252) at datadog.trace.bootstrap.Agent.start(Agent.java:379) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) at java.base/java.lang.reflect.Method.invoke(Method.java:565) at datadog.trace.bootstrap.AgentBootstrap.agentmainImpl(AgentBootstrap.java:159) at datadog.trace.bootstrap.AgentBootstrap.agentmain(AgentBootstrap.java:73) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) at java.base/java.lang.reflect.Method.invoke(Method.java:565) at datadog.trace.bootstrap.AgentPreCheck.continueBootstrap(AgentPreCheck.java:166) at datadog.trace.bootstrap.AgentPreCheck.agentmain(AgentPreCheck.java:24) at datadog.trace.bootstrap.AgentPreCheck.premain(AgentPreCheck.java:17) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) at java.base/java.lang.reflect.Method.invoke(Method.java:565)

This appears to have been introduced by https://github.com/DataDog/dd-trace-java/pull/9339 The 'http://' was previously hardcoded to avoid this error, per similar bug: https://github.com/DataDog/dd-trace-java/issues/7165 & Fix: https://github.com/DataDog/dd-trace-java/pull/7166

Expected Behavior

Java Profiler agent to startup without error when configured with Unix Domain Sockets

amuraco avatar Aug 21 '25 03:08 amuraco

Thanks for reporting! We observed the same issue and went back to 1.52.0.

There are no specific settings for the profiler.

DATADOG TRACER CONFIGURATION {"version":"1.52.1~6b6db17410","os_name":"Linux","os_version":"5.15.0-1088-azure","architecture":"amd64","lang":"jvm","lang_version":"21.0.8","jvm_vendor":"BellSoft","jvm_version":"21.0.8+12-LTS","java_class_version":"65.0","http_nonProxyHosts":"null","http_proxyHost":"null","enabled":true,"service":"redacted","agent_url":"unix:///var/run/datadog/apm.socket","agent_unix_domain_socket":"/var/run/datadog/apm.socket","agent_error":false,"debug":false,"trace_propagation_style_extract":["datadog","tracecontext","baggage"],"trace_propagation_style_inject":["datadog","tracecontext","baggage"],"analytics_enabled":false,"priority_sampling_enabled":true,"logs_correlation_enabled":true,"profiling_enabled":true,"remote_config_enabled":true,"debugger_enabled":false,"debugger_exception_enabled":false,"debugger_span_origin_enabled":false,"debugger_distributed_debugger_enabled":false,"appsec_enabled":"ENABLED_INACTIVE","rasp_enabled":true,"telemetry_enabled":true,"telemetry_dependency_collection_enabled":true,"telemetry_log_collection_enabled":true,"dd_version":"3530-65573a59","health_checks_enabled":true,"configuration_file":"no config file present","runtime_id":"7928e7e2-c3a7-4e6e-b051-95841731eb9c","logging_settings":{"levelInBrackets":false,"dateTimeFormat":"'[dd.trace 'yyyy-MM-dd HH:mm:ss:SSS Z']'","logFile":"System.err","configurationFile":"simplelogger.properties","showShortLogName":false,"showDateTime":true,"showLogName":true,"jsonEnabled":false,"showThreadName":true,"defaultLogLevel":"INFO","warnLevelString":"WARN","embedException":false},"cws_enabled":false,"cws_tls_refresh":5000,"datadog_profiler_enabled":true,"datadog_profiler_safe":true,"datadog_profiler_enabled_overridden":false,"data_streams_enabled":false}

Excerpt from a stacktrace:

... 16 more
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at com.datadog.profiling.agent.ProfilingAgent.run(ProfilingAgent.java:131)
at com.datadog.profiling.uploader.ProfileUploader.<init>(ProfileUploader.java:134)
at com.datadog.profiling.uploader.ProfileUploader.<init>(ProfileUploader.java:151)
at datadog.okhttp3.HttpUrl.get(HttpUrl.java:916)
at datadog.okhttp3.HttpUrl$Builder.parse(HttpUrl.java:1328)
Caused by: java.lang.IllegalArgumentException: Expected URL scheme 'http' or 'https' but was 'unix'
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:572)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:560)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at datadog.trace.bootstrap.AgentPreCheck.premain(AgentPreCheck.java:17)
at datadog.trace.bootstrap.AgentPreCheck.agentmain(AgentPreCheck.java:24)
at datadog.trace.bootstrap.AgentPreCheck.continueBootstrap(AgentPreCheck.java:166)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at datadog.trace.bootstrap.AgentBootstrap.agentmain(AgentBootstrap.java:73)
at datadog.trace.bootstrap.AgentBootstrap.agentmainImpl(AgentBootstrap.java:159)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at datadog.trace.bootstrap.Agent.start(Agent.java:379)
at datadog.trace.bootstrap.Agent.startProfilingAgent(Agent.java:1252)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:118)
java.lang.reflect.InvocationTargetException

jenshadlich avatar Aug 21 '25 07:08 jenshadlich

Sorry about this. Fix is in progress and will be ready shortly.

jbachorik avatar Aug 21 '25 10:08 jbachorik

Any updates on when this will be fixed? @jbachorik

zsait-clearstreet avatar Aug 29 '25 21:08 zsait-clearstreet