opentelemetry-lambda
opentelemetry-lambda copied to clipboard
open-telemetry
Inconsistency in lambda instrumentation across languages.
I have three lambda functions exactly with the same business logic written in java, node.js and python instrumented with the latest version of OpenTelemetry lambda auto-instrumentation packages. Each of these lambda functions make two outbound HTTP calls after invocation.
All the three lambda functions are being invoked exactly same way via API Gateway triggers. While they are being instrumented successfully, the spans generated from them are in completely different ways triggering inconsistencies.
Steps to reproduce a. All the three Lambda functions are attached. b. Just auto-instrument them with latest version of opentelemetry auto instrumentation libraries for lambda and configure them to send the traces to an Otel Collector. c. Configure logging exporter in the Otel Collector for trace. e. Configure API Gateway trigger for each of the lambda functions. f. Make a single invocation to each lambda function via the API gateway trigger. g. Check the OpenTelemetry collector logs.
What I expect to See: a. All three lambdas are expected to generate one SERVER type span for the handler triggered by the HTTP gateway and then generate at least two CLIENT type spans for each of the two HTTP outbound calls made. b. All the spans generated should be part of the same TraceID.
What I really see: A. Java Lambda: a. Three SERVER spans are generated from the API gateway trigger, each showing a completely different TraceID. One from instrumentation scope io.opentelemetry.aws-lambda-events-2.2 1.26.0-alpha and two from instrumentation scope io.opentelemetry.aws-lambda-core-1.0 1.26.0-alpha. b. Two CLIENT spans for two outbound HTTP calls generated from io.opentelemetry.apache-httpclient-4.0 1.26.0-alpha with the same TraceID as the SERVER Span generated from io.opentelemetry.aws-lambda-events-2.2 1.26.0-alpha.
Below is the Otel Collector log snippet for a single invocation:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: 2023-08-04T21:50:19.998Z info TracesExporter {"kind": "exporter", "data_type": "traces", "name": "logging", "resource spans": 3, "spans": 5}
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: 2023-08-04T21:50:19.998Z debug [email protected]/otlp.go:127 Preparing to make HTTP request {"kind": "exporter", "data_type": "metrics", "name": "otlphttp/appdcloud", "url": "https://support-pdx-p01-c3.observe.appdynamics.com/data/v1/metrics"}
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: 2023-08-04T21:50:19.998Z info ResourceSpans #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Resource SchemaURL: https://opentelemetry.io/schemas/1.19.0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Resource attributes:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> cloud.resource_id: Str(arn:aws:lambda:us-west-2:277288375286:function:bisnanda-aws-lambda-java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> host.arch: Str(amd64)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> host.name: Str(169.254.19.213)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> os.description: Str(Linux 4.14.255-311-248.529.amzn2.x86_64)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> os.type: Str(linux)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.command_args: Slice(["/var/lang/bin/java","-XX:MaxHeapSize=445645k","-XX:MaxMetaspaceSize=52429k","-XX:ReservedCodeCacheSize=26214k","-XX:+UseSerialGC","-javaagent:/var/runtime/amzn-log4j-security-jdk11-0.1alpha.jar","-Xshare:on","-XX:SharedArchiveFile=/var/lang/lib/server/runtime.jsa","-XX:-TieredCompilation","-Djava.net.preferIPv4Stack=true","-Dorg.crac.Core.Compat=org.crac.inmemory","-classpath","/var/runtime/lib/aws-lambda-java-core-1.2.2.jar:/var/runtime/lib/aws-lambda-java-runtime-0.2.0.jar:/var/runtime/lib/aws-lambda-java-serialization-0.2.0.jar:/var/runtime/lib/org-crac-inmemory-0.1.0.jar","lambdainternal.AWSLambda"])
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.executable.path: Str(/var/lang/bin/java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.pid: Int(8)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.runtime.description: Str(Amazon.com Inc. OpenJDK 64-Bit Server VM 11.0.19+7-LTS)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.runtime.name: Str(OpenJDK Runtime Environment)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.runtime.version: Str(11.0.19+7-LTS)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> service.name: Str(bisnanda-aws-lambda-java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> service.namespace: Str(aws-lambda)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> telemetry.auto.version: Str(1.26.0)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> telemetry.sdk.language: Str(java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> telemetry.sdk.name: Str(opentelemetry)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> telemetry.sdk.version: Str(1.26.0)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> appdynamics.controller.account: Str(ces-controller)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> appdynamics.controller.host: Str(ces-controller.saas.appdynamics.com)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> appdynamics.controller.port: Int(443)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ScopeSpans #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ScopeSpans SchemaURL:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: InstrumentationScope io.opentelemetry.apache-httpclient-4.0 1.26.0-alpha
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Span #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Trace ID : 5af9c0662e987da44724bddc6856ae55
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Parent ID : 8b8c3f952cf09cf3
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ID : 4fbdb5019fb40322
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Name : GET
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Kind : Client
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Start time : 2023-08-04 21:50:13.384374039 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: End time : 2023-08-04 21:50:13.443981299 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Status code : Unset
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Status message :
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Attributes:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> thread.name: Str(main)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> http.method: Str(GET)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> net.protocol.version: Str(1.1)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> http.url: Str(http://internal-bisnanda-ecs-ec2-internal-34806089.us-west-2.elb.amazonaws.com:30/ecs/ec2/java/hello)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> net.protocol.name: Str(http)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> thread.id: Int(1)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> http.status_code: Int(200)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> net.peer.name: Str(internal-bisnanda-ecs-ec2-internal-34806089.us-west-2.elb.amazonaws.com)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> net.peer.port: Int(30)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> http.response_content_length: Int(76)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Span #1
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Trace ID : 5af9c0662e987da44724bddc6856ae55
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Parent ID : 8b8c3f952cf09cf3
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ID : 806aede6fce14c90
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Name : GET
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Kind : Client
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Start time : 2023-08-04 21:50:13.48442222 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: End time : 2023-08-04 21:50:13.503998599 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Status code : Unset
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Status message :
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Attributes:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> thread.name: Str(main)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> http.method: Str(GET)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> net.protocol.version: Str(1.1)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> http.url: Str(http://ec2-52-43-189-49.us-west-2.compute.amazonaws.com/json/products.json)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> net.protocol.name: Str(http)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> thread.id: Int(1)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> http.status_code: Int(200)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> net.peer.name: Str(ec2-52-43-189-49.us-west-2.compute.amazonaws.com)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> http.response_content_length: Int(2485)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ScopeSpans #1
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ScopeSpans SchemaURL:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: InstrumentationScope io.opentelemetry.aws-lambda-events-2.2 1.26.0-alpha
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Span #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Trace ID : 5af9c0662e987da44724bddc6856ae55
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Parent ID :
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ID : 8b8c3f952cf09cf3
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Name : bisnanda-aws-lambda-java
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Kind : Server
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Start time : 2023-08-04 21:50:13.365332 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: End time : 2023-08-04 21:50:13.504745744 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Status code : Unset
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Status message :
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Attributes:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> thread.name: Str(main)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> cloud.resource_id: Str(arn:aws:lambda:us-west-2:277288375286:function:bisnanda-aws-lambda-java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> thread.id: Int(1)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> cloud.account.id: Str(277288375286)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> faas.invocation_id: Str(9ed131bd-c1a8-41ca-bfcb-cda1672f61ae)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ResourceSpans #1
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Resource SchemaURL: https://opentelemetry.io/schemas/1.19.0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Resource attributes:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> cloud.resource_id: Str(arn:aws:lambda:us-west-2:277288375286:function:bisnanda-aws-lambda-java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> host.arch: Str(amd64)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> host.name: Str(169.254.19.213)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> os.description: Str(Linux 4.14.255-311-248.529.amzn2.x86_64)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> os.type: Str(linux)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.command_args: Slice(["/var/lang/bin/java","-XX:MaxHeapSize=445645k","-XX:MaxMetaspaceSize=52429k","-XX:ReservedCodeCacheSize=26214k","-XX:+UseSerialGC","-javaagent:/var/runtime/amzn-log4j-security-jdk11-0.1alpha.jar","-Xshare:on","-XX:SharedArchiveFile=/var/lang/lib/server/runtime.jsa","-XX:-TieredCompilation","-Djava.net.preferIPv4Stack=true","-Dorg.crac.Core.Compat=org.crac.inmemory","-classpath","/var/runtime/lib/aws-lambda-java-core-1.2.2.jar:/var/runtime/lib/aws-lambda-java-runtime-0.2.0.jar:/var/runtime/lib/aws-lambda-java-serialization-0.2.0.jar:/var/runtime/lib/org-crac-inmemory-0.1.0.jar","lambdainternal.AWSLambda"])
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.executable.path: Str(/var/lang/bin/java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.pid: Int(8)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.runtime.description: Str(Amazon.com Inc. OpenJDK 64-Bit Server VM 11.0.19+7-LTS)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.runtime.name: Str(OpenJDK Runtime Environment)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.runtime.version: Str(11.0.19+7-LTS)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> service.name: Str(bisnanda-aws-lambda-java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> service.namespace: Str(aws-lambda)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> telemetry.auto.version: Str(1.26.0)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> telemetry.sdk.language: Str(java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> telemetry.sdk.name: Str(opentelemetry)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> telemetry.sdk.version: Str(1.26.0)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> appdynamics.controller.account: Str(ces-controller)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> appdynamics.controller.host: Str(ces-controller.saas.appdynamics.com)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> appdynamics.controller.port: Int(443)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ScopeSpans #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ScopeSpans SchemaURL:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: InstrumentationScope io.opentelemetry.aws-lambda-core-1.0 1.26.0-alpha
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Span #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Trace ID : 91228e8b067f5e2d4e92607bda5cd4ba
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Parent ID :
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ID : 095c02b9b8f0090d
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Name : bisnanda-aws-lambda-java
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Kind : Server
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Start time : 2023-08-04 21:50:13.365288 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: End time : 2023-08-04 21:50:13.64412236 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Status code : Unset
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Status message :
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Attributes:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> thread.name: Str(main)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> cloud.resource_id: Str(arn:aws:lambda:us-west-2:277288375286:function:bisnanda-aws-lambda-java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> thread.id: Int(1)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> cloud.account.id: Str(277288375286)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> faas.invocation_id: Str(9ed131bd-c1a8-41ca-bfcb-cda1672f61ae)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Links:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: SpanLink #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> Trace ID: 64cd7295464c71963515de8403d5b235
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> ID: 79fb02975ab75964
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> TraceState:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> DroppedAttributesCount: 0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> Attributes::
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> source: Str(x-ray-env)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ResourceSpans #2
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Resource SchemaURL: https://opentelemetry.io/schemas/1.19.0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Resource attributes:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> cloud.resource_id: Str(arn:aws:lambda:us-west-2:277288375286:function:bisnanda-aws-lambda-java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> host.arch: Str(amd64)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> host.name: Str(169.254.19.213)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> os.description: Str(Linux 4.14.255-311-248.529.amzn2.x86_64)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> os.type: Str(linux)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.command_args: Slice(["/var/lang/bin/java","-XX:MaxHeapSize=445645k","-XX:MaxMetaspaceSize=52429k","-XX:ReservedCodeCacheSize=26214k","-XX:+UseSerialGC","-javaagent:/var/runtime/amzn-log4j-security-jdk11-0.1alpha.jar","-Xshare:on","-XX:SharedArchiveFile=/var/lang/lib/server/runtime.jsa","-XX:-TieredCompilation","-Djava.net.preferIPv4Stack=true","-Dorg.crac.Core.Compat=org.crac.inmemory","-classpath","/var/runtime/lib/aws-lambda-java-core-1.2.2.jar:/var/runtime/lib/aws-lambda-java-runtime-0.2.0.jar:/var/runtime/lib/aws-lambda-java-serialization-0.2.0.jar:/var/runtime/lib/org-crac-inmemory-0.1.0.jar","lambdainternal.AWSLambda"])
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.executable.path: Str(/var/lang/bin/java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.pid: Int(8)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.runtime.description: Str(Amazon.com Inc. OpenJDK 64-Bit Server VM 11.0.19+7-LTS)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.runtime.name: Str(OpenJDK Runtime Environment)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> process.runtime.version: Str(11.0.19+7-LTS)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> service.name: Str(bisnanda-aws-lambda-java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> service.namespace: Str(aws-lambda)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> telemetry.auto.version: Str(1.26.0)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> telemetry.sdk.language: Str(java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> telemetry.sdk.name: Str(opentelemetry)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> telemetry.sdk.version: Str(1.26.0)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> appdynamics.controller.account: Str(ces-controller)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> appdynamics.controller.host: Str(ces-controller.saas.appdynamics.com)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> appdynamics.controller.port: Int(443)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ScopeSpans #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ScopeSpans SchemaURL:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: InstrumentationScope io.opentelemetry.aws-lambda-core-1.0 1.26.0-alpha
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Span #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Trace ID : 571cc1e464cf9a5a8edaecfc3c25a6b8
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Parent ID :
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: ID : 0ba08cd04d96c2bf
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Name : bisnanda-aws-lambda-java
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Kind : Server
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Start time : 2023-08-04 21:50:13.365175 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: End time : 2023-08-04 21:50:13.706600342 +0000 UTC
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Status code : Unset
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Status message :
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Attributes:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> thread.name: Str(main)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> cloud.resource_id: Str(arn:aws:lambda:us-west-2:277288375286:function:bisnanda-aws-lambda-java)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> thread.id: Int(1)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> cloud.account.id: Str(277288375286)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> faas.invocation_id: Str(9ed131bd-c1a8-41ca-bfcb-cda1672f61ae)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: Links:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: SpanLink #0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> Trace ID: 64cd7295464c71963515de8403d5b235
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> ID: 79fb02975ab75964
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> TraceState:
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> DroppedAttributesCount: 0
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> Attributes::
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: -> source: Str(x-ray-env)
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: {"kind": "exporter", "data_type": "traces", "name": "logging"}
Aug 04 21:50:20 ip-10-0-13-36 start-otelcol.sh[1723927]: 2023-08-04T21:50:20.006Z debug [email protected]/otlp.go:127 Preparing to make HTTP request {"kind": "exporter", "data_type": "traces", "name": "otlphttp/appdcloud", "url": "https://support-pdx-p01-c3.observe.appdynamics.com/data/v1/trace"}
B. Node.js Lambda: a. No SERVER type spans are being generated at all. b. Two INTERNAL spans from instrumentation scope @opentelemetry/instrumentation-net 0.32.0 and two CLIENT spans from instrumentation scope @opentelemetry/instrumentation-http 0.41.1 are being generated. The TraceId of one INTERNAL span matches with one of the CLIENT spans and the TraceId of another INTERNAL span matches with the TraceID of the other CLIENT span.
Below is the Otel Collector log snippet for a single invocation:
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: 2023-08-04T21:43:14.841Z info TracesExporter {"kind": "exporter", "data_type": "traces", "name": "logging", "resource spans": 1, "spans": 4}
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: 2023-08-04T21:43:14.842Z info ResourceSpans #0
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Resource SchemaURL:
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Resource attributes:
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> service.name: Str(bisnanda-aws-lambda-nodejs)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> telemetry.sdk.language: Str(nodejs)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> telemetry.sdk.name: Str(opentelemetry)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> telemetry.sdk.version: Str(1.15.1)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> cloud.provider: Str(aws)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> cloud.platform: Str(aws_lambda)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> cloud.region: Str(us-west-2)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> faas.name: Str(bisnanda-aws-lambda-nodejs)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> faas.version: Str($LATEST)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> service.namespace: Str(aws-lambda)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> cloud.resource_id: Str(arn:aws:lambda:us-west-2:277288375286:function:bisnanda-aws-lambda-nodejs)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> process.pid: Int(8)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> process.executable.name: Str(/var/lang/bin/node)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> process.executable.path: Str(/var/lang/bin/node)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> process.command_args: Slice(["/var/lang/bin/node","--expose-gc","--max-http-header-size","81920","--max-semi-space-size=6","--max-old-space-size=116","/var/runtime/index.mjs"])
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> process.runtime.version: Str(18.16.1)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> process.runtime.name: Str(nodejs)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> process.runtime.description: Str(Node.js)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> process.command: Str(/var/runtime/index.mjs)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> process.owner: Str(sbx_user1051)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> appdynamics.controller.account: Str(ces-controller)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> appdynamics.controller.host: Str(ces-controller.saas.appdynamics.com)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> appdynamics.controller.port: Int(443)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: ScopeSpans #0
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: ScopeSpans SchemaURL:
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: InstrumentationScope @opentelemetry/instrumentation-net 0.32.0
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Span #0
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Trace ID : 33bd0b8f9241c6c77b8fe9b43954a1a1
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Parent ID : 365a6c4cc24d2415
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: ID : 23382ec398a02d05
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Name : tcp.connect
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Kind : Internal
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Start time : 2023-08-04 21:42:24.536999936 +0000 UTC
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: End time : 2023-08-04 21:42:24.876628992 +0000 UTC
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Status code : Unset
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Status message :
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Attributes:
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.transport: Str(ip_tcp)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.peer.name: Str(ec2-52-43-189-49.us-west-2.compute.amazonaws.com)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.peer.port: Int(80)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.peer.ip: Str(10.0.10.214)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.host.ip: Str(169.254.76.1)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.host.port: Int(53462)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Span #1
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Trace ID : 516fe8d1ccd786cf6fc7b1e44180c3c5
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Parent ID : c3f4123cfa5757fa
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: ID : eea68d81fee6bea3
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Name : tcp.connect
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Kind : Internal
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Start time : 2023-08-04 21:42:24.536 +0000 UTC
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: End time : 2023-08-04 21:42:24.877544192 +0000 UTC
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Status code : Unset
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Status message :
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Attributes:
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.transport: Str(ip_tcp)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.peer.name: Str(internal-bisnanda-ecs-ec2-internal-34806089.us-west-2.elb.amazonaws.com)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.peer.port: Str(40)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.peer.ip: Str(10.0.40.22)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.host.ip: Str(169.254.76.1)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.host.port: Int(37510)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: ScopeSpans #1
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: ScopeSpans SchemaURL:
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: InstrumentationScope @opentelemetry/instrumentation-http 0.41.1
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Span #0
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Trace ID : 33bd0b8f9241c6c77b8fe9b43954a1a1
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Parent ID :
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: ID : 365a6c4cc24d2415
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Name : GET
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Kind : Client
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Start time : 2023-08-04 21:42:24.536999936 +0000 UTC
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: End time : 2023-08-04 21:42:24.879968768 +0000 UTC
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Status code : Unset
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Status message :
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Attributes:
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.url: Str(http://ec2-52-43-189-49.us-west-2.compute.amazonaws.com/json/books.json)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.method: Str(GET)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.target: Str(/json/books.json)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.peer.name: Str(ec2-52-43-189-49.us-west-2.compute.amazonaws.com)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.host: Str(ec2-52-43-189-49.us-west-2.compute.amazonaws.com:80)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.peer.ip: Str(10.0.10.214)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.peer.port: Int(80)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.response_content_length_uncompressed: Int(848)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.status_code: Int(200)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.status_text: Str(OK)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.flavor: Str(1.1)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.transport: Str(ip_tcp)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Span #1
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Trace ID : 516fe8d1ccd786cf6fc7b1e44180c3c5
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Parent ID :
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: ID : c3f4123cfa5757fa
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Name : GET
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Kind : Client
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Start time : 2023-08-04 21:42:24.535000064 +0000 UTC
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: End time : 2023-08-04 21:43:06.36994816 +0000 UTC
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Status code : Unset
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Status message :
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Attributes:
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.url: Str(http://internal-bisnanda-ecs-ec2-internal-34806089.us-west-2.elb.amazonaws.com:40/ecs/ec2/nodejs/hello)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.method: Str(GET)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.target: Str(/ecs/ec2/nodejs/hello)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.peer.name: Str(internal-bisnanda-ecs-ec2-internal-34806089.us-west-2.elb.amazonaws.com)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.host: Str(internal-bisnanda-ecs-ec2-internal-34806089.us-west-2.elb.amazonaws.com:40)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.peer.ip: Str(10.0.40.22)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.peer.port: Int(40)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.status_code: Int(200)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.status_text: Str(OK)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.flavor: Str(1.1)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.transport: Str(ip_tcp)
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: {"kind": "exporter", "data_type": "traces", "name": "logging"}
Aug 04 21:43:14 ip-10-0-13-36 start-otelcol.sh[1718482]: 2023-08-04T21:43:14.842Z debug [email protected]/otlp.go:127 Preparing to make HTTP request {"kind": "exporter", "data_type": "traces", "name": "otlphttp/appdcloud", "url": "https://support-pdx-p01-c3.observe.appdynamics.com/data/v1/trace"}
B. Python Lambda: a. One SERVER type span is generated corresponding to the API Gateway trigger from instrumentation scope opentelemetry.instrumentation.aws_lambda 0.40b0. b. No CLIENT type spans are generated at all. So the two outbound HTTP calls are not being tracked.
Below is the Otel Collector log snippet for a single invocation:
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: 2023-08-04T21:39:14.829Z info TracesExporter {"kind": "exporter", "data_type": "traces", "name": "logging", "resource spans": 1, "spans": 1}
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: 2023-08-04T21:39:14.830Z info ResourceSpans #0
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Resource SchemaURL:
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Resource attributes:
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> telemetry.sdk.language: Str(python)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> telemetry.sdk.name: Str(opentelemetry)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> telemetry.sdk.version: Str(1.19.0)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> cloud.region: Str(us-west-2)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> cloud.provider: Str(aws)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> faas.name: Str(bisnanda-aws-lambda-python)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> faas.version: Str($LATEST)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> faas.instance: Str(2023/08/04/[$LATEST]6e7687d2ff6a4c7e8bc577dacf4625b4)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> service.namespace: Str(aws-lambda)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> cloud.resource_id: Str(arn:aws:lambda:us-west-2:277288375286:function:bisnanda-aws-lambda-python)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> service.name: Str(bisnanda-aws-lambda-python)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> telemetry.auto.version: Str(0.40b0)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> appdynamics.controller.account: Str(ces-controller)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> appdynamics.controller.host: Str(ces-controller.saas.appdynamics.com)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> appdynamics.controller.port: Int(443)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: ScopeSpans #0
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: ScopeSpans SchemaURL:
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: InstrumentationScope opentelemetry.instrumentation.aws_lambda 0.40b0
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Span #0
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Trace ID : e5508d4afeb8280e1370ad838c9dd044
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Parent ID :
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: ID : 09a7ddff9c33a5b1
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Name : lambda_function.lambda_handler
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Kind : Server
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Start time : 2023-08-04 21:39:06.313833645 +0000 UTC
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: End time : 2023-08-04 21:39:06.448371414 +0000 UTC
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Status code : Unset
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Status message :
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: Attributes:
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> faas.id: Str(arn:aws:lambda:us-west-2:277288375286:function:bisnanda-aws-lambda-python)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> faas.execution: Str(a1581204-a256-4c00-937b-a4d6428d4555)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> faas.trigger: Str(http)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.method: Str(GET)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.user_agent: Str(Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.scheme: Str(https)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> net.host.name: Str(ssn9gz3t4l.execute-api.us-west-2.amazonaws.com)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.route: Str(/bisnanda-aws-lambda-python)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.target: Str(/bisnanda-aws-lambda-python)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: -> http.status_code: Int(200)
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: {"kind": "exporter", "data_type": "traces", "name": "logging"}
Aug 04 21:39:14 ip-10-0-13-36 start-otelcol.sh[1718482]: 2023-08-04T21:39:14.829Z debug [email protected]/otlp.go:127 Preparing to make HTTP request {"kind": "exporter", "data_type": "traces", "name": "otlphttp/appdcloud", "url": "https://support-pdx-p01-c3.observe.appdynamics.com/data/v1/trace"}
What version of collector/language SDK version did you use? Collector: Latest Version Auto instrumentation library: Latest version Runtime: Java 11, Node.js 18.x, Python 3.11
sourcecode.zip otelcol_log_snippet.zip
This issue was marked stale. It will be closed in 30 days without additional activity.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Still relevant. Would be nice if issues weren't marked as stale without a comment from a maintainer.
@gshpychka do you have thoughts on how you'd fix this? Open to suggestions.
This particular project doesn't have a lot of backing from big vendors, so the maintainers are mainly here on our free time. As such, we need volunteers to help push things forward.
@gshpychka do you have thoughts on how you'd fix this? Open to suggestions.
This particular project doesn't have a lot of backing from big vendors, so the maintainers are mainly here on our free time. As such, we need volunteers to help push things forward.
Oh, please don't take my comment the wrong way - I understand what you wrote, it's just a bit discouraging for issues to be closed without any response. So maybe the stale bot could be tweaked somewhat.
I tried to be very generous on the configuration for the bot. It only marks things as stale when there's been no interaction for a year, and it waits a month before closing something marked as stale. Do you have other suggestions?
I tried to be very generous on the configuration for the bot. It only marks things as stale when there's been no interaction for a year, and it waits a month before closing something marked as stale. Do you have other suggestions?
Yes, is it possible to configure it so that it doesn't mark issues as stale unless there's been a comment from a maintainer (or, if that's complicated to define, form any user)?
This issue was marked stale. It will be closed in 30 days without additional activity.
Closed as inactive. Feel free to reopen if this issue is still relevant.
@gshpychka I understand your frustration, but there just isn't staffing and enough interest from the community to fully implement this. I also don't think keeping issues open for years without anyone looking at them helps either.
That said, I do think @wpessers has been making some progress in making the javascript instrumentation look more like the java one.
Well yes, but also @gshpychka if you want changes to how these instrumentations work / which spans are exported with specific attributes etc. you’re looking in the wrong place. I suggest you take a look at language specific sdks / instrumentation libs. For example: https://github.com/open-telemetry/opentelemetry-js-contrib
