Application Insights not tracing Kafka Dependencies in a Java JDK 11 local App

[Justrebl]

Expected behavior

Having a Java Kafka Consumer App instrumented with the application-insights agent running locally, i am expecting to see the tracked dependencies (application insights logs and application map not showing the consumer dependency) while the Java consumer retrieves events from a Kafka enabled Event Hub.

Actual behavior

Currently, all i can see being traced with the java app insight agent are the requests logs from my java client app to the Kafka (event-hub) source, but not the dependancy logs (thus application map is not showing the link between my java client/consumer app

To Reproduce

Steps to reproduce the behavior:

• Create a kafka event producer to aliment a Event Hub Kafka enabled namespace (i created a simple Azure Function with an EventHub output binding to create new event in the same topic on a per second basis, that can be seen in the application map representation).
• Clone the following Microsoft Kafka consumer example here : https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/quickstart/java and configure the resources/consumer.config as detailed in the consumer/readme.md to point at the kafka endpoint and topic of a Premium Event Hub.
• Instrument the Java consumer app with the application-insight java agent 3.4.1
• Run the Java application to consume the Event Hub/Kafka endpoint
• Check the application insights logs (table : requests & dependencies)
• Run the following queries in Log Analytics :
  • requests | where cloud_RoleName contains "consumer" => 1 requests Line per event consumed showing in the KQL query results
  • dependencies | where cloud_RoleName contains "consumer" => 0 line result

Sample Application

This Microsoft provided Java Kafka Consumer reproduces the behavior : https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/quickstart/java

.mvn/jvm.config :

-javaagent:./applicationinsights-agent-3.4.1.jar

applicationinsights.json :

{ "connectionString": "InstrumentationKey=0000;IngestionEndpoint=https://00000.in.applicationinsights.azure.com/;LiveEndpoint=https://0000.livediagnostics.monitor.azure.com/", "role":{ "name":"Java Kafka Consumer", "instance":"OnPremise" }, "instrumentation": { "logging": { "level": "TRACE" } }, "sampling":{ "percentage":100 } }

System information

Please provide the following information:

• SDK Version:
  • jdk-11.0.16
  • App insight java agent 3.4.1
• OS type and version: Windows 11 (same behavior on Linux WSL2 - Ubuntu 20.04 LTS)
• Application Server type and version (if applicable):
• Using spring-boot? No

Screenshots

Please let me know if i can extend on above explanations to help you reproduce the behavior.

[trask]

hi @Justrebl!

i created a simple Azure Function with an EventHub output binding to create new event in the same topic on a per second basis, that can be seen in the application map representation

can you try with a local kafka app that pushes to the queue to see if that works? I suspect that this is related to Azure Functions output bindings or not propagating W3C traceparent header on the messages, but I'm not positive. thx!

[Justrebl]

Hello @trask, Thanks for reaching out and sorry for the late response here !

New test today (code here https://github.com/Justrebl/azure-event-hubs-for-kafka/tree/master/quickstart/java) with a local kafka java app pushing to the same event hub kafka enabled consumed by the local app.

It actually brings something interesting in the intelligent view (Application map preview) :

When i leave the preview i can see direct communication between the producer and the consumer (without seeing the event hub in the middle, and 0 call on the producer : is this intended?)

Checking the logs, here is what i find for requests (only traces for the consumer are appearing) :

And Dependencies (only the producer events are traced in the dependencies):

Checking the logs i have for the Azure Function discussed earlier, i can see entries in the Dependencies AND the requests (Azure function being a producer working directly with the Az Function Event Hub output binding) : Requests :

Dependencies:

So having a kafka producer AND a kafka consumer seem to be tracked as expected in the application map (even if we're losing the actual kafka/event hub queue in the picture : is this expected?), but how about tracking other producers' dependecies?

Thanks again for your help on this topic!

[trask]

hey @Justrebl, I was able to run your demo,

In "preview mode" I got this picture:

Give it another try, sometimes it takes several minutes for the backend indexer to draw the right app map. Also, it's possible that the app map team has rolled out a fix since you tried.

The "0 calls" is correct b/c it's the number of requests to that node, and it has no "requests", since it's just generating messages from the command line.

[ghost]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 7 days. It will be closed if no further activity occurs within 7 days of this comment.

[Justrebl]

Hello @trask, Interesting you have such visibility (as well as the time in queue) with the same app code. I have also tried the same with the app-insight agent on 3.4.2 version, but still see the same result in "Preview Mode", and here is what i get even 30 minutes after the execution :

Would you be so kind to share the config (is it the same i committed on the repro GH repo?). To set the app-insight agent jvm-arg, i updated my .vscode launch.json file as follows :

Also added the agent directly in the project root folder, but couldn't make it work the same you shared above :

Am i missing something important ?

[Justrebl]

Also, for the dependancy tracking to show E2E view on the application map from the Azure Function to the ultimate Java consumer (Az Function output binding to Event Hub), should i be setting explicitely the traceparent header to make sure i can have the same E2E view on the application map as the one you showed in your screenshot ?

[trask]

@Justrebl based on our offline conversation, it sounds like the issue you were seeing in the latest repro may be related to https://github.com/microsoft/ApplicationInsights-Java/issues/1814#issuecomment-892053798. I've asked the app map team about this, and they are considering to trigger the app map correlation without any minimum transaction requirement, or if not, at least to document this behavior.

for the original issue with Azure Functions EventHub output binding, I believe that is because Azure Functions EventHub binding uses "diagnostic-id" header to transport distributed trace context, while the Application Insights Java agent uses "traceparent" header.

the long-term direction is to use the "traceparent", and the EventHubs SDK which is used by Azure Functions is targeting to support "traceparent" in addition to "diagnostic-id" by end of March, and Azure Functions is targeting to update to that version of the EventHub SDK shortly after, at which time this should work

[ghost]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 7 days. It will be closed if no further activity occurs within 7 days of this comment.

[Justrebl]

Hey @trask,

Really sorry for the late reply on this one, i was expecting an offline ping and didn't check it earlier ! Thank you so much for all the efforts on this issue and for the extra work on checking with Azure Function Roadmap !

Consider the issue resolved !