Azure Function OpenTelemetry instrumentation of FastAPI

[claria]

• Package Name: azure-monitor-opentelemetry
• Package Version:
• Operating System: Windows/Linux
• Python Version: 3.11.9

Describe the bug

I am testing to use opentelemetry for a fastapi app running in azure function. I use the auto instrumentation of fastapi given by the package opentelemetry-instrumentation-fastapi.

While it is working and all logs and spans are exported to Azure AppInsights, the specific entry in the requests table is created in an unfortunate way.

I have one route "/hello/{name}". As you can see from the spans created from opentelemetry the spans get the name "GET /hello/{name}" while the entry in the requests table gets the name "GET /hello/1" (with the actual url). This destroys the grouping in appInsights as can bee seen in the screenshots:

To Reproduce

function_app.py

import logging

import azure.functions as func
import fastapi
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

fastapi_app = fastapi.FastAPI()

FastAPIInstrumentor.instrument_app(fastapi_app)

logger = logging.getLogger(__name__)


@fastapi_app.get("/hello/{name}")
async def get_name(name: str):
    logger.info(f"Hello {name}")
    return {
        "name": name,
    }


app = func.AsgiFunctionApp(app=fastapi_app,
                           http_auth_level=func.AuthLevel.ANONYMOUS,
                           function_name='fastapi')
import logging

import azure.functions as func
import fastapi
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

fastapi_app = fastapi.FastAPI()

FastAPIInstrumentor.instrument_app(fastapi_app)

logger = logging.getLogger(__name__)


@fastapi_app.get("/hello/{name}")
async def get_name(name: str):
    logger.info(f"Hello {name}")
    return {
        "name": name,
    }


app = func.AsgiFunctionApp(app=fastapi_app,
                           http_auth_level=func.AuthLevel.ANONYMOUS,
                           function_name='fastapi')

host.json:

{
  "version": "2.0",
  "extensions": {
    "http": {
      "routePrefix": ""
    }
  },
  "telemetryMode": "OpenTelemetry"
}

local.settings.json

{
  "IsEncrypted": false,
  "Values": {
    "PYTHON_ENABLE_INIT_INDEXING": "1",
    "PYTHON_ENABLE_OPENTELEMETRY": true,
    "PYTHON_APPLICATIONINSIGHTS_ENABLE_TELEMETRY": true,
    "FUNCTIONS_WORKER_RUNTIME": "python",
    "AzureWebJobsStorage": "XXXXXXXXXXXXXXXXXXXXXX",
    "APPLICATIONINSIGHTS_CONNECTION_STRING": "XXXXXXXXXXXXXXXXXXXXXXX"
  }
}

requirements.txt:

azure-functions==1.24.0b3
fastapi==0.116.1
azure-monitor-opentelemetry==1.6.13
opentelemetry-instrumentation-fastapi==0.57.b0

Expected behavior Use the route name (with route param template) or the actual endpoint (function) name.

Screenshots

Additional context Add any other context about the problem here.

[github-actions[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @gulopesd @Haiying-MSFT @jairmyree @joshfree @KarishmaGhiya @KevinBlasko @kurtzeborn @pvaneck @scottaddie @srnagar @ToddKingMSFT.

[claria]

Hi, is there any feedback to this issue?

I tested a bit and found out the following:

• the entries in the requests table in appinsights are populated by server type spans in opentelemetry

• Fastapi opentelemetry integration (correctly) does not produce a server type span as we have an external correlation context.

• The current entries in the requests table are not produced by the python azure function worker but already sometime before (in the c# worker?)

• I could not find a configuration setting to disable the logging for these entries (like setting Host.Results logging to error)

The only way i found to correctly show the routes as separates operations i had to do the following.

• break the correlation context by calling configure_azure_monitor(...) from within the app and not setting "telemetryMode": "OpenTelemetry" in host.results and disabling the duplicated logs within host.json logging configuration

But of course this brings again more downsides than advantages. In my opinion, there should be a possibility to set the span/requests name from within the application itself.

[claria]

@hectorhdzg Any recommendation what to do here?

[rads-1996]

@claria Thanks for reaching out! I’ve tested using the same environment and code you provided, and I’m seeing the grouping behave as expected. I’ve set telemetryMode to OpenTelemetry in host.json, and everything works correctly on my end. For reference, I’m using azure-monitor-opentelemetry==1.8.1 and opentelemetry-instrumentation-fastapi==0.58b0.

[github-actions[bot]]

Hi @claria. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[claria]

Hi @rads-1996,

i switched to the latest version of azure function core tools and now see the same behaviour as you do. However, i still find this very inconvenient.

In our production API, we use a fastapi app with ~100 routes. By using

app = func.AsgiFunctionApp(app=fastapi_app,
                           http_auth_level=func.AuthLevel.ANONYMOUS,
                           function_name='fastapi')

this adds one azure function matching /{*route} which subsequently groups all operations in appinsights unter the operations GET /*{route}, POST /*{route} which does not make sense, e.g. when using the (very helpful) Performance/Failures tabs

It would be much more useful to use the fastapi defined routes as the operationName to have everything grouped according to their actual operation:

GET /items/{item_id}
GET /orders/{order_id}
GET /orders/

instead of grouping everything in one GET /*{route} operation

[rads-1996]

@claria I agree with you about the function app behavior, they have hardcoded all endpoints to fall under *route and that behavior is correctly reflected in application insights -

I tried running the application without using function app and discovered that the grouping is being done correctly because FastApi handles the requests directly without the azure function wrapper -

Since the logic has been hardcoded by the azure-functions team, you can reach out to them for an alternative or a possible solution, the azure-monitor-opentelemetry is correctly grouping the requests as demonstrated above without using the function app.

This is the place where the logic for *route has been added - https://github.com/Azure/azure-functions-python-library/blob/b28a46a23d4feebef89d8675efad2b988f867100/azure/functions/decorators/function_app.py and the repository where you can raise the issue.

[github-actions[bot]]

Hi @claria. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[github-actions[bot]]

Hi @claria, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

[claria]

Hi, please keep the issue open until somebody reacts to the issue in the azure-functions-python-worker repo. Thanks

[claria]

Hi, the issue in the azure-functions-worker repository was closed with the reccomendation to follow-up here:

Hi @hallvictoria ,

the hard-coded path+function is not the issue. My issue is that Azure Functions treats this single function as the defining operation in regard to what is exported to Appinsights requests. I think the application that is called by functionapp should make the decision how it defines/names an operation.

AppInsights makes all its grouping based on the server span that is exported to the requests tables. If i understand correctly, this span is not exported by the python worker. Only the trace context is passed to the application. The OpenTelemetry Instrumentors (not only fastapi instrumentor) create a server or internal span based on if there is already an (external) server span or not.

• This means we can either ignore/disable the trace correlation and create another server span from inside FastAPI (not desireable since we are intrested in the trace correlation)
• We use the trace correlation which leads to the OpenTelemetry Instrumentors creating an internal span. This would be fine if we would have the possibility to group by these spans in AppInsights. This is not possible. Also not a desirable solution.

https://github.com/open-telemetry/opentelemetry-python-contrib/blob/185502b3f8bd45f955777004725b7140f8c18e65/opentelemetry-instrumentation/src/opentelemetry/instrumentation/utils.py#L115-L153

[rads-1996]

@claria Since the grouping is being done to *route in azure functions and without that the sdk correctly groups the endpoints, you can try using a custom middleware that attempts to override the grouping being done in azure functions -

import logging

import azure.functions as func
import fastapi
from opentelemetry import trace
from opentelemetry.trace import SpanKind

tracer = trace.get_tracer(__name__)

fastapi_app = fastapi.FastAPI()
logger = logging.getLogger(__name__)


@fastapi_app.get("/hello/{name}")
async def get_name(name: str):
    logger.info(f"Hello {name}")
    return {"name": name}

@fastapi_app.get("/items/{item_id}")
async def get_item(item_id: str):
    logger.info(f"Item requested: {item_id}")
    return {"item": item_id}


class TelemetryMiddleware:
    def __init__(self, app):
        self.app = app

    async def __call__(self, scope, receive, send):
        if scope["type"] != "http":
            return await self.app(scope, receive, send)

        with tracer.start_as_current_span("request", kind=SpanKind.SERVER) as span:
            await self.app(scope, receive, send)
            route = scope.get("route")
            if route:
                span.update_name(f"{scope['method']} {route.path}")
                span.set_attribute("http.route", route.path)


app = func.AsgiFunctionApp(app=TelemetryMiddleware(fastapi_app),
                           http_auth_level=func.AuthLevel.ANONYMOUS,
                           function_name='fastapi')

This is what it looks post using this middleware -

Now you will still see the traces for GET {*route} from azure functions but will see your expected grouping under the performance tile.

You can customize the middleware further according to your requirements. This is just a basic skeleton.

[github-actions[bot]]

Hi @claria. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[github-actions[bot]]

Hi @claria, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

[claria]

@rads-1996

This solution works really well. We tested this now for several days in our setup and found only one downside. We were not able to silence/disable the server spans for "Get /{*route}" emitted by the function host.

In the before-Opentelemetry world we were able to use host.json settings to disable these entries

{
...
  "logging": {
    "ApplicationInsights": {
      "logLevel": {
        "Host.Results": "Error",
      }
    }
}

However, adapting this to OpenTelemetry according to this link does not seem to work.

host.json

{
...
  "logging": {
    "OpenTelemetry": {
      "logLevel": {
        "Host.Results": "Warning",
      }
    }
}

Do you have an idea for this final puzzle piece to achieve?