Asserterror when using in flask app, gevent, magic libraries

[ddorian]

Description

The error below happens only when using the combination of gevent, logfire, and importing magic.

Full code:

import gevent.monkey

gevent.monkey.patch_all()

import logfire

logfire.configure(token="-----------------------------------")
# todo doing this import shows the error
import magic  
from flask import Flask

app = Flask(__name__)


@app.route("/")
def hello_world():
    return "<p>Hello, World!</p>"


if __name__ == "__main__":
    app.run()

Logfire project URL: https://logfire.pydantic.dev/--------------
Traceback (most recent call last):
  File "src/gevent/_abstract_linkable.py", line 287, in gevent._gevent_c_abstract_linkable.AbstractLinkable._notify_links
  File "src/gevent/_abstract_linkable.py", line 333, in gevent._gevent_c_abstract_linkable.AbstractLinkable._notify_links
AssertionError: (None, <callback at 0x7f4ec2dfdf40 args=([],)>)
2024-05-01T09:06:11Z <callback at 0x7f4ec2dfdf40 args=([],)> failed with AssertionError

 * Serving Flask app 'fshije'
 * Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:5000
Press CTRL+C to quit

gevent==24.2.1
flask==3.0.3
python-magic==0.4.27

Python, Logfire & OS Versions, related packages

logfire="0.28.0"
platform="Linux-5.15.0-105-generic-x86_64-with-glibc2.35"
python="3.12.3 (main, Apr 27 2024, 19:00:26) [GCC 9.4.0]"
[related_packages]
requests="2.31.0"
pydantic="2.7.1"
protobuf="4.25.3"
rich="13.7.1"
opentelemetry-api="1.24.0"
opentelemetry-exporter-otlp-proto-common="1.24.0"
opentelemetry-exporter-otlp-proto-http="1.24.0"
opentelemetry-instrumentation="0.45b0"
opentelemetry-instrumentation-aiohttp-client="0.45b0"
opentelemetry-instrumentation-dbapi="0.45b0"
opentelemetry-instrumentation-flask="0.45b0"
opentelemetry-instrumentation-jinja2="0.45b0"
opentelemetry-instrumentation-psycopg2="0.45b0"
opentelemetry-instrumentation-redis="0.45b0"
opentelemetry-instrumentation-requests="0.45b0"
opentelemetry-instrumentation-sqlalchemy="0.45b0"
opentelemetry-instrumentation-sqlite3="0.45b0"
opentelemetry-instrumentation-system-metrics="0.45b0"
opentelemetry-instrumentation-urllib="0.45b0"
opentelemetry-instrumentation-urllib3="0.45b0"
opentelemetry-instrumentation-wsgi="0.45b0"
opentelemetry-proto="1.24.0"
opentelemetry-sdk="1.24.0"
opentelemetry-semantic-conventions="0.45b0"
opentelemetry-util-http="0.45b0"

[Kludex]

I can't reproduce it.

I've also added the FlaskInstrumentor.instrument(app) to actually test that the traces are being sent:

import gevent.monkey

gevent.monkey.patch_all()

import logfire

logfire.configure()
# todo doing this import shows the error
import magic
from flask import Flask
from opentelemetry.instrumentation.flask import FlaskInstrumentor

app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)


@app.route('/')
def hello_world():
    return '<p>Hello, World!</p>'


if __name__ == '__main__':
    app.run()

This is the actual packages installed retrieved from pip freeze (I've installed the packages mentioned in the description):

pip freeze

annotated-types==0.6.0

blinker==1.8.1 certifi==2024.2.2 charset-normalizer==3.3.2 click==8.1.7 Deprecated==1.2.14 Flask==3.0.3 gevent==24.2.1 googleapis-common-protos==1.63.0 greenlet==3.0.3 idna==3.7 importlib-metadata==7.0.0 itsdangerous==2.2.0 Jinja2==3.1.3 logfire==0.28.0 markdown-it-py==3.0.0 MarkupSafe==2.1.5 mdurl==0.1.2 opentelemetry-api==1.24.0 opentelemetry-exporter-otlp-proto-common==1.24.0 opentelemetry-exporter-otlp-proto-http==1.24.0 opentelemetry-instrumentation==0.45b0 opentelemetry-instrumentation-aiohttp-client==0.45b0 opentelemetry-instrumentation-dbapi==0.45b0 opentelemetry-instrumentation-flask==0.45b0 opentelemetry-instrumentation-jinja2==0.45b0 opentelemetry-instrumentation-psycopg2==0.45b0 opentelemetry-instrumentation-redis==0.45b0 opentelemetry-instrumentation-requests==0.45b0 opentelemetry-instrumentation-sqlalchemy==0.45b0 opentelemetry-instrumentation-sqlite3==0.45b0 opentelemetry-instrumentation-system-metrics==0.45b0 opentelemetry-instrumentation-urllib==0.45b0 opentelemetry-instrumentation-urllib3==0.45b0 opentelemetry-instrumentation-wsgi==0.45b0 opentelemetry-proto==1.24.0 opentelemetry-sdk==1.24.0 opentelemetry-semantic-conventions==0.45b0 opentelemetry-util-http==0.45b0 packaging==24.0 protobuf==4.25.3 psutil==5.9.8 pydantic==2.7.1 pydantic_core==2.18.2 Pygments==2.17.2 python-magic==0.4.27 requests==2.31.0 rich==13.7.1 setuptools==69.5.1 typing_extensions==4.11.0 urllib3==2.2.1 Werkzeug==3.0.2 wrapt==1.16.0 zipp==3.18.1 zope.event==5.0 zope.interface==6.3

Am I doing something wrong?

[ddorian]

I created a new project with fresh installs just to be sure. What if you try this exact poetry project ? poetry_project.zip

And then do python main.py:

❯ python main.py
Logfire project URL: https://logfire.pydantic.dev/------------
Traceback (most recent call last):
  File "src/gevent/_abstract_linkable.py", line 287, in gevent._gevent_c_abstract_linkable.AbstractLinkable._notify_links
  File "src/gevent/_abstract_linkable.py", line 333, in gevent._gevent_c_abstract_linkable.AbstractLinkable._notify_links
AssertionError: (None, <callback at 0x7f622d828840 args=([],)>)
2024-05-01T09:49:40Z <callback at 0x7f622d828840 args=([],)> failed with AssertionError

 * Serving Flask app 'main'
 * Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:5000
Press CTRL+C to quit

[Kludex]

I still can't reproduce it.

Are you sure you are running the virtual environment created by poetry? I had to create the logfire_test/ directory, and the __init__.py file on your example, and I actually run the app with poetry run python main.py.

[ddorian]

Are you sure you are running the virtual environment created by poetry?

Yes, the IDE uses it for me.

I had to create the logfire_test/ directory,

Agree

and the init.py file on your example,

Not needed.

and I actually run the app with poetry run python main.py.

Yes. This is the full project without .idea & .logfire dirs logfire_test.zip

@Kludex maybe you need the os to be the same? (ubuntu 22.04)

I replicated in a github codespace https://github.com/ddorian/jubilant-space-potato

[davidhewitt]

I could reproduce in your codespace. It seems to be a combination of gevent + opentelemetry + magic that's the root cause. I get the same error from the following minimal repro:

import gevent.monkey

gevent.monkey.patch_all()

from opentelemetry.sdk.trace.export import BatchSpanProcessor, SpanExporter

processor = BatchSpanProcessor(SpanExporter(), schedule_delay_millis=500)

# todo doing this import shows the error
import magic

requirements.txt:

Deprecated==1.2.14
gevent==24.2.1
greenlet==3.0.3
importlib-metadata==7.0.0
opentelemetry-api==1.24.0
opentelemetry-sdk==1.24.0
opentelemetry-semantic-conventions==0.45b0
python-magic==0.4.27
typing_extensions==4.11.0
wrapt==1.16.0
zipp==3.18.1
zope.event==5.0
zope.interface==6.3

[davidhewitt]

I can also reproduce locally with the same example (Ubuntu 24.04, Python 3.12.3)

[davidhewitt]

Reported upstream at https://github.com/open-telemetry/opentelemetry-python/issues/3897

[imnotdev25]

I'm also facing the same issue with Fastapi. Any quick fix?

[Kludex]

I'm also facing the same issue with Fastapi. Any quick fix?

It shouldn't affect FastAPI... 🤔

[Kludex]

Since this issue is on OTel side, I'll be closing it here.

If interested, you can watch this: https://github.com/open-telemetry/opentelemetry-python/issues/3897.

[davidhewitt]

As per https://github.com/open-telemetry/opentelemetry-python/issues/3897#issuecomment-2345712705 - it seems like the root cause is interaction between gevent and ctypes. The recommended workaround is to import those libraries before calling logfire.configure.