feat: add example using Sentry V2 SDK

[gregbrowndev]

What was changed

Added a new example to set up Sentry using the V2 SDK.

Why?

The original Sentry integration only works with Sentry SDK v1, due to issues with the warnings and threading modules that Sentry uses not being appropriately included in the Temporal Workflow's sandbox when SDK v2 is installed. (Presumably, Sentry SDK v2 made some internal changes, so these modules are now causing issues.)

Given that v1 is now deprecated and that v2 will be installed by default if the user runs poetry add sentry-sdk in their own project, this new sample provides the required integration for Sentry SDK v2 to work without sandbox issues.

The necessary changes only required migrating the use of the Hub object (which is now deprecated) to using Sentry scopes.

The original Sentry v1 sample is kept as the SDK still receives security patches and is not yet EOL.

Checklist

1. Closes

https://temporalio.slack.com/archives/CTT84RS0P/p1725394300914329

2. How was this tested:

Tested manually in my application using Sentry.io

3. Any docs updates needed?

Added a new example with a README explaining the reasons/difference between the original example and the v2 example.

[CLAassistant]

All committers have signed the CLA.

[gregbrowndev]

@cretz just a note on Sentry SDK v1 EOL:

v1 is now only receiving security patches and we will likely EOL it with the release of v3 of the SDK, which is coming in the next few months. link

[cretz]

Uh oh, v3 so soon? Ok, I guess we will cross that bridge when we get there. Otherwise, I think this looks good and I'll merge if/when CI passes. The purpose is to serve as a sample, so users can adapt it as they need for their version of tooling.

[gregbrowndev]

Hey @cretz, do you want me to rebase the PR?

[cretz]

@gregbrowndev - I just merged main into your branch, no problem. Looks like just the CLA needs to be signed (see link above pointing to https://cla-assistant.io/temporalio/samples-python?pullRequest=140). May need to make a comment here when done so I can get notified to merge this.

[M0NsTeRRR]

Hello,

I just tested with temporalio~=1.11.1 and sentry-sdk~=2.25.1 and it works perfectly fine except I have to pass sentry_sdk as passthrough module in my worker.

worker = Worker(
       [...]
        workflow_runner=SandboxedWorkflowRunner(
            restrictions=SandboxRestrictions.default.with_passthrough_modules(
                "sentry_sdk"
            )
        ),
    )

If I don't do it I get

Failed activation on workflow XXXXX with ID 6cded3f4-73d0-4707-b96b-14bd7a03fa70 and run ID 0196ce1e-1b49-77e2-9493-417350b7be16
temporalio.exceptions.ApplicationError: NameError: name 'x' is not defined

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/developpement/temporal-project/temporal-project/temporal_project/utils/sentry.py", line 73, in execute_workflow
    return await super().execute_workflow(input)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/temporalio/worker/_interceptor.py", line 333, in execute_workflow
    return await self.next.execute_workflow(input)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/temporalio/worker/_workflow_instance.py", line 2328, in execute_workflow
    return await input.run_fn(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/developpement/temporal-project/temporal-project/temporal_project/workflows/crc/radius.py", line 365, in run
    await workflow.execute_activity(
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/temporalio/workflow.py", line 2360, in execute_activity
    return await _Runtime.current().workflow_start_activity(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/temporalio/worker/_workflow_instance.py", line 1559, in run_activity
    return await asyncio.shield(handle._result_fut)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
temporalio.exceptions.ActivityError: Activity task failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/temporalio/worker/_workflow_instance.py", line 406, in activate
    self._run_once(check_conditions=index == 1 or index == 2)
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/temporalio/worker/_workflow_instance.py", line 1929, in _run_once
    raise self._current_activation_error
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/temporalio/worker/_workflow_instance.py", line 1947, in _run_top_level_workflow_function
    await coro
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/temporalio/worker/_workflow_instance.py", line 893, in run_workflow
    result = await self._inbound.execute_workflow(input)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/developpement/temporal-project/temporal-project/temporal_project/utils/sentry.py", line 88, in execute_workflow
    scope.capture_exception()
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/sentry_sdk/scope.py", line 1263, in capture_exception
    event, hint = event_from_exception(
                  ^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/sentry_sdk/utils.py", line 1134, in event_from_exception
    "values": exceptions_from_error_tuple(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/sentry_sdk/utils.py", line 942, in exceptions_from_error_tuple
    single_exception_from_error_tuple(
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/sentry_sdk/utils.py", line 736, in single_exception_from_error_tuple
    serialize_frame(
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/sentry_sdk/utils.py", line 608, in serialize_frame
    from sentry_sdk.serializer import serialize
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/temporalio/worker/workflow_sandbox/_importer.py", line 442, in __call__
    return self.current(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/temporalio/worker/workflow_sandbox/_importer.py", line 234, in _import
    mod = importlib.__import__(name, globals, locals, fromlist, level)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1466, in __import__
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/sentry_sdk/__init__.py", line 1, in <module>
    from sentry_sdk.scope import Scope
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/temporalio/worker/workflow_sandbox/_importer.py", line 442, in __call__
    return self.current(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/temporalio/worker/workflow_sandbox/_importer.py", line 234, in _import
    mod = importlib.__import__(name, globals, locals, fromlist, level)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1466, in __import__
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/sentry_sdk/scope.py", line 1788, in <module>
    from sentry_sdk.client import NonRecordingClient
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/temporalio/worker/workflow_sandbox/_importer.py", line 442, in __call__
    return self.current(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/temporalio/worker/workflow_sandbox/_importer.py", line 234, in _import
    mod = importlib.__import__(name, globals, locals, fromlist, level)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1466, in __import__
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/sentry_sdk/client.py", line 28, in <module>
    from sentry_sdk.transport import BaseHttpTransport, make_transport
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/temporalio/worker/workflow_sandbox/_importer.py", line 442, in __call__
    return self.current(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/temporalio/worker/workflow_sandbox/_importer.py", line 234, in _import
    mod = importlib.__import__(name, globals, locals, fromlist, level)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1466, in __import__
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/sentry_sdk/transport.py", line 18, in <module>
    import urllib3
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/temporalio/worker/workflow_sandbox/_importer.py", line 442, in __call__
    return self.current(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/temporalio/worker/workflow_sandbox/_importer.py", line 234, in _import
    mod = importlib.__import__(name, globals, locals, fromlist, level)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1466, in __import__
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/urllib3/__init__.py", line 14, in <module>
    from . import exceptions
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/temporalio/worker/workflow_sandbox/_importer.py", line 442, in __call__
    return self.current(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/temporalio/worker/workflow_sandbox/_importer.py", line 234, in _import
    mod = importlib.__import__(name, globals, locals, fromlist, level)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1486, in __import__
  File "<frozen importlib._bootstrap>", line 1415, in _handle_fromlist
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/home/user/developpement/temporal-project/.venv/lib/python3.12/site-packages/urllib3/exceptions.py", line 261, in <module>
    class IncompleteRead(HTTPError, httplib_IncompleteRead):
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

[gregbrowndev]

Sorry @cretz, I've only just seen that this PR was blocked by me not signing the CLA. I've done so now.

Could you comment on @M0NsTeRRR 's message above? Is the SandboxedWorkflowRunner passthrough restrictions the same thing as using the decorator in the workflow like below or are both necessary? I haven't looked at the project I used the Sentry integration in a while, but can re-test this sample again if I get some time today/tomorrow

with workflow.unsafe.imports_passed_through():
    import sentry_sdk

    from sentry.interceptor import SentryInterceptor

[M0NsTeRRR]

After reading it again, I realized that I didn’t import the interceptor in my workflow, which is likely why I got this error — my bad. However, I think it makes more sense for the example to whitelist it at the worker level, since every exception will be caught by the Sentry interceptor.

Thanks for your answer @gregbrowndev :)

[cretz]

I've only just seen that this PR was blocked by me not signing the CLA. I've done so now.

Thanks! Would you be willing to merge main and resolve conflicts?

Is the SandboxedWorkflowRunner passthrough restrictions the same thing as using the decorator in the workflow like below or are both necessary?

Yes, they are the same thing, but note some imports are done lazily by some libraries so we have to account for that too. What we usually do in newer samples and what we encourage users to do (and would like to here, though don't have to now), is to have workflows be in their own file instead of a shared file.

[Natim]

I have the same TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases issue even though I import it in the worker with the context manager workflow.unsafe.imports_passed_through()

Should I import the interceptor in all my workflow files too?

Edit: The answer is that you should do it where you import your workflows to build the list of workflows and activities.

[gregbrowndev]

@cretz ready to review again, sorry for dragging on this. The uv migration caused an issue that took me ages to fix 🙏🏻

Notes:

1. To confirm, I saw the "metaclass conflict" error that @Natim reported, causing the workflow task to fail, after I split the worker, workflow, and activity into separate modules. It doesn't seem that having the pass-through in the workflow file made a difference, even when I updated it to pass through the whole package:

   with workflow.unsafe.imports_passed_through():
       import sentry_sdk
   

   Passing through the package via the worker's workflow_runner fixed the error. Note: sure what you make of this?

2. As already mentioned, I found an issue with uv. Because the gevent sample is included in default-groups, it installs the gevent library, which breaks Sentry's isolation_scope as it relies on setting contextvars. See the "Context Variables vs gevent/eventlet" in Sentry troubleshooting. I've had to update the README to exclude default groups so it doesn't get installed.

   Could this be a problem for other samples, too? Maybe excluding the gevent sample from the default groups would be better, and only install it specifically for that sample. Note: uv run also needs to have the --group on it, otherwise it just reinstalls all the default groups and then you get the same errors again. (Same also after doing poe format.)

3. Lastly, I noticed that the Worker SDK is using a _ShutdownRequested exception to trigger shutdowns. I guess because the exception is never extracted from wait_task = asyncio.wait(tasks, return_when=asyncio.FIRST_EXCEPTION) the AsyncioIntegration` raises it as an unhandled error.

[cretz]

To confirm, I saw the "metaclass conflict" error [...]

Is it possible to add some test that doesn't require Sentry for this sample? Our Sentry V1 sample started breaking and we didn't know because there were no tests. Also, when the test is added, we can easily replicate this error and I can help debug it.

As already mentioned, I found an issue with uv. Because the gevent sample is included in default-groups, it installs the gevent library, which breaks Sentry's isolation_scope as it relies on setting contextvars.

Yeah, we have an issue to remove default groups, they were added by accident: #190

Lastly, I noticed that the Worker SDK is using a _ShutdownRequested exception to trigger shutdowns. I guess because the exception is never extracted from wait_task = asyncio.wait(tasks, return_when=asyncio.FIRST_EXCEPTION) the AsyncioIntegration` raises it as an unhandled error.

The worker just uses this inside its function to interrupt the wait, it is never raised outside of that call (I don't believe wait raises in this instance)

[gregbrowndev]

@cretz

Is it possible to add some test that doesn't require Sentry for this sample?

Yes, good idea I will add these tests. I should get a bit of time this weekend

Yeah, we have an issue to remove default groups, they were added by accident: https://github.com/temporalio/samples-python/issues/190

Sounds good 👍🏻

The worker just uses this inside its function to interrupt the wait, it is never raised outside of that call (I don't believe wait raises in this instance)

It's not that the exception bubbles out of the worker, it's just that the wait doesn't extract the result of the coroutine explicitly, so Sentry treats it as an unhandled exception, similar to if you did asyncio.create_task but didn't read the result. It's no big deal, I just wanted to raise it.

[gregbrowndev]

@cretz one last thing to mention. I've been working on a fix for the TracingInterceptor in the Python SDK. It doesn't propagate the trace context for process pool workers so log-correlation and other tracing features don't work within our activity impl. inside the subprocess, not sure if its been fixed yet. I've got a working implementation, just trying to get the tests to pass.

I noticed it was easy to break the Sentry interceptor's reflection features, e.g. input.fn.__module__, the way I did it initially, and because the Sentry interceptor is in user land, I didn't notice until I deployed onto staging.

Would you consider making the Sentry interceptor a native integration of the Python SDK? I'd be happy to open a PR to the main SDK repo! I'm also working on a Sentry interceptor for the TypeScript SDK too which doesn't have any user-provided samples yet.

[cretz]

I've got a working implementation, just trying to get the tests to pass

:+1: Approved CI to run. This may benefit from #212.

Would you consider making the Sentry interceptor a native integration of the Python SDK?

Not sure this is something we can officially maintain and support outside of a sample at this time. Ideally the sample can just be referenced by users. The tests in the sample will ensure it does not break so it can be sure to continue to work.

[Natim]

Not sure this is something we can officially maintain and support outside of a sample at this time.

Making Sentry a first class citizen of your SDK would be great I believe for the Python's adoption. Having a sample is better than nothing though, but the inherent complexity makes it hard to work with.

[cretz]

@gregbrowndev - may need a main merge and a regen the lock file

[gregbrowndev]

@cretz - merged main and tested/linted locally on 3.10 through 3.13, so hopefully CI will pass 👍🏻