Multitoken substitution

[david-alm]

Hey 👋

I recently came across unasync and wanted to commend you on this incredibly promising & useful project.

In that vein, I wanted to note that certain substitutions (notably additional_replacements={"@pytest.mark.trio": ""}) effectively results in a no-op as a result of the tokenization converting @pytest.mark.trio to the tokens @pytest, mark and trio. Although token-based execution is theoretically superior, it adds a layer of complexity that httpcore's unasync.py doesn't face.

My recommendation would be to also tokenize the additional_replacements and attempt to match the token streams. Happy to implement it once we're aligned on a solution.

[sethmlarson]

Multi-token replacements are a little more complex, but if you want a workaround for this now you can use the following:

# pytest marks
pytest_mark_trio = pytest.mark.trio

def pytest_mark_sync(f):
    return f  # no-op decorator

# unasync config

additional_replacements={"pytest_mark_trio": "pytest_mark_sync"}

# test files

@pytest_mark_trio
async def test_example(): ...

@pytest_mark_sync
def test_example(): ...

[johtso]

I'm doing something even more basic, just replacing the trio or asyncio token with sync..

https://github.com/johtso/httpx-caching/blob/master/tests/_async/test_chunked_response.py#L11

https://github.com/johtso/httpx-caching/blob/master/gen_sync.py#L24

I guess that's probably not the best idea, and only works because I'm not referencing trio or asyncio directly anywhere. But yeah, if that's an issue, you can just assign it to a variable to make it more explicit.

[david-alm]

Appreciate the workarounds. Just flagging that this doesn't behave as expected. I'll PR this enhancement if you're okay with it.

[sethmlarson]

That'd be great, thanks!