Running guidance within Gradio

[HaukurPall]

Hi,

I'm trying to run guidance within a Gradio app. I'm not sure about the async framework behind Gradio but it seems to be AnyIO and is using some worker threads.

The problem I am experiencing is that when I attempt to run a program I get an error that there is no current event loop in the thread. I am not trying to use the program asynchronously (the flag passed in).

  File "/data/scratch/haukurpj/miniconda3/envs/greynirqa/lib/python3.9/site-packages/guidance/_program.py", line 194, in __call__
    new_program = Program(
  File "/data/scratch/haukurpj/miniconda3/envs/greynirqa/lib/python3.9/site-packages/guidance/_program.py", line 110, in __init__
    self._execute_complete = asyncio.Event() # fires when the program is done executing to resolve __await__
  File "/data/scratch/haukurpj/miniconda3/envs/greynirqa/lib/python3.9/asyncio/locks.py", line 177, in __init__
    self._loop = events.get_event_loop()
  File "/data/scratch/haukurpj/miniconda3/envs/greynirqa/lib/python3.9/site-packages/nest_asyncio.py", line 45, in _get_event_loop
    loop = events.get_event_loop_policy().get_event_loop()
  File "/data/scratch/haukurpj/miniconda3/envs/greynirqa/lib/python3.9/asyncio/events.py", line 642, in get_event_loop
    raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'AnyIO worker thread'.

Any help is welcome.

[slundberg]

Thanks for the heads up. Do you have a minimal example? Having not used Gradio I don't know, but it could be that it is doing something special with async loops (we use async internally for streaming etc.)

[HaukurPall]

Here is a fairly minimal example:

import gradio
import guidance

guidance.llm = guidance.llms.OpenAI("gpt-4")
prog = guidance("""{{#system~}}
You are a helpful assistant.
{{~/system}}
{{#user~}}
Hi, I'm {{name}}. 
{{~/user}}
{{#assistant~}}
{{gen 'response' n=5 temperature=1.0 max_tokens=500}}
{{~/assistant}}""")


def greet(name):
    return prog(name=name)["response"]


iface = gradio.Interface(fn=greet, inputs="text", outputs="text")
iface.launch()

I'm using Python 3.9.16, gradio==3.27.0 and guidance==0.0.43

[slundberg]

Thanks! It just turned out to be an issue with nest_asyncio causing issues when it is not needed but asyncio is also in use. I just pushed a fix to master.

[elBarkey]

using guidance 0.0.54 facing the same issue