simular-ai
Many setup issues with OSWorld
Hello, I’m trying to run Agent S2 in the OSWorld environment and I need your help. I’m running into the following issues:
1. Failure to import the Agent S2 classes
After following the setup instructions in the README and those specific to OSWorld, simply running a task with the run.py script in OSWorld fails with the following error:
Traceback (most recent call last):
File "/home/user/Documents/OSWorld/run.py", line 18, in <module>
from gui_agents.s2.agents.grounding import OSWorldACI
ModuleNotFoundError: No module named 'gui_agents'
It makes sense, because we are instructed to use a different virtual environment for OSWorld than the one for Agent S, so it doesn’t have the gui-agents package installed. (I’m using uv and not conda, but I don’t think that should make any difference.)
2. Failure loading the initial knowledge base
On the first run of my run.py script with Agent S2, the knowledge base successfully gets downloaded. On the following runs, I see the following being logged:
Downloading knowledge base initial Agent-S knowledge...
Path local_kb_path /home/user/OSWorld/kb/ubuntu already exists. Skipping download.
If you'd like to re-download the initial knowledge base, please delete the existing knowledge base at /home/user/OSWorld/kb/ubuntu.
Despite this, I have the following error right after the VM starts:
[2025-03-31 16:03:07,457 INFO run_single/198-MainProcess] [Example ID]: bb5e4c0d-f964-439c-97b6-bdb9747de3f4
[2025-03-31 16:03:07,457 INFO run_single/202-MainProcess] [Instruction]: Can you make Bing the main search thingy when I look stuff up on the internet?
[2025-03-31 16:03:08,379 INFO python/32-MainProcess] Got screenshot successfully
[2025-03-31 16:04:08,955 INFO python/32-MainProcess] Got screenshot successfully
[2025-03-31 16:04:08,975 INFO python/318-MainProcess] Recording started successfully
[2025-03-31 16:04:08,976 INFO agent_s/193-MainProcess] (RE)PLANNING...
search query: How to set Bing as default search engine in Ubuntu browser
Error loading knowledge base: [Errno 2] No such file or directory: '/home/user/OSWorld/kb/ubuntu/narrative_memory.json'
I tried deleting the kb folder, which let do it being re-downloaded successfully, but the error persisted.
However, simply creating this file with an empty json object {} makes the error go away. But I’m not sure if this is sufficient for things to run successfully, because then I run into this next error:
3. Perplexica config not found
After getting rid of the previous error by creating a kb/ubuntu/narrative_memory.json file that just contains {}, I get the following execution trace:
[2025-04-01 21:00:57,530 INFO run_single/198-MainProcess] [Example ID]: bb5e4c0d-f964-439c-97b6-bdb9747de3f4
[2025-04-01 21:00:57,530 INFO run_single/202-MainProcess] [Instruction]: Can you make Bing the main search thingy when I look stuff up on the internet?
[2025-04-01 21:01:05,548 INFO python/32-MainProcess] Got screenshot successfully
[2025-04-01 21:02:06,207 INFO python/32-MainProcess] Got screenshot successfully
[2025-04-01 21:02:06,241 INFO python/318-MainProcess] Recording started successfully
[2025-04-01 21:02:06,241 INFO agent_s/193-MainProcess] (RE)PLANNING...
[2025-04-01 21:02:06,246 INFO manager/131-MainProcess] SIMILAR TASK EXPERIENCE: None
None
[2025-04-01 21:02:06,248 ERROR run_single/232-MainProcess] Exception in chrome/bb5e4c0d-f964-439c-97b6-bdb9747de3f4: [Errno 2] No such file or directory: '/home/user/OSWorld/.venv/lib/python3.12/site-packages/Perplexica/config.toml'
I don’t know why it’s looking for the config there, because I followed the instructions, and I thus have the Agent-S repository cloned together with Perplexica and its config in the parent of my OSWorld folder. That is, my Perplexica config file is in /home/user/Agent-S/Perplexica/config.toml, so why would the code look for it in my virtual environment’s python packages?
I would need your help to fix this and to get Agent S2 running with OSWorld. What could I do to fix this?
Thanks a lot! 🙏
Hi @thomas-kuntz , I'm working on #53 to help simplify the setup for the kb and the library.
- Failure to import the Agent S2 classes
The setup requires you to pip install and clone the repo. My PR will make it pip installable, and the kb can be downloaded separately. If the the package still isn't recognized, you can add the path to the top of your script:
import sys
sys.path.append("path/to/Agent-S")
- Failure loading the initial knowledge base
Hmm, this one is weird. Have you checked that the narrative_memory.json file exists in that directory? Creating an empty JSON file with {} does work, but it also means there will be no prior narrative experience to retrieve from (which will lead to some performance losses).
- Perplexica config not found
Follow the Perplexica setup here
Hi, just following up, please check out the osworld_setup folder for more information. 😃
I've updated the folder names of the knowledge base assets. This was causing the no file found error.
Hi, thanks a lot for your answer and sorry for the delay! 🙏 In the meantime I had found a way to make my project work around these issues by using another agent, but, should I try S2 again, I will let you know if some issues persist!