crewAI crash after first task completion by agent

hi, I want to report a bug or possibly something that I did wrong, but here is full example of my code and also the error, it seemed to crash right after the completion of first task or two, but I don't know why 😢 .

also, is there a way to turn off these logs? it shows even when verbose=False in the agent params:

40483732952832 - sqlite.py-sqlite:284 - WARNING: Insert of existing embedding ID: default-app-id--8bfc331f185daae036dafdd9f284c445fd090491b16b2c3735194749afd18c00
2024-03-22 03:01:52,317 - 140483732952832 - sqlite.py-sqlite:284 - WARNING: Insert of existing embedding ID: default-app-id--834eee17eae23fb6a803ce2aca8df73561e831b1094e42896e17b147e4abda8c
2024-03-22 03:01:52,317 - 140483732952832 - sqlite.py-sqlite:284 - WARNING: Insert of existing embedding ID: default-app-id--c43d4b8b654991798b8d295c3bfab9dee21dc1978e4e20f8d7b3c4a6c0414c5b
2024-03-22 03:01:52,318 - 140483732952832 - sqlite.py-sqlite:284 - WARNING: Insert of existing embedding ID: default-app-id--d84afc87598f0a867e0fab558ecf23476790a56be0393e778c44cb7f8a40d495

my code example:


verse_glossary_search = TXTSearchTool(txt='docs/glossary/UEFN_glossary.txt')
verse_language_reference = DirectorySearchTool(directory='docs/verse-language-reference')
verse_language_documentation = DirectorySearchTool(directory='docs/verse-complete-tutorials')
verse_example_projects = DirectorySearchTool(directory='docs/sample-projects')

# google search
# search_tool = SerperDevTool()

# Creating a senior researcher agent with memory and verbose mode
researcher = Agent(
  role='New programming languages Researcher',
  goal='Researching and Understanding the new Verse programming language concepts and unique new syntax which differs from all other programming languages. Keep a short list of concepts you already learned, to avoid repeating same research.',
  verbose=True,
  memory=True,
 # llm=llm1,
  backstory=(
    "You are learning the new Verse programming language which was made by Epic games for UEFN Unreal Editor For Fortnite and has full documentation provided in the tools."
  ),
  tools=[verse_glossary_search, verse_language_reference, verse_language_documentation, verse_example_projects],
  allow_delegation=True,
  max_rpm=10,
  max_iter=10
)

advanced_researcher = Agent(
  role='Verse programming language Researcher',
  goal='Researching and Understanding the new Verse programming language advanced and unique concepts like new syntax which differs from all other programming languages, concurrency approach, operators and functions. Keep a short list of concepts you already learned, to avoid repeating same research.',
  verbose=True,
  memory=True,
  #llm=llm2,
  backstory=(
    "You are looking for unique and advanced feature of the Verse programming language which was made by Epic games for UEFN Unreal Editor For Fortnite and you base your conclusions on the full Verse documentation and reference which is provided in the tools."
  ),
  tools=[verse_glossary_search, verse_language_reference, verse_language_documentation, verse_example_projects],
  allow_delegation=True,
  max_rpm=10,
  max_iter=10
)

developer = Agent(
  role='Developer',
  goal='Understand information provided by Verse programming language researcher and write Verse code examples and explanations.',
  verbose=True,
  memory=True,
  #llm=llm1,
  backstory=(
    "You are Verse programmer who gets tasks from researchers and writes code for game programming concepts strictly in Verse language."
  ),
  tools=[verse_glossary_search, verse_language_reference, verse_language_documentation, verse_example_projects],
  allow_delegation=True,
  max_rpm=10,
  max_iter=10
)

critic = Agent(
  role='Critic',
  goal='Verify facts and syntax correctness of written code, based on provided Verse programming reference, glossary and documentation. Find the mistakes of the developer if they used imagined, hypothetical, or probable constructs not confirmed by the provided documentation. The goal is to ensure accuracy and relevance of the code to the specific characteristics of the Verse programming language as described in the reference material.',
  verbose=True,
  memory=True,
  #llm=llm2,
  backstory=(
    "You are expert Verse programmer and critic who makes sure everything written by the Developer can be fully confirmed by the provided Verse programming language documentation, examples, and code references, which are provided in the tools."
  ),
  tools=[verse_glossary_search, verse_language_reference, verse_language_documentation, verse_example_projects],
  allow_delegation=True,
  max_rpm=10,
  max_iter=10
)


totalExamplesWritten = 0
totalReviews = 0



def example_code_written(output):
    global totalExamplesWritten
    totalExamplesWritten += 1
    log_message = f"""
    _______________________
        Code completed!
        Task: {output.description}
        --------------------------
        Output: {output.raw_output}
    
    """
    print(log_message)
    with open('log/examples.txt', 'a') as log_file:
        log_file.write(log_message)

    source_file = 'results/verse-code-example.txt'
    destination_file = f'results/verse-code-example-{totalExamplesWritten}.txt'
    shutil.copy(source_file, destination_file)


def reviewed_code_written(output):
    global totalReviews
    totalReviews += 1
    log_message = f"""
    _____________________
        Review completed!
        Task: {output.description}
        --------------------------
        Output: {output.raw_output}

    """
    print(log_message)
    with open('log/reviews.txt', 'a') as log_file:
        log_file.write(log_message)

    source_file = 'reviewed-results/fixed-verse-code-example.txt'
    destination_file = f'reviewed-results/fixed-verse-code-example-{totalReviews}.txt'
    shutil.copy(source_file, destination_file)


# Research task
simple_research_task = Task(
  description=(
    "Start reading the Verse programming documentation provided, start from simple concepts, then ask Developer to write example code that strictly adheres to the documented language concepts, keywords, and constructs. Pay attention that Verse code in our documentation is enclosed into special tags VERSE_CODE_START and VERSE_CODE_END for your convenience. Keep track of what concepts you already asked developer to write about, to avoid repeating."
  ),
  expected_output='High quality chosen Verse programming concept information, syntax, rules, explanation in a short form, for the developer to write a good example of the concept. Do not use imagined, hypothetical, or probable constructs of Verse language not confirmed by the provided documentation.',
  agent=researcher,
  async_execution=True
)

complex_research_task = Task(
  description=(
    "Look through Verse programming documentation provided, for advanced concepts, then suggest Researcher to consider these new concepts and features of the language in his next task for the developer. Pay attention that Verse code in our documentation is enclosed into special tags VERSE_CODE_START and VERSE_CODE_END for your convenience. Keep track of the concepts which you already researched, to avoid repeating yourself."
  ),
  expected_output='Explanation about an advanced or unusual and difficult Verse programming concept, the syntax, rules, described in short form, for the Researcher, so he will consider assigning the developer to write an advanced example of the difficult concept.',
  agent=advanced_researcher,
  async_execution=True
)

# Writing task with language model configuration
write_task = Task(
  description=(
    "Write a short example code and explanation about the concepts provided by the Researcher. Do not use imagined, hypothetical, or probable constructs of Verse language not confirmed by the provided documentation. All Verse code in our documentation is enclosed into special tags VERSE_CODE_START and VERSE_CODE_END for your convenience."
  ),
  expected_output='Verse code example about a specific concept of the language, and a short description about where did you find the proof in documentation which confirmed the example correctness and validity. Make sure to separate the code using VERSE_CODE_START and VERSE_CODE_END tags as well.',
  agent=developer,
  async_execution=True,
  context=[simple_research_task],
  output_file='results/verse-code-example.txt',
  callback=example_code_written
)

review_task = Task(
  description=(
    "Review the example code and explanation written by developer and search for confirmation in the provided documentation. If the confirmation is found for all the syntax and concepts used by the developer in his example, approve the task, and if not, ask him to modify and fix the code and example and explain how."
  ),
  expected_output='Explanation for developer about what is wrong in his Verse code example or description, and how to fix it, and where in documentation did you find the proof about this mistake.',
  agent=critic,
  async_execution=True,
  context=[simple_research_task, write_task],
  output_file='reviewed-results/fixed-verse-code-example.txt',
  callback=reviewed_code_written
)

crew = Crew(
  agents=[researcher, developer, critic],
  tasks=[simple_research_task, complex_research_task, write_task, review_task],
  process=Process.sequential 
)

result = crew.kickoff()
print(result)

the error:

> Finished chain.
...
...   some output from the callback is here
...
> Entering new CrewAgentExecutor chain...
Exception in thread Thread-41 (_execute):
Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/envs/conda_env/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/home/ubuntu/miniconda3/envs/conda_env/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/miniconda3/envs/conda_env/lib/python3.10/site-packages/crewai/task.py", line 157, in _execute
    result = agent.execute_task(
  File "/home/ubuntu/miniconda3/envs/conda_env/lib/python3.10/site-packages/crewai/agent.py", line 193, in execute_task
    result = self.agent_executor.invoke(
  File "/home/ubuntu/miniconda3/envs/conda_env/lib/python3.10/site-packages/langchain/chains/base.py", line 163, in invoke
    raise e
  File "/home/ubuntu/miniconda3/envs/conda_env/lib/python3.10/site-packages/langchain/chains/base.py", line 153, in invoke
    self._call(inputs, run_manager=run_manager)
  File "/home/ubuntu/miniconda3/envs/conda_env/lib/python3.10/site-packages/crewai/agents/executor.py", line 64, in _call
    next_step_output = self._take_next_step(
  File "/home/ubuntu/miniconda3/envs/conda_env/lib/python3.10/site-packages/langchain/agents/agent.py", line 1138, in _take_next_step
    [
  File "/home/ubuntu/miniconda3/envs/conda_env/lib/python3.10/site-packages/langchain/agents/agent.py", line 1138, in <listcomp>
    [
  File "/home/ubuntu/miniconda3/envs/conda_env/lib/python3.10/site-packages/crewai/agents/executor.py", line 118, in _iter_next_step
    output = self.agent.plan(
  File "/home/ubuntu/miniconda3/envs/conda_env/lib/python3.10/site-packages/langchain/agents/agent.py", line 397, in plan
    for chunk in self.runnable.stream(inputs, config={"callbacks": callbacks}):
  File "/home/ubuntu/miniconda3/envs/conda_env/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 2685, in stream
    yield from self.transform(iter([input]), config, **kwargs)
  File "/home/ubuntu/miniconda3/envs/conda_env/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 2672, in transform
    yield from self._transform_stream_with_config(
  File "/home/ubuntu/miniconda3/envs/conda_env/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 1743, in _transform_stream_with_config
    chunk: Output = context.run(next, iterator)  # type: ignore
  File "/home/ubuntu/miniconda3/envs/conda_env/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 2636, in _transform
    for output in final_pipeline:
  File "/home/ubuntu/miniconda3/envs/conda_env/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 1209, in transform
    for chunk in input:
  File "/home/ubuntu/miniconda3/envs/conda_env/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 4532, in transform
    yield from self.bound.transform(
  File "/home/ubuntu/miniconda3/envs/conda_env/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 1209, in transform
    for chunk in input:
  File "/home/ubuntu/miniconda3/envs/conda_env/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 1209, in transform
    for chunk in input:
  File "/home/ubuntu/miniconda3/envs/conda_env/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 3058, in transform
    yield from self._transform_stream_with_config(
  File "/home/ubuntu/miniconda3/envs/conda_env/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 1743, in _transform_stream_with_config
    chunk: Output = context.run(next, iterator)  # type: ignore
  File "/home/ubuntu/miniconda3/envs/conda_env/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 3031, in _transform
    futures = {
  File "/home/ubuntu/miniconda3/envs/conda_env/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 3032, in <dictcomp>
    executor.submit(next, generator): (step_name, generator)
  File "/home/ubuntu/miniconda3/envs/conda_env/lib/python3.10/site-packages/langchain_core/runnables/config.py", line 452, in submit
    return super().submit(
  File "/home/ubuntu/miniconda3/envs/conda_env/lib/python3.10/concurrent/futures/thread.py", line 169, in submit
    raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown

Mar 22 '24 02:03 aliensouls

does it repopulate all RAG by a directory with each next step? it seems very long list of operations as if it populates all from scratch, and the folder has many files, about 20-30 in each folder, and each file is between 1 and 5 pages long approximately. if it reimports them all into the db, it's another bug report then 😁 🚀

Mar 22 '24 02:03 aliensouls

versions: Python 3.10.13

crewai                    0.22.5                   pypi_0    pypi
crewai-tools              0.0.15                   pypi_0    pypi

Mar 22 '24 02:03 aliensouls

i turned off "async_execution=False" and this error doesn't show up 🎉 . but the crew completes task once and does not proceed to 'research', I guess it was naive to assume that they're AGI agents. what would be the best way to make them iterate over documentation pages and summarize it in a way that core concept could later be provided in full into gpt4 32k context window? I'm trying to make them iterate over docs and write short format of what they learned, into new files, and then possibly use that as proper in-context language reference (i'm dealing with new language that's not yet in any of openai models or claude models, so they're supposed to somehow learn and summarize, and only then i'll instruct them to make proper data set format from that summary and will try to fine-tune a new model).

Mar 22 '24 02:03 aliensouls

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

Aug 20 '24 10:08 github-actions[bot]

This issue was closed because it has been stalled for 5 days with no activity.

Aug 25 '24 12:08 github-actions[bot]