Support Local Language Model (LLM) with API

[Zakarialabib]

Subject: Support Local Language Model (LLM) with API

I hope this message finds you well. I am writing to bring attention to an issue I have encountered while working with the project, specifically regarding the support for the Local Language Model (LLM) through the API.

Issue Overview: The current version which integrates with the OpenAI API, appears to lack support for using the Local Language Model (LLM) locally through the API.

Expected Behavior: Ideally, users should be able to seamlessly interact with the Local Language Model (LLM) through the OpenAI API, like overiding the api url , in this context the api key isnt required.

Thank you for your attention to this matter. I am happy to provide any additional information or clarification if needed.

Best regards,

[TheBitmonkey]

try pop this at the top of your example for LM Studio...

os.environ["OPENAI_API_BASE"]="http://localhost:1234/v1"

[sirajperson]

It already supports local LLMs via langchain integration. When creating the agent, simply set the llm to langchain's OpenAI interface and set the api base while creating the object. In the following example I've loaded a model using text generation webui and enabled the openai ai extension and set the API port to 5000 on my localhost. Here's the example code:

import os
from crewai import Agent, Task, Crew
from langchain_community.chat_models import ChatOpenAI
from langchain.tools import DuckDuckGoSearchRun

os.environ["OPENAI_API_KEY"] = "Null"
openai_api_base = "http://127.0.0.1:5000/v1"
llm = ChatOpenAI(model_name="gpt-3.5", temperature=0.7, openai_api_base=openai_api_base)
search_tool = DuckDuckGoSearchRun()

researcher = Agent(
    role='Senior Research Analyst',
    goal='Uncover cutting-edge developments in AI and data science',
    backstory="""You work at a leading tech think tank.
  Your expertise lies in identifying emerging trends.
  You have a knack for dissecting complex data and presenting
  actionable insights.""",
    verbose=True,
    allow_delegation=False,
    tools=[search_tool],
    llm=llm
)
writer = Agent(
    role='Tech Content Strategist',
    goal='Craft compelling content on tech advancements',
    backstory="""You are a renowned Content Strategist, known for
  your insightful and engaging articles.
  You transform complex concepts into compelling narratives.""",
    verbose=True,
    allow_delegation=True,
    llm=llm
)
task1 = Task(
    description="""Conduct a comprehensive analysis of the latest advancements in AI in 2024.
  Identify key trends, breakthrough technologies, and potential industry impacts.
  Your final answer MUST be a full analysis report""",
    agent=researcher
)

task2 = Task(
    description="""Using the insights provided, develop an engaging blog
  post that highlights the most significant AI advancements.
  Your post should be informative yet accessible, catering to a tech-savvy audience.
  Make it sound cool, avoid complex words so it doesn't sound like AI.
  Your final answer MUST be the full blog post of at least 4 paragraphs.""",
    agent=writer
)
crew = Crew(
    agents=[researcher, writer],
    tasks=[task1, task2],
    verbose=2,  # You can set it to 1 or 2 to different logging levels
)
result = crew.kickoff()

print("######################")
print(result)

[Zakarialabib]

Thank you guyz, it worked 👍🏼

[M1ch43l-D]

Thank you all for this!