Vllm or Huggingface for local LLMs for CrewAI

[rajeshkochi444]

Would it be possible for us to use Huggingface or vLLM for loading models locally. Ollama implantation bit more challenging

[tridungduong-unsw]

I have the same request features.

[FBR65]

I have the same request features.

[Atakey]

I have the same request features.

[FBR65]

Hi,

my solution on this request:

from langchain_openai import ChatOpenAI as LOAI

os.environ["OPENAI_API_KEY"] = "NA"

llm_model = LOAI(base_url="http://your_endpoint/v1", model_name = 'your_model')

your_agent = Agent(
    role="Your Role",
    goal="Your goal",
    backstory="Your backstory",
    async_execution=False,
    allow_delegation = False,
    llm=llm_model
)

or setting the environment varaibles:

os.environ["OPENAI_API_KEY"] = "NA"
os.environ["OPENAI_API_BASE"] = "Your Endpoint"
os.environ["OPENAI_MODEL_NAME"] = "Your Model"

I'm using the first solution because of the ability to use several models in my crew. The second is for those who only need one model.

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.

[MohamedAliRashad]

This solution is giving me this

Provider List: https://docs.litellm.ai/docs/providers

ERROR:root:Failed to get supported params: argument of type 'NoneType' is not iterable

[ruianlc]

same situation here. anyone has solved it?

[kellyaa]

You'll want to preface your model name with the provider.

I.e. for VLLM and llama3 it would be: model="hosted_vllm/meta-llama/Llama-3.3-70B-Instruct"

[Pranjal-Tripathi-01]

You'll want to preface your model name with the provider.

I.e. for VLLM and llama3 it would be: model="hosted_vllm/meta-llama/Llama-3.3-70B-Instruct"

I was working with vLLM hosted model and adding prefix "hosted_vllm/" worked for me. Thanks @kellyaa