Draft: Consolidate External Service Plugins and Refactor code

[jayeshp19]

Proposal: Consolidate External Service Plugins

Currently we have to manage separate plugins for all the external services. This is a draft PR for removing the need to manage all plugins separately. Instead, have all the code in a single plugin without importing extra packages. and it also simplifies the file structure

• current installation

pip install livekit-agents livekit-plugins-openai livekit-plugins-deepgram

• proposed installation

pip install livekit-agents openai

we just need to install live-agents and external model providers as required

try:
    import openai
    from openai.types.chat import ChatCompletionChunk, ChatCompletionMessageParam
    from openai.types.chat.chat_completion_chunk import Choice
except ImportError:
    logger.error("OpenAI API not installed. Please install it using `pip install openai`")
    raise ImportError

• current import method

from livekit.plugins import deepgram, openai, silero

    agent = VoicePipelineAgent(
        vad=silero.VAD.load(),
        stt=deepgram.STT(model=dg_model),
        llm=openai.LLM(),
        tts=openai.TTS(),
        chat_ctx=initial_ctx,
    )

• proposed import method

from livekit.agents.llm import OpenAI
from livekit.agents.stt import DeepGramSTT
from livekit.agents.tts import OpenAITTS
from livekit.agents.vad import Silero

    agent = VoicePipelineAgent(
        vad=Silero.load(),
        stt=DeepgramSTT(model=dg_model),
        llm=OpenAI(),
        tts=OpenAITTS(),
        chat_ctx=initial_ctx,
    )

Proposed File Structure

agents
├── llm
│   ├── openai
│   ├── anthropic
│   └── ... other providers
├── stt
│   ├── deepgram
│   ├── openai
│   ├── google
│   └── ... other providers
├── tts
│   ├── elevenlabs
│   ├── openai
│   └── ... other providers
├── embedder
│   ├── openai
│   └── ... other providers

--- future ideas ---
├── tools
│   ├── slack
│   ├── email
│   ├── crm
│   ├── Tavily (search api)
│   └── ... other tools

with current file structure, we will need to manage plugin separately for each tool we add in future and existing plugins aswell. I think this new structure keeps each service organized within categories and makes future additions easier with just file or two files.

Progress

For the initial MVP, I’ve tested with OpenAI and Silero and below is the implementation progress of current plugins. end-to-end testing is remaining. If we proceed with this proposal, I’ll continue development. Please let me know if you have any questions.

• [x] Openai
• [ ] OpenAI realtime
• [x] Anthropic
• [x] DeepGram
• [x] Elevanlabs
• [x] Google
• [x] Silero
• [x] Azure
• [x] Cartesia
• [ ] Clova
• [ ] llamaindex
• [ ] Browser
• [ ] nltk
• [ ] PlayHT
• [ ] RAG

https://github.com/jayeshp19/agents/tree/refactor-code

@davidzhao @theomonnom I’d love to know your thoughts

[changeset-bot[bot]]

⚠️ No Changeset found

Latest commit: 8ddbf7b691eabea2e7047842586446149c98fcda

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR