Chat History Pipeline

[ddematheu]

As chat histories get longer, passing the entire history on every call is not a good practice. More so, user expects information from several messages ago to be available as context.

Goal: Improve size of the chat history context window to allow users to reference messages that fall outside existing window.

Solution: Leverage semantic search to index the entire chat history of a conversation and pull messages that are related to the latest message from the user.

Implementation:

• Create a pseudo-Pipeline object that uses a custom source connector that simply bypasses messages written to it into a vector database.
• Pipeline is declared with an Embed Connector and Sink Connector to be used as part of the operation.
• At search we would run a normal search against the sink with filters to only pull messages from the given conversation.
• Then the user would add the retrieved messages as context into the conversation alongside the last 3-4 messages

Prototyped: https://github.com/NeumTry/Pensieve

Other ideas:

• Any chat systems that are worth integrating? (Twilio?)