[Feature Request] Indexing existing data sources

[rosieks]

Context / Scenario

Common use case of using AI is to provide answer based on given existing knowledge base (it might be some company wiki, git repository, system database)

The problem

Right now the only way to fill kernel memory is by importing all documents one by one.

Proposed solution

It would be nice to have some mechanism to synchronize kernel memory with external data source (eg. azure blob storage, git repository, wiki, sharepoint). The idea is to provide abstraction so people could use it to implement their own data sources. Concepts that it has to provide is:

• Initial data feed
• Data synchronization (polling or listening - depending of data source possibilities)

Importance

None