versatile-data-kit icon indicating copy to clipboard operation
versatile-data-kit copied to clipboard

Efficiently handle new data

Open murphp15 opened this issue 2 years ago • 0 comments

What is the feature request? What problem does it solve? We want to scrape jira or confluence on a periodic interval. However we want to make sure that we don't republish all content to the vector store and instead only publish the latest changed data.

Defintion of done After a first scrape of the datasource is done, every successive scrape should update the minimum amount of rows.

murphp15 avatar Jan 08 '24 12:01 murphp15