[Idea] Vector DBs to extend dataset sizes

[ethanabowen]

Love this tool! A limitation that I've run across is the token limits of LLMs when working with real-world large datasets.

I'd love to be able to point this tool to a Vector DB to extend the amount of data being worked on.

What do we think? Would this really solve the problem I've facing with token limits or is there a general limitation to LIDA because of LLM token sizes?

[ethanabowen]

After more insight into the code, it looks like the limitation I was facing was based on the Summarization of datasets with many many columns. Still a limitation that I'd be interested in overcoming.

[lawyinking]

do you have any solutions or suggestions now?