Feature/externalize files from chatflow - do not save as base64

Open vinodkiran opened this issue 1 year ago • 2 comments

Currently Flowise saves all uploaded docs as base64 string to the chatflow. It is not scalable with large documents. Issue - https://github.com/FlowiseAI/Flowise/issues/1247

Solution: save the files to a local path and only save the filepath to the chatflow.

This PR extracts the base64 string and saves it to the path specified in env.BLOB_STORAGE_PATH (creates a sub-folder with the chatflowid)

This is backwards compatible, when a current chatflow (with base64 in flowData) is opened and saved, the files are extracted and saved.

Mar 17 '24 16:03 vinodkiran

if you do a search type: 'file', there are other places that allow uploading files as well

Mar 18 '24 08:03 HenryHengZJ

if you do a search type: 'file', there are other places that allow uploading files as well

Yes, working on the rest of the item. Draft PR to ensure that the loaders work as expected with this change.

Mar 19 '24 06:03 vinodkiran

All nodes with type: 'file' are refactored, ready for review and merge.

Mar 25 '24 07:03 vinodkiran