vercel
Support fileId for ImagePart
Feature Description
I am adding the ability to attach images (and hopefully non images) to my app, in which some of this data will be confidential so public URLs won't work. The issue I am running into now is, I am storing the chat history in a dynamo db table which has a pretty low limit that images and files quickly exceed so I am storing the images in an s3 bucket. This is proving to be more challenging, because I have to pull out the base64 string, and replace it with the fileId before updating the db. I am on a stateless server, so I am storing the fileId in the aiState.
I am wanting to store the fileId (or any other field(s)) that make sense, so that I can store it in dynamoDb.
Or if there is another, better way, I am all ears!
Use Case
Storing files in a different location than the chat history.
Additional context
No response
A few ideas:
a) can you store your files in an S3 bucket and keep them as URLs (that point to S3) in the messages? then you could have a resolution step before calling say generateText (or a different function), in which you would get all the s3 urls from the messages, download the images (with correct authentication), and replace the urls in the messages with the base 64 images
b) if you want to identify the files, another option could be hashing. you could create a sufficiently unique hash over the content (with a super low chance of collisions), and make this the filename. Again you could store this in a URI and then do the resolution. However, this solutions seems more complicated than a)
@lgrammel How would this work with displaying them on the client side?
Yup, currently I'm storing the url in dynamoDB, and before needing the image, reading it from s3. My main issue at the moment is constant re-loadings of the image. Right now, onGetUIState gets called multiple times (which I have to reload the image), but it doesn't remember that I set the image on the last time... so it makes another db call again. Any idea of how to set it so that it gets the base64 images once and the aiState knows about it from the rest of the app?
I was thinking of doing it on (chat)/chat/[id].page.tsx, but that gets called multiple times within nextjs for some reason, resulting in retrieving the image even more times lol.
I am also interested in decoupling image part from message history. My problem extends this further in that I will now be duplicating and worse into LangFuse for all multi-modal events. It would be great if there was a way to do a just-in-time field resolution callback on the front and back end that would let me resolve the full heavy image only when making calls to the llm itself or on the frontend to resolve with lazy load. I am going to dig into the code to see if this is something I can maybe contribute. I expect with the recent LangFuse integration that many will appreciate their images not getting duplicated into every trace.
