dream-encoding icon indicating copy to clipboard operation
dream-encoding copied to clipboard

Appears not to support streaming

Open aantron opened this issue 4 years ago • 2 comments

e.g.:

https://github.com/tmattio/dream-encoding/blob/dde420426ba98da2bc7d26b0d67c3f0f2eb7b7ab/lib/dream_encoding.ml#L194-L195

This appears to read the entire body into memory, and then compress it. I assume (perhaps wrongly) that the underlying compression algorithms support streaming, and it seems like streaming would be more beneficial for precisely the kind of responses that you wouldn't want to load into memory an extra time (large ones).

aantron avatar Jul 05 '21 06:07 aantron

assume (perhaps wrongly) that the underlying compression algorithms support streaming

It does! In fact, the compression function (which is almost identical to decompress's documentation example) creates a stream from the given string, so refactoring this to work on stream responses should be trivial 🤞

I'll get to it as soon as I have some spare time

tmattio avatar Jul 05 '21 13:07 tmattio

I'll get to it as soon as I have some spare time

No rush! And I think you will likely need some improvements in Dream's streaming APIs to get it done, so do bug me :)

aantron avatar Jul 05 '21 14:07 aantron