wasm-tools icon indicating copy to clipboard operation
wasm-tools copied to clipboard

Streaming writing/encoding with io::Write

Open danleh opened this issue 3 years ago • 4 comments

Similar to this prior question on streaming reading with wasmparser, I am wondering if it is possible to do streaming encoding with wasm-encoder?

Right now, the central trait seems to be wasm_encoder::Encode, which takes the sink as a &mut Vec<u8>. Would it be possible to get rid of that trait and instead serialize to any impl io::Write? That should strictly be more general, as io::Write is already implemented in the standard library for Vec<u8> anyway.

Thanks a lot for the great selection of crates!

danleh avatar Sep 28 '22 21:09 danleh

At this time there's no support for using io::Write, it's required to use a Vec<u8>. Would it work for your use case to consider the Vec<u8> a form of BufWriter where it's periodically flushed, e.g. between sections or between functions?

alexcrichton avatar Sep 28 '22 22:09 alexcrichton

Sure, that would work for me. I was more wondering what the reason for the separate Encode trait is, and whether it could be replaced by the standard io::Write.

danleh avatar Sep 30 '22 10:09 danleh

The Encode trait is necessary to indicate "write yourself into this thing with the wasm encoding" and the choice of taking Vec<u8> vs io::Write was mostly a result of me to cut down on the size of compiled and monomorphized code. Dealing with errors in io::Write can cause a lot of code bloat if used everywhere and not much in wasm is so big it can't fit in memory (was my thinking)

alexcrichton avatar Sep 30 '22 14:09 alexcrichton

Thanks a lot, that makes sense and answers my question! I hadn't thought about code size before.

danleh avatar Sep 30 '22 14:09 danleh