Can we merge all dictionaries into single compressed stream?

[arai-a]

Just an idea for the next format.

Currently prelude and content sections have multiple small streams (1 stream for each dictionary), and when decoding the file, decoder has to allocate buffer for each stream. also, currently there's no information about uncompressed size for brotli stream, afaik. That means decoder has to do allocation/reallocation multiple times while decoding prelude/content sections.

If there's no merit for compressing each stream, using single stream can make the decoder simpler. Also, adding uncompressed size might make it simpler (of course we need to validate the size before allocation tho)

[Yoric]

I agree, we definitely need to investigate this.

[Yoric]

I'll run a quick test on this.

[Yoric]

My quick test seems to indicate a small decrease in total file size. Unfortunately, I didn't write down a before/after, but it's encouraging.