Does compressor update dictionary?
Sorry for using issues for questions, but the generic forum has no ideas.
Does ZstdCompressor with dictionary add new data to dictionary when it runs compress() or is dictionary only limited to data that was used for its creation? In other words, if I build a dictionary from A, and then use it to compress B and C with .compress(), will it help the compressor to find similarities between B and C ?
Use case: I receive a big set of small (4kb-16kb) pieces of data from a remote device. They may or may not share a lot of similar sequences between themselves. I look for a way to compress them on the fly, without first storing them raw. I need to be able to decompress any piece by itself, so streaming API can't help me because of all this tails and flush() stuff.
Thanks for the help!
Dictionaries are immutable once created: compress() will not update a dictionary.
Note that you can use arbitrary content as a dictionary via the DICT_TYPE_RAWCONTENT constant. This is as opposed to first training a dictionary. So if you have a series of discrete elements, you could for example use the raw value of the last N elements as the "raw dictionary" data. The README has an example of this. Search for decompress_content_dict_chain.