akiradeveloper
Split into low and high level parser
I don't know if you want this, but it could be interesting to split msgpack-nim into a low level parser (like parsexml) and high level parser (like xmlparser). Right now with big msgpack objects there is a lot of overhead in heap and string allocations. This could be reduced significantly by using the lower level parser. See the example usage at the bottom of parsexml.
Right now the performance of msgpack-nim is about on par with Python's. With some shallow strings and seqs it could be doubled, but no way near a real low level parser that doesn't allocate any heap objects, but walks through the msgpack peace by peace.
I am interesting in improving performance but I am not sure what you mean by a word "low level".
What example usage of parsexml are you pointing at? I read the main routine of the following source code but couldn't get anything about low level. https://github.com/Araq/Nim/blob/master/lib/pure/parsexml.nim
Is low-level parser able to improve both packing and unpacking?
If the stream is always the same size and type, the resulting object can be reused to store the result which can avoid allocations. Do you mean this?
or do you mean to create iterator style unpacking?
I believe you are using https://github.com/msgpack/msgpack-python as the python implementation.
At the first look, it is almost written in C for performance. So, I think par isn't bad. How do you think?
Anyway, I will port the benchmark to msgpack-nim.
Yes, I used msgpack-python for comparison. Sure, the performance is not bad, but I think it could be improved by a factor of 4 or more. I'm pretty happy with msgpack-nim. I could unpack all my ~20 MB of real world data immediately.
I guess I wasn't clear when I wrote this, but what I mean with "low level" is iterator style unpacking.
If the stream is always the same size and type, the resulting object can be reused to store the result which can avoid allocations. Do you mean this?
Yes.