exprotobuf icon indicating copy to clipboard operation
exprotobuf copied to clipboard

Very large proto file takes a long time to compile

Open fireyang opened this issue 8 years ago • 6 comments

Contains a large number of message proto file, compiled very slowly

fireyang avatar Apr 15 '17 07:04 fireyang

Do you have an example file I can test with? How many messages are defined in the proto?

bitwalker avatar Apr 19 '17 16:04 bitwalker

https://github.com/fireyang/exprotobuf_performance
40 message and 3 enum; Add a new message every time recompile all

fireyang avatar May 03 '17 09:05 fireyang

Thanks! I'll take a look as soon as possible

bitwalker avatar May 04 '17 00:05 bitwalker

similar experience here w/ https://gist.github.com/dweinstein/5c2fa9a197974ee0fcf766e67d683854

dweinstein avatar May 09 '17 21:05 dweinstein

It looks like the bulk of the time is split between gpb parsing and the Elixir compiler - ultimately there isn't much I can do to ease this with the current architecture.

I'm not sure if there is anyway to generate the Elixir source such that compilation is meaningfully faster (at least from a clean state), however one approach would be to generate Elixir source files, and which are then subsequently compiled. This is how one consumes gpb directly, and I am interested in going that route, because it would be much easier to ensure that only those things which actually changed are recompiled.

I'm considering a rewrite of exprotobuf for that reason, amongst others, but time is elusive, so I haven't been able to take a good crack at it yet.

bitwalker avatar Jun 30 '17 21:06 bitwalker

@bitwalker Would you please consider to use enif_protobuf? It looks it is at least 3 times faster than gbp

mustafaturan avatar May 12 '18 17:05 mustafaturan