Specify protobuf version when compiling from source

[jchwenger]

Hi,

I'm working on training a model using sentencepiece for the decoding part, and I encounter conflicts when trying to perform distributed training with Horovod.

I have posted this and have been working on resolving the issue.

The situation is the following:

1. When finetuning openAI's gpt-2 on one GPU, with TF 1.14, with or without sentencepiece, things work.
2. I can train with Horovod without sentencepiece, and when I try integrating sentencepiece I get the following error:

[1,1]<stderr>:[libprotobuf FATAL external/protobuf_archive/src/google/protobuf/stubs/common.cc:86] This program was compiled against version 3.6.1 of the Protocol Buffer runtime library, which is not compatible with the installed version (3.7.1).  Contact the program author for an update.  If you compiled the program yourself, make sure that your headers are from the same version of Protocol Buffers as your link-time library.  (Version verification failed in "/sentencepiece/src/builtin_pb/sentencepiece_model.pb.cc".)
[1,0]<stderr>:[libprotobuf FATAL external/protobuf_archive/src/google/protobuf/stubs/common.cc:86] This program was compiled against version 3.6.1 of the Protocol Buffer runtime library, which is not compatible with the installed version (3.7.1).  Contact the program author for an update.  If you compiled the program yourself, make sure that your headers are from the same version of Protocol Buffers as your link-time library.  (Version verification failed in "/sentencepiece/src/builtin_pb/sentencepiece_model.pb.cc".)
[1,1]<stderr>:terminate called after throwing an instance of 'google::protobuf::FatalException'
[1,1]<stderr>:  what():  This program was compiled against version 3.6.1 of the Protocol Buffer runtime library, which is not compatible with the installed version (3.7.1).  Contact the program author for an update.  If you compiled the program yourself, make sure that your headers are from the same version of Protocol Buffers as your link-time library.  (Version verification failed in "/sentencepiece/src/builtin_pb/sentencepiece_model.pb.cc".)

This seems very odd as this does not appear without horovod.

As suggested (and following this this), I compiled protobuf from source and user-wide to version 3.7.1, and attempted to compile sentencepiece with the flag -DSPM_USE_BUILTIN_PROTOBUF=OFF . However, that failed, and for now I went back to the latest version of protobuf and the standard sentencepiece install (from source).

(I just tried to downgrade to TF 1.13 but 1) it seems much slower than 1.14 for some reason, 2) Horovod throws another error. that does not appear with 1.14)

Am I on the right track trying to compile sentencepiece with protobuf 3.7.1, and is there a way I could get some guidance on how to achieve this? Also, should it not be better to specify the version of the built-in protobuf version, if possible, rather than install this specific version user-wide (as per the discussion here) and if so what would be the best way to proceed?

Thanks very much for your help.

sentencepiece: 0.1.86 protoc 3.11.4