protobuf 3.2.0 not working but protobuf 3.1.0.post1 is

[mreinhardt]

Getting the following stack trace error when trying to start up an application that's using riemann-client:

[2017-02-10 22:49:19 +0000] [3567] [INFO] Worker exiting (pid: 3567)
[2017-02-10 22:49:19 +0000] [3568] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/MyCo/my.app/env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 557, in spawn_worker
    worker.init_process()
  File "/MyCo/my.app/env/local/lib/python2.7/site-packages/gunicorn/workers/ggevent.py", line 190, in init_process
    super(GeventWorker, self).init_process()
  File "/MyCo/my.app/env/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 126, in init_process
    self.load_wsgi()
  File "/MyCo/my.app/env/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 136, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/MyCo/my.app/env/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/MyCo/my.app/env/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 65, in load
    return self.load_wsgiapp()
  File "/MyCo/my.app/env/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/MyCo/my.app/env/local/lib/python2.7/site-packages/gunicorn/util.py", line 357, in import_app
    __import__(module)
  File "/MyCo/my.app/env/local/lib/python2.7/site-packages/gevent/builtins.py", line 93, in __import__
    result = _import(*args, **kwargs)
  File "/MyCo/my.app/env/local/lib/python2.7/site-packages/my/app/wsgi.py", line 3, in <module>
    from my.app.app import application, set_logger
  File "/MyCo/my.app/env/local/lib/python2.7/site-packages/gevent/builtins.py", line 93, in __import__
    result = _import(*args, **kwargs)
  File "/MyCo/my.app/env/local/lib/python2.7/site-packages/my/app/app.py", line 16, in <module>
    from my.app.riemann import get_alerting_events, get_events                                                                        
  File "/MyCo/my.app/env/local/lib/python2.7/site-packages/gevent/builtins.py", line 93, in __import__
    result = _import(*args, **kwargs)
  File "/MyCo/my.app/env/local/lib/python2.7/site-packages/my/app/riemann.py", line 7, in <module>
    from riemann_client.client import Client
  File "/MyCo/my.app/env/local/lib/python2.7/site-packages/gevent/builtins.py", line 93, in __import__
    result = _import(*args, **kwargs)
  File "/MyCo/my.app/env/local/lib/python2.7/site-packages/riemann_client/client.py", line 27, in <module>
    import riemann_client.riemann_pb2
  File "/MyCo/my.app/env/local/lib/python2.7/site-packages/gevent/builtins.py", line 93, in __import__
    result = _import(*args, **kwargs)
  File "/MyCo/my.app/env/local/lib/python2.7/site-packages/riemann_client/riemann_pb2.py", line 10, in <module>
    from riemann_client.riemann_pb2_py2 import (Event, Msg, Query, Attribute)
  File "/MyCo/my.app/env/local/lib/python2.7/site-packages/gevent/builtins.py", line 93, in __import__
    result = _import(*args, **kwargs)
  File "/MyCo/my.app/env/local/lib/python2.7/site-packages/riemann_client/riemann_pb2_py2.py", line 32, in <module>
    options=None),
  File "/MyCo/my.app/env/local/lib/python2.7/site-packages/google/protobuf/descriptor.py", line 494, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors should not be created directly, but only retrieved from their parent.

This is happening with protobuf 3.2.0 only. When I manually move the protobuf 3.1.0.post1 site-package from my last build into my app env everything works as expected.

I notice that the wheels are built differently and vary greatly in size as well:

-rw-r--r--   347537 bytes   protobuf-3.1.0.post1-py2.py3-none-any.whl

vs

-rw-r--r--   5618195 bytes   protobuf-3.2.0-cp27-cp27mu-manylinux1_x86_64.whl

Does the riemann-client need to be forced to use protobuf < 3.2 or is there a problem outside the scope of riemann-client that I'm not recognizing?

[mreinhardt]

Note: I was able to get around this by pinning protobuf==3.1.0.post1 in my setup.py

[borntyping]

This was written long before protobuf 3, so unfortunately I don't know what would cause this specific error. I'll look into it if I get a chance, but for the time being I'll gladly accept a PR to depend on rotobuf<=3.1.0.post1 or that fixes the issue.

[mreinhardt]

Thanks, I'll go ahead and make the PR. I'll also see if I can find some other verification of the root cause of the issue and whether it's needed for Python 3 also as we're running 2.7.

[aduston]

More information on the problem here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=835271 . We didn't experience this problem until we upgraded Debian.

It's not at all a pull request, but I was able to "fix" this issue for myself by trivially replacing riemann_pb2.py with the python 3 implementation (since I'm using python 3, I don't care about the python 2 implementation).

[borntyping]

I might not be reading the bug correctly, but it sounds like renaming riemann_pb2_py3 to riemann_py3_pb2 (and same with the python 2 version) might also fix the problem?

[aduston]

I might not be reading the bug correctly, but it sounds like renaming riemann_pb2_py3 to riemann_py3_pb2 (and same with the python 2 version) might also fix the problem?

Reading the underlying change in the protobuf library, I believe this will work.