grpc icon indicating copy to clipboard operation
grpc copied to clipboard

Grpc Server crash

Open Trisfald opened this issue 5 years ago • 3 comments

Hello, I'm trying to include this library in a suit of grpc benchmarks.

I have an issue with the server, when I send requests from a ghz client it fails with this error:

E0805 16:39:28.639090500       1 byte_buffer.cc:91]          Should never reach here.

 *** caught segfault ***
address (nil), cause 'unknown'

Traceback:
 1: run(server_functions, channel, hooks)
 2: start_server(impl, "0.0.0.0:50051")
An irrecoverable exception occurred. R is aborting now ...

It's possible to reproduce this on a linux or macos machine by just cloning this branch: https://github.com/LesnyRumcajs/grpc_bench/tree/r-bench

(the relevant code is inside the r_grpc directory)

Build the docker image with the server docker build --force-rm -f r_grpc/Dockerfile -t r_grpc_test .

Start the server docker run --rm -it --network host r_grpc_test

Start the client docker run --rm --network=host -v "${PWD}"/proto:/proto:ro --entrypoint=ghz infoblox/ghz:0.0.1 --proto=/proto/helloworld/helloworld.proto --call=helloworld.Greeter.SayHello --insecure --connections=5 --duration 5s -d "{\"name\":\"it's not as performant as we expected\"}" 127.0.0.1:50051

output of the client:

Status code distribution:
  [OK]            4 responses        
  [Unavailable]   273676 responses   
  [Canceled]      41 responses 

I may be doing some silly mistake in the server implementation.

Trisfald avatar Aug 05 '20 16:08 Trisfald

Thank you for reporting this issue.

It's interesting that you get a few OKs and then a lot of Unavailables.

Here are my two guesses about what it could be:

  1. library versions

This package was originally built against grpc 1.3/1.4, and hadn't really been updated since. We now have a GSoC student who is updating it to use the Ubuntu 20.04 versions of libgrpc, but I'll also check that it works against the current versions in Debian testing.

  1. Server in bad state

The original release of this R package has a server that will terminate the connection after firing the R callback for a (non-streaming) service. If, in your benchmark client, you connect and make sequential calls without reconnecting in between, it probably wouldn't work yet.

Also, if you find anything in your benchmarking, we'd be interested in the results.

nfultz avatar Aug 05 '20 18:08 nfultz

Thanks for the answer.

Currently the benchmark client is opening 5 connections and in each connection it performs 10 requests in parallel. Can this be the problem?

Trisfald avatar Aug 05 '20 19:08 Trisfald

It might be. If you try it with 50 connections and 1 req per connection, you might see a different distribution of success / failures.

On Wed, Aug 5, 2020 at 12:41 PM Andrea [email protected] wrote:

Thanks for the answer.

Currently the benchmark client is opening 5 connections and in each connection it performs 10 requests in parallel. Can this be the problem?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/nfultz/grpc/issues/35#issuecomment-669448297, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADGGTUYWKGTEDTPOFQXS5TR7GYVBANCNFSM4PVWFHRQ .

nfultz avatar Aug 05 '20 23:08 nfultz