lxning

This feature requires 1. define inference messages in proto 2. apply protobuf encode/decode in inference endpoints 3. add test cases

@MHatemAbdelhamid The issue reported in this ticket is the workerthread hang and never got chance to recreate a new workerthread. Your case is different since a new workerthread is created....

@MHatemAbdelhamid According to your description ("the error only happens on large number of users, suddenly when the number of users increases"). It seems it is a capacity issue. I suggest...

@hgong-snap I tried the steps you provided. There is no exception or died workerthread in the [ts_log.log.zip](https://github.com/pytorch/serve/files/9589837/ts_log.log.zip). Message "grpc client call already cancelled" was logged when TS was trying to...

@hgong-snap Thank you for recording the workflow, I can see the following exception in the video. ``` io.grpc.StatusRuntimeException: CANCELLED: call already cancelled at io.grpc.Status.asRuntimeException(Status.java:524) ~[model-server.jar:?] at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onNext(ServerCalls.java:335) ~[model-server.jar:?] at org.pytorch.serve.job.GRPCJob.response(GRPCJob.java:66)...

@hgong-snap I verified master branch on both local host and docker nightly build. You can fetch torchserve nightly build at https://hub.docker.com/r/pytorch/torchserve-nightly/tags. - docker pull pytorch/torchserve-nightly:latest-gpu - docker pull pytorch/torchserve-nightly:latest-cpu

@hgong-snap gpu docker image should specify cuda version. For example: ./build_image.sh -bt dev -g -cv cu102 [Here](https://github.com/pytorch/serve/tree/master/docker) is the detail information about torchserve docker image build. Could you please directly...

@hgong-snap Great, thank you for the verification.

@Chichilele I tested the docker prod, it works fine. Here are the steps what i did. - cd docker - mkdir model_store - cd model_store - wget https://torchserve.pytorch.org/mar_files/mnist_v2.mar - ./build_image.sh...

@Chichilele I still can not reproduce your case. And also I don't understand "+X" mode prevents you to start container.