flower icon indicating copy to clipboard operation
flower copied to clipboard

_MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:

Open Saugatkafley opened this issue 4 years ago • 3 comments

Describe the bug

I am using flower in Colab , starting the Client creates `_MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "failed to connect to all addresses" debug_error_string = "{"created":"@1644417421.199890350","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3134,"referenced_errors":[{"created":"@1644417421.199889433","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}"

`

Steps/Code to Reproduce

fl.client.start_numpy_client("localhost:8080", client=MathClients())

Expected Results

Something like this INFO flower 2021-02-25 14:15:46,741 | app.py:76 | Flower server running (insecure, 3 rounds) INFO flower 2021-02-25 14:15:46,742 | server.py:72 | Getting initial parameters INFO flower 2021-02-25 14:16:01,770 | server.py:74 | Evaluating initial parameters INFO flower 2021-02-25 14:16:01,770 | server.py:87 | [TIME] FL starting DEBUG flower 2021-02-25 14:16:12,341 | server.py:165 | fit_round: strategy sampled 2 clients (out of 2) DEBUG flower 2021-02-25 14:21:17,235 | server.py:177 | fit_round received 2 results and 0 failures

Actual Results

DEBUG flower 2022-02-09 14:37:01,188 | connection.py:36 | ChannelConnectivity.IDLE
INFO flower 2022-02-09 14:37:01,196 | app.py:61 | Opened (insecure) gRPC connection
DEBUG flower 2022-02-09 14:37:01,201 | connection.py:36 | ChannelConnectivity.TRANSIENT_FAILURE
DEBUG flower 2022-02-09 14:37:01,402 | connection.py:68 | Insecure gRPC channel closed
---------------------------------------------------------------------------
_MultiThreadedRendezvous                  Traceback (most recent call last)
[<ipython-input-35-f56f4dd630ed>](https://localhost:8080/#) in <module>()
----> 1 fl.client.start_numpy_client("localhost:8080", client=MathClients())

4 frames
[/usr/local/lib/python3.7/dist-packages/grpc/_channel.py](https://localhost:8080/#) in _next(self)
    824                     raise StopIteration()
    825                 elif self._state.code is not None:
--> 826                     raise self
    827 
    828 

_MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses"

Saugatkafley avatar Feb 09 '22 14:02 Saugatkafley

I have the same issue! Is there a way out?

shg550 avatar Aug 28 '22 10:08 shg550

I think it was an issue of Remote Procedure Call(RPC) .. I didn't figure it out . So i simulated my own federated environment . Flower works in distributed fashion ,it may be that it cannot connect to other clients.

Saugatkafley avatar Aug 28 '22 12:08 Saugatkafley

Any update on this?

I have the same issue running "quickstart-pytorch" code https://flower.dev/docs/quickstart-pytorch.html#flower-server

Daiknow avatar Sep 22 '22 20:09 Daiknow