[core] GcsSubscriber hangs in shutdown if the connection broke on MacOS

[rynewang]

What happened + What you expected to happen

In #45029 we disabled the logger threads on MacOS for test_unavailable_actors.py. This is because the logger threads hangs in a poll in GcsSubscriber. Under the hood it's a simple gRPC call (not streaming), and the GCS server should already replied the message. It's not clear if this is some gRPC bug and if it manifests in real world use cases.

Versions / Dependencies

master

Reproduction script

On MacOS, change test_unavailable_actors.py by removing the decorator for log_to_driver=False and run many times. Eventually you get the test hanged.

Issue Severity

Low: It annoys or frustrates me.

[rynewang]

cc @jjyao