distributed
distributed copied to clipboard
Allow scheduler to preserve worker hostnames
This provides a configuration option allowing the scheduler to preserve worker hostnames (as opposed to the current & default behavior of resolving hostnames immediately to IP address).
Preserving the hostname should convey advantages in two situations:
- Dual-network setups where some hosts have IPv4-only or IPv6-only. This allows the host that wants to communicate with the worker to determine the best network to route over (as opposed to the scheduler).
- Clusters that route TLS connections based on SNI. If the hostname is kept internally, it will be pased to OpenSSL when making a connection. This allows the OpenSSL client to embed the desired hostname in a SNI header in the TLS handshake. If the worker is behind a TLS-router (not entirely uncommon in a Kubernetes setup), then this SNI is required to successfully route the request.
Closes #4070
Also cc'ing @jcrist who might have experience here to suggest a default choice.
@bbockelm I opened https://github.com/dask/distributed/pull/4938 to replace your PR since it has conflicts and it was not anymore easy to rebase on master. I took patch directly from our distributed.git fork.