distributed icon indicating copy to clipboard operation
distributed copied to clipboard

Allow scheduler to preserve worker hostnames

Open bbockelm opened this issue 5 years ago • 2 comments

This provides a configuration option allowing the scheduler to preserve worker hostnames (as opposed to the current & default behavior of resolving hostnames immediately to IP address).

Preserving the hostname should convey advantages in two situations:

  1. Dual-network setups where some hosts have IPv4-only or IPv6-only. This allows the host that wants to communicate with the worker to determine the best network to route over (as opposed to the scheduler).
  2. Clusters that route TLS connections based on SNI. If the hostname is kept internally, it will be pased to OpenSSL when making a connection. This allows the OpenSSL client to embed the desired hostname in a SNI header in the TLS handshake. If the worker is behind a TLS-router (not entirely uncommon in a Kubernetes setup), then this SNI is required to successfully route the request.

Closes #4070

bbockelm avatar Sep 14 '20 02:09 bbockelm

Also cc'ing @jcrist who might have experience here to suggest a default choice.

mrocklin avatar Sep 15 '20 03:09 mrocklin

@bbockelm I opened https://github.com/dask/distributed/pull/4938 to replace your PR since it has conflicts and it was not anymore easy to rebase on master. I took patch directly from our distributed.git fork.

oshadura avatar Jun 20 '21 14:06 oshadura