Support Docker containers in the spark client

Open pschorf opened this issue 7 years ago • 3 comments

Sep 04 '18 16:09 pschorf

Is it possible that the (undocumented) option spark.executor.cook.container [1] could be used to emulate "native" docker support?

https://github.com/twosigma/spark/blob/a72488df9e89d2570a0dbbb80767b398a87e1d8d/resource-managers/cook/src/main/scala/org/apache/spark/scheduler/cluster/cook/CookSchedulerContext.scala#L47

Sep 04 '18 18:09 PerilousApricot

Okay, I started work on this, my goal is to wire the existing mesos/docker config options into Cook as opposed to making cook-specific keys for the ame functinality. Is that acceptable?

Sep 08 '18 00:09 PerilousApricot

Sounds good to me.

Sep 10 '18 13:09 pschorf