Integration with Mesos maintenance mode

[stevenschlansker]

On Mar 14, 2016, at 12:12 PM, Joseph Wu [email protected] wrote:

Managing maintenance is currently up to the operator (you, presumably). If you have something to contribute (code, docs, or examples), that would be greatly appreciated :)

We haven't prioritized other integration (like the CLI or web UI) since maintenance primitives themselves need to be supported by frameworks using the V1 HTTP API; and frameworks developers have not all started/finished migrating to the new API yet. (Without framework support, frameworks will not react to your maintenance windows.)

Here's the tracking ticket for adding maintenance info to the web UI (no work has been done on this yet): https://issues.apache.org/jira/browse/MESOS-2082

[janisz]

Is anybody working on this? It require switch from native mesoslib to HTTP API first.

[ssalinas]

Nobody is actively working on this at the moment. We have discussed the switch to using the HTTP API, but that work has not been started or been scheduled yet. Internally we use Singularity's Decommissioning and Freezing of slaves to provide maintenance windows for us.

[stevenschlansker]

We're running experimental Jenkins and Spark frameworks on our Mesos / Singularity clusters. We use Singularity decommissioning to take hosts out for maintenance.

This means that our other schedulers can still place tasks. This is surprising and disappointing from an operator's perspective who hasn't watched the history of this issue (that it predates maintenance mode entirely).

[yuriy-filatov]

Any updates here? We also need this badly.

[ssalinas]

@yuriy-filatov it wasn't currently in our planned work. By 'need this badly' is the current decommission buggy or causing you issues, or is it just an inconvenience to also have to remember to set mesos maintenance mode manually? I don't have an operator api client in here yet, but if you'd like to PR this, the steps would be:

• Add a client for the operator endpoints on the mesos master (or at least just that one and we can fill out more later)
• Add a call to it on the slave decommission code path
• Add a call to undo this on the reactivate slave code path

[yuriy-filatov]

@ssalinas thanks for your quick answer. Well, we're using Singularity with Mesos 1.6.1 in our Production env. It's using numerous spot fleets with hundreds of instances in AWS, so we usually upscale/downscale racks in accordance with required capacity by moving unneeded slaves into maintenance mode in Mesos in order to fence them from scheduler and frameworks. After some timeout these slaves are terminated, but sometimes singularity still tries to operate with these slaves.

Would be great if mesos maintenance state will be synced with singularity in order to mark slaves as dead and drain spot clusters correctly.

Maybe there's another way - simply decomission unneeded slave in singularity, but that's another API call which might transform into some sort of attack in the future.

[ssalinas]

Ah, so you’re concerned with the opposite of what I was thinking, syncing mesos -> singularity instead of the other way around. That makes more sense in terms of urgency. I will have to check and see what data we are able to get from the mesos master for those events. As far as I knew from previous versions maintenance mode did not send a notification to schedulers.

As for the ‘attack’ a few things:

• there is a config for the max number of concurrent decommissions you can use as a protection in Singularity
• can’t lifecycle hooks hit an endpoint inside a vpc somit isn’t publicly exposed for an attack?
• wouldn’t your mesos master already be hit this same way if that’s how you decommission currently ^?

[yuriy-filatov]

• Max number of concurrent decomissions within one tick sounds reasonable here.

• Under 'attack' I've meant our own 'decomission' scripts. We may spin up/down up to thousands slaves at the moment and this number is constantly growing.

• Mesos cluster is much more powerful than Singularity one at the moment, but we can upscale it. On the other hand - sending two requests for one operation sounds like an overkill, but easily doable.

[bmerry]

Hi - any news on having Singularity pick up maintenance information from Mesos? I'm also running into the issue that when we use the Mesos API to put slaves into maintenance to prevent our other frameworks from starting new tasks, Singularity is still placing tasks. Maintenance information is attached to resource offers, so even if it's not nicely integrated with Singularity's system for maintenance, it would be nice if it would decline to use offers that are marked as currently unavailable.

To me it's strange that Mesos even makes such offers, but they seem to have made maintenance windows completely advisory rather than enforcing anything.