test_balanced_host_constraint sometimes fails

[dposada]

=================================== FAILURES ===================================
____________________ CookTest.test_balanced_host_constraint ____________________
[gw1] linux -- Python 3.6.3 /opt/python/3.6.3/bin/python3.6
self = <tests.cook.test_basic.CookTest testMethod=test_balanced_host_constraint>
    def test_balanced_host_constraint(self):
        state = util.get_mesos_state(self.mesos_url)
        num_hosts = len(state['slaves'])
        minimum_hosts = min(10, num_hosts)
        group = {'uuid': str(uuid.uuid4()),
                 'host-placement': {'type': 'balanced',
                                    'parameters': {'attribute': 'HOSTNAME',
                                                   'minimum': minimum_hosts}}}
        job_spec = {'group': group['uuid'],
                    'command': 'sleep 600',
                    'mem': 100,
                    'cpus': 0.1}
        max_jobs_per_host = 3
        num_jobs = minimum_hosts * max_jobs_per_host
        uuids, resp = util.submit_jobs(self.cook_url, job_spec, num_jobs, groups=[group])
        self.assertEqual(201, resp.status_code, resp.content)
        try:
            jobs = util.wait_for_jobs(self.cook_url, uuids, 'running')
            hosts = [j['instances'][0]['hostname'] for j in jobs]
            host_count = Counter(hosts)
            self.assertGreaterEqual(len(host_count), minimum_hosts, hosts)
>           self.assertLessEqual(max(host_count.values()), max_jobs_per_host, host_count)
E           AssertionError: 4 not less than or equal to 3 : Counter({'172.17.0.7': 4, '172.17.0.6': 4, '172.17.0.5': 4, '172.17.0.10': 4, '172.17.0.9': 1, '172.17.0.8': 1})
tests/cook/test_basic.py:1571: AssertionError

[pschorf]

When I looked at this a while ago, I saw messages like this in the cook logs:

2017-11-17 14:12:57,263 INFO  com.netflix.fenzo.AssignableVMs [async-thread-macro-5] - Removed inactive host 172.17.0.8
2017-11-17 14:12:57,263 INFO  com.netflix.fenzo.AssignableVMs [async-thread-macro-5] - Removed inactive host 172.17.0.9
2017-11-17 14:12:57,263 INFO  com.netflix.fenzo.AssignableVMs [async-thread-macro-5] - Removed inactive host 172.17.0.6
2017-11-17 14:12:57,263 INFO  com.netflix.fenzo.AssignableVMs [async-thread-macro-5] - Removed inactive host 172.17.0.7
2017-11-17 14:12:57,263 INFO  com.netflix.fenzo.AssignableVMs [async-thread-macro-5] - Removed inactive host 172.17.0.10
2017-11-17 14:12:57,263 INFO  com.netflix.fenzo.AssignableVMs [async-thread-macro-5] - Removed inactive host 172.17.0.11```

Need to dig into it some more and see if it's the same issue, and if so what's happening with it.