Satish Kumar Matti

Dask leaks memory with Batched Kafka and cudf

5

Dask workers' memory shooting up gradually for long running jobs and eventually job crashes when the memory of workers exceeds 80%(or around). Refer to the image below:  Dask is...

Issues scaling kafka consumers with Dask

20

I have run two simple pipelines(Non-dask and Dask) to read stream of messages from Kafka using Streamz FromKafkaBatched method. I seem to get similar throughput for both the pipelines. **Non-Dask...

Reuse Kafka consumer for polling next batch

4

I would like to propose an idea of having cached consumers for FromKafkaBatched class and reuse consumers for getting next batch. The current FromKafkaBatched class create a new Kafka consumer...

Profiling streamz pipeline with from_kafka_batched

2

I am trying to profile the below simple streamz pipeline with from_kafka_batched method, ``` from streamz import Stream from time import time, sleep from tornado import gen def increment(x): return...

Overwrite old node labels for first node update to the endpoint manager

13

This fixes a race condition during agent startup where some node label updates were missed. If the k8s node watcher initializes before the host endpoint is created, the agent might...

fix: trigger host endpoint regeneration only when required

4

Explicitly check if host endpoint's identity labels are changed before calling endpoint.UpdateLabelsFrom function. Without this change, host endpoint is regenerated even if the old and new identity labels are the...