Churro
Churro
this is awesome, but i see that the parallel index task now uses the shuffle which now relies on the middle manager as something more than just launching and managing...
> @zhangyue19921010 are you still working on this PR? Happy to help with the reviewing if needed. @abhishekagarwal87 I am happy to pick this up. I have reviewed the code...
> Thank you for taking this up @churromorales. It will be nice to keep this as extension. You can certainly add extension points in the core if you need them....
I noticed a slight issue when testing on heavily loaded k8s clusters. Those that take a while to spin up / down pods. Currently the way things work: 1. The...
Thank you for your kind words. I hope this patch will be useful to folks upstream. > The resource use for overlords and tasks is vastly different, using the same...
> > This doesn't use the resources of the overlord, we overwrite the resources for podSpec. The memory is derived from your JAVA_OPTS (for your task) basically 1.2*(Xmx + Dbb)...
@gianm I was testing the MM-less patch on the msq work you did. I ran a test ingestion and the tasks just hang forever, after a bit of debugging here...
> @abhishekagarwal87 > > > may not be an issue but is there a limit in k8s on how large the env variable value can be? The task JSON could...
> * Avoid creating TaskLocations with `null` host and non-`-1` values for `port` and `tlsPort`. This may be happening in `K8sWorkItem` if `mainPod.getStatus().getPodIP()` is `null`. Do you think this is...
i do find the branch name quite intriguing.