jtuglu1
jtuglu1
@kfaraz thoughts here? The idea behind this is to scale proportionally to the lag, since scaling is an expensive operation (from an accrued lag perspective).
> Do you feel that the current scale up logic adds too many tasks or too few? Depending on the scenario, it could be either. What I was aiming for...
> So it would be nice if we could do take a combined approach in this PR i.e. proportional scaling to a valid task count. Yes, this is something I've...
> Sounds good, @jtuglu-netflix , please update this PR accordingly with both the changes. I will be happy to do a review. > > For the "valid" task count logic,...
> > In this case, the proportional scaling config parameters' meanings differ from what their documented use is with the threshold-based scaler. I'm wondering if this is something we'd want...
I can take this up cc @maytasm
> This could be an extension of the multiTopic functionality, I suppose: the different topics may be able to come from different Kafka clusters. Yeah. I just prefix the partition...
> That way, we could scale independently, configure separate consumerProperties for different clusters, etc. With a single supervisor, I think the latter requirement is still doable. The former is the...
> Perhaps reading from multiple topics in a single supervisor was more straightforward. Regarding this, I think that was a much simpler issue since it was still only reading from...
@kfaraz I'm slightly against supervisor UUIDs since that might make things less readable on the console (users would have to look at the spec to get an idea of what...