fluent-operator icon indicating copy to clipboard operation
fluent-operator copied to clipboard

bug: With stakater reloader restart option, one replica is getting restarted

Open btalakola opened this issue 10 months ago • 4 comments

Describe the issue

Hi @cw-Guo & Team, Deployed fluent-bit using fluent-operator with v3.3.0. I am using stakater reloader as below to restart the fluent-bit pods in case of any configurational changes:

Restart fluent-bit daemonset using stakater relaoder if there are any configurational changes.

fluent-bit
annotations: secret.reloader.stakater.com/reload: "fluent-bit-config"

This is restarting only one replica of fluent-bit daemonset though the "kubectl rollout restart daemonsets fluent-bit -n namespace-dummy" restarting all the replicas. Is there anything that I need to work on using stakater reloader? Can you please please help me out here. Thanks in advance!

Similar issue ref : https://github.com/fluent/fluent-operator/issues/1479

To Reproduce

Deployed fluent-bit using fluent-operator with v3.3.0. I am using stakater reloader to restart the fluent-bit pods in case of any configurational changes. This actually restarting only one replica of fluent-bit daemonset

Expected behavior

With stakater reload option, should actually restart all the replicas.

Your Environment

- Fluent Operator version:v3.3.0
- Container Runtime:
- Operating system:
- Kernel version:

How did you install fluent operator?

No response

Additional context

No response

btalakola avatar Mar 17 '25 12:03 btalakola

Hi @btalakola that's depends on how the tool stakater reloader implement the rollout.

Steps to resolve this issue:

  1. understand how stakater reloader implement the rollout, I would assume that they use an annotation to implement that.
  2. refer to this https://github.com/fluent/fluent-operator/pull/1375 to retain that annotation too.

cw-Guo avatar Mar 17 '25 20:03 cw-Guo

@cw-Guo Thanks for the response.

  1. For the Point#1: Stakater reloader implmenets the rollout restart in the following manner : https://github.com/stakater/Reloader/blob/master/internal/pkg/callbacks/rolling_upgrade.go#L382-L387

// UpdateDaemonSet performs rolling upgrade on daemonSet func UpdateDaemonSet(clients kube.Clients, namespace string, resource runtime.Object) error { daemonSet := resource.(*appsv1.DaemonSet) _, err := clients.KubernetesClient.AppsV1().DaemonSets(namespace).Update(context.TODO(), daemonSet, meta_v1.UpdateOptions{FieldManager: "Reloader"}) return err } 2. For the Point#2 : With PR-1375, fluent operator "kubectl rollout restart daemonsets fluent-bit -n ns-dummy " issue is fixed but not the Reloader use case. We have Reloader uses for other deamonsets where the restart of the pods is working fine in case configmap/secret changes but it is not working fine for fluent operator. Can you provide any suggestions to make it to work as expected?

Thanks in advance!

btalakola avatar Mar 24 '25 14:03 btalakola

any inputs on this?

btalakola avatar Oct 07 '25 08:10 btalakola

@cw-Guo bump -- same issue here, and PR-1375 does not resolve the Reloader case.

DanSalt avatar Oct 07 '25 09:10 DanSalt