operator-sdk icon indicating copy to clipboard operation
operator-sdk copied to clipboard

Managed field changes cause unnecessary reconciliation loops

Open erinc1915 opened this issue 3 years ago • 1 comments

Bug Report

We are observing repeated reconciliation loops due to changes to managed fields in dependent resources. For example, a deployment whose status changes from readyReplicas: 1 to unavailableReplicas: 1 triggers reconciliation. I know that changes to the status field are ignored, but it seems that the corresponding changes to the managed fields are not.

In my example, the managed fields change as shown:

      f:status:
        f:availableReplicas: {}

to

      f:status:
        f:unavailableReplicas: {}

The only other changes to the deployment are to resourceVersion, the status field, and time under managedFields, all of which I believe are already ignored.

What did you do?

Deployed operator and monitored progress.

What did you expect to see?

The operator resource to complete install successfully.

What did you see instead? Under which circumstances?

The operator continuously enters reconciliation loops. Logs point to a deployment change triggering the reconciliation. Upon watching the specified deployment, the only changes are as reported above.

Environment

Operator type: Ansible Operator /language ansible

Kubernetes cluster type: Openshift 4.6.52

$ operator-sdk version

ansible-operator version: "v1.17.0", commit: "704b02a9ba86e85f43edb1b20457859e9eedc6e6", kubernetes version: "v1.21", go version: "go1.17.8", GOOS: "linux", GOARCH: "amd64"

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"2c6062082e8dbbf36936b35dcc27c6882d3a2f67", GitTreeState:"clean", BuildDate:"2021-12-03T19:10:26Z", GoVersion:"go1.15.14", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.16+3d19195", GitCommit:"aa3d30d3ccfe7b5e2c2bac7f7377725893c9a928", GitTreeState:"clean", BuildDate:"2021-11-23T18:35:13Z", GoVersion:"go1.15.14", Compiler:"gc", Platform:"linux/amd64"}

Possible Solution

Ideally it would be nice to be able to disable watching managed fields altogether, perhaps via an option in watches.yaml. At minimum the managed fields for status should not be watched, as to replicate operator behavior prior to their introduction (don't reconcile when only status changes).

erinc1915 avatar Aug 05 '22 18:08 erinc1915

This seems like it could be reasonable, but IMO it will need some design discussion prior to implementation. If anyone needs this, please feel free to propose a design.

asmacdo avatar Aug 15 '22 16:08 asmacdo

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-bot avatar Nov 14 '22 01:11 openshift-bot

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten /remove-lifecycle stale

openshift-bot avatar Dec 14 '22 08:12 openshift-bot

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen. Mark the issue as fresh by commenting /remove-lifecycle rotten. Exclude this issue from closing again by commenting /lifecycle frozen.

/close

openshift-bot avatar Jan 14 '23 00:01 openshift-bot

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen. Mark the issue as fresh by commenting /remove-lifecycle rotten. Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci[bot] avatar Jan 14 '23 00:01 openshift-ci[bot]