Error message in use with argo-rollouts

[huikang]

I setup the notification-engine with the argo-rollouts deployment; everything looks good and messages are delivered to the slack channel as expected. However, I noticed that there are several error message each time some event is triggered:

INFO[2021-09-26T23:42:29-04:00] Start processing                              resource=default/rollouts-demo
INFO[2021-09-26T23:42:29-04:00] Reconciliation completed                      generation=3 namespace=default resourceVersion=2148175 rollout=rollouts-demo time_ms=15.550582
INFO[2021-09-26T23:42:29-04:00] Started syncing rollout                       generation=3 namespace=default resourceVersion=2148177 rollout=rollouts-demo
ERRO[2021-09-26T23:42:29-04:00] Failed to process: Object 'Kind' is missing in '{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"argoproj.io/v1alpha1\",\"kind\":\"Rollout\",\"metadata\":{\"annotations\":{\"notifications.argoproj.io/subscribe.on-rollout-completed.slack\":\"sdpl-test\",\"notifications.argoproj.io/subscribe.on-rollout-step-completed.slack\":\"sdpl-test\",\"notifications.argoproj.io/subscribe.on-rollout-updated.slack\":\"sdpl-test\"},\"name\":\"rollouts-demo\",\"namespace\":\"default\"},\"spec\":{\"replicas\":5,\"revisionHistoryLimit\":2,\"selector\":{\"matchLabels\":{\"app\":\"rollouts-demo\"}},\"strategy\":{\"canary\":{\"steps\":[{\"setWeight\":20},{\"analysis\":{\"args\":[{\"name\":\"duration\",\"value\":\"1s\"},{\"name\":\"exit-code\",\"value\":\"0\"}],\"templates\":[{\"templateName\":\"sleep-job\"}]}},{\"pause\":{}},{\"setWeight\":40},{\"pause\":{}},{\"setWeight\":60},{\"pause\":{}},{\"setWeight\":80},{\"pause\":{}}]}},\"template\":{\"metadata\":{\"labels\":{\"app\":\"rollouts-demo\"}},\"spec\":{\"containers\":[{\"image\":\"argoproj/rollouts-demo:blue\",\"name\":\"rollouts-demo\",\"ports\":[{\"containerPort\":8080,\"name\":\"http\",\"protocol\":\"TCP\"}],\"resources\":{\"requests\":{\"cpu\":\"5m\",\"memory\":\"32Mi\"}}}]}}}}\n","notifications.argoproj.io/subscribe.on-rollout-completed.slack":"sdpl-test","notifications.argoproj.io/subscribe.on-rollout-step-completed.slack":"sdpl-test","notifications.argoproj.io/subscribe.on-rollout-updated.slack":"sdpl-test","rollout.argoproj.io/revision":"3"},"creationTimestamp":"2021-09-27T02:28:28Z","generation":3,"managedFields":[{"apiVersion":"argoproj.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{},"f:notifications.argoproj.io/subscribe.on-rollout-completed.slack":{},"f:notifications.argoproj.io/subscribe.on-rollout-step-completed.slack":{},"f:notifications.argoproj.io/subscribe.on-rollout-updated.slack":{}}},"f:spec":{".":{},"f:replicas":{},"f:revisionHistoryLimit":{},"f:selector":{".":{},"f:matchLabels":{".":{},"f:app":{}}},"f:strategy":{".":{},"f:canary":{".":{},"f:steps":{}}},"f:template":{".":{},"f:metadata":{".":{},"f:labels":{".":{},"f:app":{}}},"f:spec":{".":{},"f:containers":{}}}}},"manager":"kubectl-client-side-apply","operation":"Update","time":"2021-09-27T02:28:28Z"},{"apiVersion":"argoproj.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}}},"manager":"kubectl-argo-rollouts-darwin-amd64","operation":"Update","time":"2021-09-27T03:37:42Z"},{"apiVersion":"argoproj.io/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:rollout.argoproj.io/revision":{}}},"f:status":{".":{},"f:HPAReplicas":{},"f:availableReplicas":{},"f:blueGreen":{},"f:canary":{},"f:conditions":{},"f:currentPodHash":{},"f:currentStepHash":{},"f:currentStepIndex":{},"f:message":{},"f:observedGeneration":{},"f:phase":{},"f:readyReplicas":{},"f:replicas":{},"f:selector":{},"f:stableRS":{},"f:updatedReplicas":{}}},"manager":"rollouts-controller","operation":"Update","time":"2021-09-27T03:42:29Z"}],"name":"rollouts-demo","namespace":"default","resourceVersion":"2148177","uid":"f7377772-b6a9-41d4-baaa-f25bd7d670a4"},"spec":{"replicas":5,"revisionHistoryLimit":2,"selector":{"matchLabels":{"app":"rollouts-demo"}},"strategy":{"canary":{"steps":[{"setWeight":20},{"analysis":{"args":[{"name":"duration","value":"1s"},{"name":"exit-code","value":"0"}],"templates":[{"templateName":"sleep-job"}]}},{"pause":{}},{"setWeight":40},{"pause":{}},{"setWeight":60},{"pause":{}},{"setWeight":80},{"pause":{}}]}},"template":{"metadata":{"creationTimestamp":null,"labels":{"app":"rollouts-demo"}},"spec":{"containers":[{"image":"argoproj/rollouts-demo:red","name":"rollouts-demo","ports":[{"containerPort":8080,"name":"http","protocol":"TCP"}],"resources":{"requests":{"cpu":"5m","memory":"32Mi"}}}]}}},"status":{"HPAReplicas":5,"availableReplicas":4,"blueGreen":{},"canary":{},"conditions":[{"lastTransitionTime":"2021-09-27T03:37:42Z","lastUpdateTime":"2021-09-27T03:37:42Z","message":"Rollout is paused","reason":"RolloutPaused","status":"False","type":"Paused"},{"lastTransitionTime":"2021-09-27T03:42:28Z","lastUpdateTime":"2021-09-27T03:42:28Z","message":"RolloutCompleted","reason":"RolloutCompleted","status":"False","type":"Completed"},{"lastTransitionTime":"2021-09-27T03:42:29Z","lastUpdateTime":"2021-09-27T03:42:29Z","message":"Rollout does not have minimum availability","reason":"AvailableReason","status":"False","type":"Available"},{"lastTransitionTime":"2021-09-27T03:37:42Z","lastUpdateTime":"2021-09-27T03:42:29Z","message":"ReplicaSet \"rollouts-demo-6b5cbf7465\" is progressing.","reason":"ReplicaSetUpdated","status":"True","type":"Progressing"}],"currentPodHash":"6b5cbf7465","currentStepHash":"89b96d8b9","currentStepIndex":0,"message":"more replicas need to be updated","observedGeneration":"3","phase":"Progressing","readyReplicas":4,"replicas":5,"selector":"app=rollouts-demo","stableRS":"75f756d67b","updatedReplicas":1}}'  resource=default/rollouts-demo

The above is the argo-rollouts controller's log, where I think the error message of the last line comes from the notification-engine. It could be

https://github.com/argoproj/notifications-engine/blob/3203039170aff46026e8c7fe1d82e62b8f330997/pkg/controller/controller.go#L229-L231

Although not harmful to the rollouts CR, the error messages are annoying. Do I miss anything in the configuration step? Thanks.

[anup1384]

I'm also receiving the same issue while using notification engine for sending alert on slack/teams.

level=error msg="Failed to process: Object 'Kind' is missing in '{\"metadata\":{\"annotations\":{\"kubectl.kubernetes.io/last-applied-configuration\":\"{\\\"apiVersion\\\":\\\"argoproj.io/v1alpha1\\\",\\\"kind\\\":\\\"Rollout\\\",\\\"metadata\\\":{\\\"annotations\\\":{\\\"notifications.argoproj.io/subscribe.on-rollout-completed.teams\\\":\\\"channelName\\\",\\\"notifications.argoproj.io/subscribe.on-rollout-step-completed.teams\\\":\\\"channelName\\\",\\\"notifications.argoproj.io/subsc