Include ConfigMaps and Secrets into the spec

[linki]

When deployments reference a ConfigMap or Secret they do get updated in place when a new blue/green rollout is triggered. This can already have unintended effects on the old deployment.

1. We could include ConfigMaps as a subsection of the spec, similar to ingress, so that the controller can transparently create and reference dedicated config maps per version (similar to kustomize that creates dedicated config maps for each overlay: https://github.com/kubernetes-sigs/kustomize/blob/37f03b4d018235d1a26dda1f031d374776b381a7/examples/ldap/base/kustomization.yaml#L4-L7)

2. Alternatively, we could use CDP_BUILD_VERSION version as part of the ConfigMap's name and its reference in the podTemplateSpec. This would create a new ConfigMap for each rollout with immutable content. Old versions could be cleaned up by leveraging ownerReference from the Deployment to the ConfigMap, so that once the Stack is deleted the old ConfigMap gets cleaned up too. (Assuming ownerReferences work for ConfigMaps)

See https://github.bus.zalan.do/teapot/tokeninfo-router/pull/10/files for an example of a Deployment using a ConfigMap.

[mikkeloscar]

It should probably be a list of resources in case you have multiple configmaps or secrets.

Alternatively, we could use CDP_BUILD_VERSION version as part of the ConfigMap's name and its reference in the podTemplateSpec. This would create a new ConfigMap for each rollout with immutable content. Old versions could be cleaned up by leveraging ownerReference from the Deployment to the ConfigMap, so that once the Stack is deleted the old ConfigMap gets cleaned up too. (Assuming ownerReferences work for ConfigMaps)

This could work but you have to attach the ownerReference after deployment since the uid is needed in the reference. Would be more magic to do it this way IMO, so I think it would be nicer to have it as part of the stackset definition.

[szuecs]

I created and closed #198 , that is basically the same issue. See also the description which showed how to cause an incident with the current behavior, if users think a stack will have isolation for secrets and configmaps.

A drawback of having secrets inside stackset is, that we somehow need to protect from reading secrets from the stackset object.

[hjacobs]

I think it's a great idea to include ConfigMaps which are managed by a Stack. This would help avoid problems when rolling out config map changes where either all Pods load the new config or not at all (if app does not load). Also rolling back configuration would be easy and fast.

This topic came up during our current discussion around decommissioning the legacy Zalando-internal Config Service.