apisix-ingress-controller icon indicating copy to clipboard operation
apisix-ingress-controller copied to clipboard
verified publisher icon apache

ApisixTls status not updated

Open papdaniel opened this issue 3 years ago • 6 comments

Issue description

I use cert manager to generate certificates and create the Certificate and ApisixTls resources at the same time. Cert-manager needs some time to request the certificate, so when apisix-ingress-controller tries to find the cert secret it doesn't exist yet. The problem is apisix-ingress-controller won't retry to find the secret, so we are stucked. I know it is documented here that the secret should exist when we create the ApisixTls resource, but with a retry mechanism it would be much more relieable.

Edit: as I see, there is a retry rule in apisix-ingress-controller, but in this case it doesn't seem to retry. Maybe because of the ResourceSyncAborted status reason?

Environment

  • your apisix-ingress-controller version (output of apisix-ingress-controller version --long):
Version: 1.4.1
Git SHA: no-git-module
Go Version: go1.16.15
Building OS/Arch: linux/amd64
Running OS/Arch: linux/amd64
  • your Kubernetes cluster version (output of kubectl version):
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:26:19Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.11-eks-18ef993", GitCommit:"b9628d6d3867ffd84c704af0befd31c7451cdc37", GitTreeState:"clean", BuildDate:"2022-07-06T18:06:23Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

Minimal test code / Steps to reproduce

Install cert-manager and apisix-ingress-controller, and
create the following:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: test
  namespace: default
spec:
  dnsNames:
  - '*.test.example.com'
  issuerRef:
    group: cert-manager.io
    kind: ClusterIssuer
    name: letsencrypt-prod
  secretName: test-wildcard
---
apiVersion: apisix.apache.org/v2beta3
kind: ApisixTls
metadata:
  name: test
  namespace: default
spec:
  hosts:
  - '*.test.example.com'
  secret:
    name: test-wildcard
    namespace: default

Actual result

status:
  conditions:
  - lastTransitionTime: "2022-08-04T13:23:20Z"
    message: secret "test-wildcard" not found
    observedGeneration: 1
    reason: ResourceSyncAborted
    status: "False"
    type: ResourcesAvailable

Error log

Expected result

No response

papdaniel avatar Aug 04 '22 19:08 papdaniel

Ehh, my bad, it works as expected, the status tricked me, it is not updated.

papdaniel avatar Aug 05 '22 07:08 papdaniel

Ehh, my bad, it works as expected, the status tricked me, it is not updated.

Do you mean the status of the ApisixTls resource is not updated? @papdaniel

tao12345666333 avatar Aug 05 '22 14:08 tao12345666333

Yes @tao12345666333

papdaniel avatar Aug 06 '22 02:08 papdaniel

We expected that all status should be updated correctly, so I reopened this issue

tao12345666333 avatar Aug 06 '22 05:08 tao12345666333

Will reproduce when I'm free, if this is confirmed we can fix it

Can you provide a minimal complete reproduction steps if you are interested?

tao12345666333 avatar Aug 06 '22 06:08 tao12345666333

@tao12345666333 sure, I will on the weekend if that's fine, don't have time right now

papdaniel avatar Aug 08 '22 12:08 papdaniel

Here is how to reproduce:

create namespace and the ApisixTls:

kubectl create namespace apisix-tls-test

cat <<EOF | kubectl apply -f -
apiVersion: apisix.apache.org/v2beta3
kind: ApisixTls
metadata:
  name: test-tls
  namespace: apisix-tls-test
spec:
  hosts:
  - "*.example.com"
  secret:
    name: test-tls
    namespace: apisix-tls-test
EOF

check the ApisixTls status:

kubectl -n apisix-tls-test get apisixtls test-tls -o yaml

...
status:
  conditions:
  - lastTransitionTime: "2022-08-14T19:01:48Z"
    message: secret "test-tls" not found
    observedGeneration: 1
    reason: ResourceSyncAborted
    status: "False"
    type: ResourcesAvailable
...

ingress controller logs (secret not found):

kubectl -n ingress-apisix logs <apisix-ingress-controller-pod>
...
022-08-14T19:02:46+08:00       warn    ingress/apisix_tls.go:178       sync ApisixTls failed, will retry       {"object": {"Type":1,"Object":"apisix-tls-test/test-tls","Tombstone":null}, "error": "secret \"test-tls\" not found"}
2022-08-14T19:02:46+08:00       error   ingress/apisix_tls.go:118       failed to translate ApisixTls   {"error": "secret \"test-tls\" not found", "ApisixTls": {"metadata":{"name":"test-tls","namespace":"apisix-tls-test","uid":"44895291-29d5-4454-a011-079a64140420","resourceVersion":"49712","generation":1,"creationTimestamp":"2022-08-14T11:02:46Z","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apisix.apache.org/v2beta3\",\"kind\":\"ApisixTls\",\"metadata\":{\"annotations\":{},\"name\":\"test-tls\",\"namespace\":\"apisix-tls-test\"},\"spec\":{\"hosts\":[\"*.example.com\"],\"secret\":{\"name\":\"test-tls\",\"namespace\":\"apisix-tls-test\"}}}\n"},"managedFields":[{"manager":"apisix-ingress-controller","operation":"Update","apiVersion":"apisix.apache.org/v2beta3","time":"2022-08-14T11:02:46Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:conditions":{}}},"subresource":"status"},{"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"apisix.apache.org/v2beta3","time":"2022-08-14T11:02:46Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{".":{},"f:hosts":{},"f:secret":{".":{},"f:name":{},"f:namespace":{}}}}}]},"spec":{"hosts":["*.example.com"],"secret":{"name":"test-tls","namespace":"apisix-tls-test"}},"status":{"conditions":[{"type":"ResourcesAvailable","status":"False","observedGeneration":1,"lastTransitionTime":"2022-08-14T11:02:46Z","reason":"ResourceSyncAborted","message":"secret \"test-tls\" not found"}]}}}
...

wait a few seconds, then create the tls secret

cat <<EOF | kubectl apply -f -
kind: Secret
apiVersion: v1
metadata:
  name: test-tls
  namespace: apisix-tls-test
data:
  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURmVENDQW1XZ0F3SUJBZ0lVQlpyZ0M2ZFFTWmNWYzNhWlJlU2R6Ulg4aktJd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0ZqRVVNQklHQTFVRUF3d0xaWGhoYlhCc1pTNWpiMjB3SGhjTk1qSXdPREE0TVRJMU56QXpXaGNOTkRreApNakkwTVRJMU56QXpXakFTTVJBd0RnWURWUVFEREFkMFpYTjBMV051TUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGCkFBT0NBUThBTUlJQkNnS0NBUUVBeWtuU2pJMjRkSHg5L1hxRGwwcHQ2emVydUh0N29FQXhwOHp2bW85TUp1aFQKSTQxL2NNZENCVi93TFFPQnk5TzRTMVhBcU9tUEpPUDNPQ1JsbWZPdndyYXhHY0V0NWlYNG4vT2dmRzhuMVJQLwo3emlwZG5DZWpzMVQzNFh0TFZnL3RzNWM4MDBJU0NtRGlLaUtLcUl6SktaUDVQRVB2b2tXNlV2TTVaNVNCbTJKClBiU2psL0I2MUtBVW5UaEVzSjUvc2tqckgyOVB0N1JONFVBTDNlM1RmL1YxZW9ESmZJOWo0QlRjSGdTQUNvL1QKOXBWNmo3cWs4aWdqSEJpWWtFaW1RSi9vKzRXSFhTb2IwYzdIN3lRbVBXLzJ0UFNvd2twMnlTUGhZQWRla0NBMQo1TDVHTjl6ZUl4TFJpZDVXKzd0WldkYzliU3RIWGFoQkY5OUVHTFJ4UndJREFRQUJvNEhHTUlIRE1GRUdBMVVkCkl3UktNRWlBRktnMnk0ZGlzR1BUaVhJY2lwTjUyMjNYa04wUW9ScWtHREFXTVJRd0VnWURWUVFEREF0bGVHRnQKY0d4bExtTnZiWUlVT1FMZlZqa094elQ2KzZDZEZlTGFnNHd1ZGVvd0NRWURWUjBUQkFJd0FEQUxCZ05WSFE4RQpCQU1DQkRBd0hRWURWUjBsQkJZd0ZBWUlLd1lCQlFVSEF3RUdDQ3NHQVFVRkJ3TUNNQmdHQTFVZEVRUVJNQStDCkRTb3VaWGhoYlhCc1pTNWpiMjB3SFFZRFZSME9CQllFRkZuS2hmQkhoT1JwYVB0eVF5NVBBMExmMDBtOE1BMEcKQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUFBQ2NSK0J2S1FPYmw2YUhwZnpObE1GajVXTmhucWE1N3NVelpYREhERQpzVEpJMFlDa3hVZHBEdXNCS3RlSno1dWgwTmlXZkI5c2RSa3F1c2V6b3dvSkU0UnZnMUNqOWRrWlQ4TzZPN2VPCjljczFkM0hydEdoODMvemk5ajM3WFNldEIzQXNkUHA1ZnhjcXQzbDVLVGE4enZIUHF5ZWhxS3JHamlodkFRUGkKeUZzMGsra0R1TXIvSk96QmY2b09xS2dGdy9LVmZrTFdQSlRmSWlvNjYzcUp4dDQzTmlsbnpQNGFpY2dXQm93RgpHcXJndDdOcHhBRWkvMlEwYkF5b0JFdURuZngwamxSMytKaWtuYTNxUENMQlAzNEFqZ0ZJZnFUOGNjWjIwQm5ECk9aMitDM2xra1JhZ3luMlRtUU5RTzBXbzh5YVJLRUY1T0g5Wk5mZThIbDZICi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
  tls.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2pBZ0VBQW9JQkFRREtTZEtNamJoMGZIMzkKZW9PWFNtM3JONnU0ZTN1Z1FER256TythajB3bTZGTWpqWDl3eDBJRlgvQXRBNEhMMDdoTFZjQ282WThrNC9jNApKR1daODYvQ3RyRVp3UzNtSmZpZjg2QjhieWZWRS8vdk9LbDJjSjZPelZQZmhlMHRXRCsyemx6elRRaElLWU9JCnFJb3Fvak1rcGsvazhRKytpUmJwUzh6bG5sSUdiWWs5dEtPWDhIclVvQlNkT0VTd25uK3lTT3NmYjArM3RFM2gKUUF2ZDdkTi85WFY2Z01sOGoyUGdGTndlQklBS2o5UDJsWHFQdXFUeUtDTWNHSmlRU0taQW4rajdoWWRkS2h2Ugp6c2Z2SkNZOWIvYTA5S2pDU25iSkkrRmdCMTZRSURYa3ZrWTMzTjRqRXRHSjNsYjd1MWxaMXoxdEswZGRxRUVYCjMwUVl0SEZIQWdNQkFBRUNnZ0VBQklld1JMRDlkN0J5T2c3S3JQcTNqOEdiK3Mwc0VNYzRnVDJKbDNiQXJ0SmYKRk1qbDhIL3RhWFlqVmVGQVlpMTFwdE1VdktMSTNyNzlPQ1U0R0RQUlQ0c2d3NmJ3R0hZZWwrb3dUeFFEQXdkWApRcGxrRlJNd2Rab2NROVdNSFpoUkE3QnNIVEZ6aUp0dU94NDVBUWo5bFNoRHF1d242T1QvUGNYcUdmVUY4bUVUCkRsRWtRUXowSi9BNmhqTURDR244U1Z1V29lallOdWJHRnV6OGdQUEdhUFowcjltbXlrRHBpQkMrbTluZkpjQnUKM1REWDY5Rmc1dzJSSWs4STQ4VTdVWW5yZjQ3aDRVbDA2NFU3a0VtekNYcERwdDJvQk1pMzFqWDMvajJuL2tkUApGMjNEdEY0RHgvT1FackFBMzVtaTRkZ2x5WCtQWHZTZ2hIa2dHaUF6cVFLQmdRRFZybGlONFFVbFJmTTlCU0F6Cmw3YWEySnpTNXYyTldVNVpXWlVhd1NIRFNlUERDL21LZnpLVzBOYWNvQmFkWnJ2VkJibTMrTUszbjBvVlRDaFkKUVN3c1ByZmFjV3VIRk1tejZZWFFjT0djWWVOeW8yelBRbU9LcnB0WC9Vck5GeGozRUV6QTcrelBPOUxRQXZFego1NFJWYkFRQUJoMUU5ZmFISHpORjljbjByUUtCZ1FEeVdkM29sZ2p6amlZbjgwMGJ1a0JEWUxRVEFHN2tRd1BDCjd4TFQzN21zL001S1I2d1dEVUE3VzlXWFlqcjAxbGJidm5Ha0RNdHNLM0poRlZMQ3dMN1VHWUU4dWVKZmdyNHgKOVFtcFMwN05XWXVqdC81MGdEa3VsNjRUdndIZ25JWFJTK0ZSRHE1K0I5OWtlaWkyZktwQjUrRlFUWWI4Y3VCawpNR2V5L0lRSVF3S0JnRlNSRUlqdUlrd3pGK1h4Ym1JY3krOFNoVlpPaHRCM0pMUGw2VVhTWTV0WFZlV3VpbzhPCmNpWVA5cSs4QjljNWZtTDdSaG9TUUhYTmU5TmI5YWVmKzJ4elVlNEk3MFpocXNZaVNJeEs4RzhrcXBDSEx4d0QKcVVPTWdQeGN5VXJMdGx5cFBjY3A3UzNDL2xiR28xRW9jazRnVnFEdnlvdXRVR2ovcHhvc0FVWnRBb0dBSHZ3awpaZ2RBcjdyOTM5KzRmUklNQTFGSzFKOGFqbTZPL1k1SmJ2Tk1ydlJIZjdHNEs3Mlg5NzdmQ1JMVEROaVdsRk85Ci9jWncvNnh5WEQwNHJwT3NFandNalBJTElrRDZCak1aQUI4VHZTZDB0Z0xoTW8zdVFhbTg1T05uM3dyRUJTVloKY2w2WktwM2hEdHliZGZGc1p5NjVYT1g4bXl3b3lrWnlDNjFEU0RjQ2dZRUFnK1hkWVdYTDQwNzY5MVlKSEJDTQpKcG5BZjM2V2dOUXNLVGxwbkR0MFNGNk9LRms3VjZPRXdXdFgxWm1VUmY4a1B2YWxUTFh0OEZYcVAyRk90TkRYClUzTlNNdEhxS3lWeHZwTm5YdUFLSlkrNnNtckZhdXZTTlVHWEVoUElUdi9rcXZOdGRmMXhFaml6cTJ4NGpoaDAKZkZUUEppWThoWjlORVdTWkVoaHB0RWs9Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K
type: kubernetes.io/tls
EOF

ingress controller logs (only healthcheck, no error):

kubectl -n ingress-apisix logs  <apisix-ingress-controller-pod>
...
[GIN] 2022/08/14 - 19:03:11 | 200 |      42.061µs |     10.0.43.201 | GET      "/healthz"
[GIN] 2022/08/14 - 19:03:11 | 200 |       36.23µs |     10.0.43.201 | GET      "/healthz"
[GIN] 2022/08/14 - 19:03:21 | 200 |      36.421µs |     10.0.43.201 | GET      "/healthz"
[GIN] 2022/08/14 - 19:03:21 | 200 |      22.711µs |     10.0.43.201 | GET      "/healthz"
...

apisix logs (ssl resource created successfully in apisix):

kubectl -n ingress-apisix logs <apisix-pod>
...
10.0.41.91 - - [14/Aug/2022:11:03:51 +0000] apisix-admin.ingress-apisix.svc.cluster.local:9180 "PUT /apisix/admin/ssl/21210e5d HTTP/1.1" 201 3878 0.049 "-" "Go-http-client/1.1" - - - "http://apisix-admin.ingress-apisix.svc.cluster.local:9180"
...

Check the ApisixTls status again, not updated:

kubectl -n apisix-tls-test get apisixtls test-tls -o yaml

...
status:
  conditions:
  - lastTransitionTime: "2022-08-14T19:01:48Z"
    message: secret "test-tls" not found
    observedGeneration: 1
    reason: ResourceSyncAborted
    status: "False"
    type: ResourcesAvailable
...

papdaniel avatar Aug 14 '22 12:08 papdaniel

thanks!

tao12345666333 avatar Aug 14 '22 20:08 tao12345666333

Would you like to try to fix this problem?

There is a recordStatus function in the code to record the change of the status

tao12345666333 avatar Aug 25 '22 04:08 tao12345666333

Yes, sure,I'll try.

papdaniel avatar Aug 28 '22 10:08 papdaniel

Just cloned and tried out on master, now it's working: image

I think it got solved by this commit a few weeks ago: d32c728 on these and these lines.

papdaniel avatar Aug 28 '22 13:08 papdaniel

Thanks for your confirmation, then I think this issue can be closed, right?

tao12345666333 avatar Aug 29 '22 02:08 tao12345666333

Also, I thought maybe we should create a patch for 1.5.0 to use . Because we haven't released the official version of v1.5.0, the current release is the rc1 version

You can try v1.5.0 branch https://github.com/apache/apisix-ingress-controller/tree/v1.5.0 Thanks!

tao12345666333 avatar Aug 29 '22 02:08 tao12345666333

I won't really have time for this, sorry.

papdaniel avatar Sep 04 '22 13:09 papdaniel

Don't worry, thanks for your contribution

tao12345666333 avatar Sep 05 '22 08:09 tao12345666333