The invoker pod was cleaned up for some reason and the function call timed out.

[zijiwork]

apisix calls openwhisk, when the invoker pod is cleaned up for some reason, this will cause apisix to return 503.

POST /api/v1/namespaces/guest/actions/user/getUser?blocking=true&result=true&timeout=8000 HTTP/1.1 499

Environment details:

• Kubernetes 1.26

Steps to reproduce the issue:

1. kubectl get po -n openwhisk -owide
2. kubectl delete po -n openwhisk wskowdev-invoker-00-65-guest-getuser

# kubectl  get po -n openwhisk 
NAME                                                    READY   STATUS      RESTARTS   AGE
owdev-alarmprovider-67b79fcf46-m7jl8                    1/1     Running     0          62d
owdev-apigateway-6fc74d7fcd-4442p                       1/1     Running     0          62d
owdev-controller-0                                      1/1     Running     0          2d23h
owdev-couchdb-84b8749bbf-dcstw                          1/1     Running     0          62d
owdev-gen-certs-zcqnh                                   0/1     Completed   0          62d
owdev-init-couchdb-nfhv7                                0/1     Completed   0          62d
owdev-invoker-0                                         1/1     Running     0          2d22h
owdev-kafka-0                                           1/1     Running     0          62d
owdev-kafkaprovider-b4ddfd888-bwk66                     1/1     Running     0          62d
owdev-nginx-844c58dd89-ghdmp                            1/1     Running     0          62d
owdev-redis-57b7889b49-g7pf6                            1/1     Running     0          62d
owdev-wskadmin                                          1/1     Running     0          62d
owdev-zookeeper-0                                       1/1     Running     0          62d
wskowdev-invoker-00-18-prewarm-go115                    1/1     Running     0          2d22h
wskowdev-invoker-00-19-prewarm-go115                    1/1     Running     0          2d22h
wskowdev-invoker-00-65-guest-getuser            1/1     Running     0          21s

3. Call the function

Provide the expected results and outputs:

Normal return function call results

Provide the actual results and outputs:

root@owdev-nginx-844c58dd89-ghdmp:/logs# tail -f *
==> nginx_access.log <==
172.22.17.143 - 23bc46b1-71f6-4ed5-8c54-816aa4f8c502 [09/Sep/2024:06:38:36 +0000] [#tid_806bbecd8407144cde4ba08e42459828] POST /api/v1/namespaces/guest/actions/user/getUser?blocking=true&result=true&timeout=8000 HTTP/1.1 499 0 - lua-resty-http/0.16.1 (Lua) ngx_lua/10025 10.103.10.24:8080
172.22.49.138 - 23bc46b1-71f6-4ed5-8c54-816aa4f8c502 [09/Sep/2024:06:39:08 +0000] [#tid_a349ba08dedc07338ffdcff80153ede4] POST /api/v1/namespaces/guest/actions/user/getUser?blocking=true&timeout=8000&result=true HTTP/1.1 499 0 - lua-resty-http/0.16.1 (Lua) ngx_lua/10025 10.103.10.24:8080

==> nginx_error.log <==
2024/08/26 10:08:56 [error] 31#31: *14464 connect() failed (111: Connection refused) while connecting to upstream, client: 172.22.49.138, server: ~^(?<namespace>[0-9a-zA-Z-]+)\.$, request: "POST /api/v1/namespaces/guest/actions/user/getUser?blocking=true&timeout=60000&result=true HTTP/1.1", upstream: "http://10.103.10.24:8080/api/v1/namespaces/guest/actions/user/getUser?blocking=true&timeout=60000&result=true", host: "10.103.184.89"