bug: use secret can cause 500 stack error in some case.

[ronething]

Current Behavior

When I use secret, it will cause a 500 stack error when I delete the secret and send request to apisix.

Expected Behavior

After delete secret and update consumer, if send request to apisix, it will return http status 401 but not cause 500.

maybe i can fix it if i have time.

Error Logs

2024/01/15 17:25:54 [error] 27990#390598690: *25233 lua entry thread aborted: runtime error: ...cuments/api7projects/opensource/apisix/apisix/secret.lua:61: attempt to index local 'v' (a boolean value)
stack traceback:
coroutine 0:
        ...cuments/api7projects/opensource/apisix/apisix/secret.lua: in function 'create_obj_fun'
        .../api7projects/opensource/apisix/apisix/core/lrucache.lua:95: in function 'secret_kv_lrucache'
        ...cuments/api7projects/opensource/apisix/apisix/secret.lua:88: in function 'secret_kv'
        ...cuments/api7projects/opensource/apisix/apisix/secret.lua:156: in function 'fetch_by_uri'
        ...cuments/api7projects/opensource/apisix/apisix/secret.lua:188: in function 'fetch'
        ...cuments/api7projects/opensource/apisix/apisix/secret.lua:211: in function 'fetch_secrets'
        ...ments/api7projects/opensource/apisix/apisix/consumer.lua:107: in function 'create_obj_fun'
        .../api7projects/opensource/apisix/apisix/core/lrucache.lua:95: in function 'lrucache'
        ...ments/api7projects/opensource/apisix/apisix/consumer.lua:116: in function 'consumers_kv'
        ...i7projects/opensource/apisix/apisix/plugins/key-auth.lua:88: in function 'phase_func'
        ...cuments/api7projects/opensource/apisix/apisix/plugin.lua:1095: in function 'run_plugin'
        ...Documents/api7projects/opensource/apisix/apisix/init.lua:633: in function 'http_access_phase'
        access_by_lua(nginx.conf:328):2: in main chunk, client: 127.0.0.1, server: _, request: "GET /anything HTTP/1.1", host: "127.0.0.1:9080"

Steps to Reproduce

$ docker run --name vault \
  --cap-add=IPC_LOCK \
  -e VAULT_DEV_ROOT_TOKEN_ID="root" \
  -e VAULT_ADDR="http://0.0.0.0:8200" \
  -p 8200:8200 \
  vault:1.13.3 server -dev

docker exec -it vault /bin/sh
export VAULT_TOKEN='root'
vault secrets enable -path="kv-v1" kv
/ # vault kv put kv-v1/apisix/jack auth-key=auth-one
Success! Data written to: kv-v1/apisix/jack
/ # vault kv get kv-v1/apisix/jack
====== Data ======
Key         Value
---         -----
auth-key    auth-one

# 1. create secret
$ curl http://127.0.0.1:9180/apisix/admin/secrets/vault/mysecret \
-H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "uri": "http://127.0.0.1:8200",
    "prefix": "kv-v1/apisix",
    "token": "root"
}'
{"key":"/apisix/secrets/vault/mysecret","value":{"id":"vault/mysecret","uri":"http://127.0.0.1:8200","token":"root","prefix":"kv-v1/apisix","create_time":1705310640,"update_time":1705310640}}

# 2. create consumer
$ curl http://127.0.0.1:9180/apisix/admin/consumers \
-H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "username": "jack",
    "plugins": {
        "key-auth": {
            "key": "$secret://vault/mysecret/jack/auth-key"
        }
    }
}'
{"key":"/apisix/consumers/jack","value":{"plugins":{"key-auth":{"key":"$secret://vault/mysecret/jack/auth-key"}},"username":"jack","update_time":1705310647,"create_time":1705310647}}

# 3. create route
$ curl http://127.0.0.1:9180/apisix/admin/routes/1 \
-H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -i -d '
{
    "uri": "/anything",
    "upstream": {
        "type": "roundrobin",
        "nodes": {
            "httpbin.org": 1
        }
    },
    "plugins": {
        "key-auth": {}
    }
}'
HTTP/1.1 201 Created
Date: Mon, 15 Jan 2024 09:24:13 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
Server: APISIX/3.2.2
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: *
Access-Control-Max-Age: 3600
X-API-VERSION: v3

{"key":"/apisix/routes/1","value":{"plugins":{"key-auth":{"query":"apikey","hide_credentials":false,"header":"apikey"}},"status":1,"id":"1","uri":"/anything","upstream":{"pass_host":"pass","nodes":{"httpbin.org":1},"type":"roundrobin","hash_on":"vars","scheme":"http"},"priority":0,"create_time":1705310653,"update_time":1705310653}}

# 4. send request, return 200
$ curl http://127.0.0.1:9080/anything -H "apikey: auth-one" -v
* Uses proxy env variable no_proxy == '192.168.49.2,localhost,127.0.0.1'
*   Trying 127.0.0.1:9080...
* Connected to 127.0.0.1 (127.0.0.1) port 9080
> GET /anything HTTP/1.1
> Host: 127.0.0.1:9080
> User-Agent: curl/8.4.0
> Accept: */*
> apikey: auth-one
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 413
< Connection: keep-alive
< Date: Mon, 15 Jan 2024 09:25:26 GMT
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
< Server: APISIX/3.2.2
<
{
  "args": {},
  "data": "",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Apikey": "auth-one",
    "Host": "127.0.0.1",
    "User-Agent": "curl/8.4.0",
    "X-Amzn-Trace-Id": "Root=1-65a4fa06-6d7a8d426c9a503a56b23c81",
    "X-Forwarded-Host": "127.0.0.1"
  },
  "json": null,
  "method": "GET",
  "origin": "127.0.0.1, 58.253.51.238",
  "url": "http://127.0.0.1/anything"
}
* Connection #0 to host 127.0.0.1 left intact

# 5. delete secret
$ curl http://127.0.0.1:9180/apisix/admin/secrets/vault/mysecret \
-H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X DELETE

{"deleted":"1","key":"/apisix/secrets/vault/mysecret"}

# 6. get secret
$ curl http://127.0.0.1:9180/apisix/admin/secrets/vault/mysecret \
-H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X GET

{"message":"Key not found"}

# 7. update consumer for refresh lrucache
$ curl http://127.0.0.1:9180/apisix/admin/consumers \
-H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "username": "jack",
    "plugins": {
        "key-auth": {
            "key": "$secret://vault/mysecret/jack/auth-key"
        }
    }
}'
{"key":"/apisix/consumers/jack","value":{"plugins":{"key-auth":{"key":"$secret://vault/mysecret/jack/auth-key"}},"username":"jack","update_time":1705310750,"create_time":1705310647}}

# 8. send request, then return 500
$ curl http://127.0.0.1:9080/anything -H "apikey: auth-one" -v
* Uses proxy env variable no_proxy == '192.168.49.2,localhost,127.0.0.1'
*   Trying 127.0.0.1:9080...
* Connected to 127.0.0.1 (127.0.0.1) port 9080
> GET /anything HTTP/1.1
> Host: 127.0.0.1:9080
> User-Agent: curl/8.4.0
> Accept: */*
> apikey: auth-one
>
< HTTP/1.1 500 Internal Server Error
< Date: Mon, 15 Jan 2024 09:25:54 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 174
< Connection: close
< Server: APISIX/3.2.2
<
<html>
<head><title>500 Internal Server Error</title></head>
<body>
<center><h1>500 Internal Server Error</h1></center>
<hr><center>openresty</center>
</body>
</html>
* Closing connection

Environment

• APISIX version (run apisix version): https://github.com/apache/apisix/tree/release/3.2
• Operating system (run uname -a):
• OpenResty / Nginx version (run openresty -V or nginx -V):
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[Vacant2333]

hello! do u want to help to fix this issue and raise a pr @ronething , it looks like was a bug now

[ronething]

do u want to help to fix this issue and raise a pr

ok, I will fix it when i have free time.

[Vacant2333]

do u want to help to fix this issue and raise a pr

ok, I will fix it when i have free time.

thanks, @shreemaan-abhishek can u help assign himself

[hanqingwu]

I have check this issue and I find when etcd delete item , the values array set to false instead of remove it ,

So I suggest add check in create_secret_kvs whether v is false

[shreemaan-abhishek]

@hanqingwu would you like to fix this?

[hanqingwu]

OK, I will submit a PR

[ronething]

So I suggest add check in create_secret_kvs whether v is false

I don't recommend this solution, maybe we can use secret_values:get(manager .. "/" .. confid) to get secret in secret_kv function.

[monkeyDluffy6017]

@ronething Do you have any idea about this pr: https://github.com/apache/apisix/pull/10902, please show your comments