help request: Apisix 404 route not found

[1024-zzw]

Description

1.首先通过dashboard界面创建了Apisix路由；并且通过Apisix-admin接口以及在etcd中均可以查询到相关路由信息。但是通过 Apisix的域名去访问这个路由的时候却是报404 route not found。 2.检查Apisix后面的etcd发现报错： 2022-07-29 13:09:09.411077 W | etcdserver: read-only range request "key:"/apisix-ew/routes/good-shop" range_end:"/apisix-ew/routes/good-shop" " with result "range_response_count:1 size:733" took too long (1.350541197s) to execute 2022-07-29 13:09:10.295095 W | etcdserver: request "header:<ID:8694339545952555948 > put:<key:"/apisix-ew/routes/good-shop" value_size:647 prev_kv:true >" with result "size:731" took too long (878.022818ms) to execute 2022-07-29 13:09:10.295267 W | etcdserver: read-only range request "key:"/apisix-ns/phantomkey" range_end:"/apisix-ns/phantomkez" " with result "range_response_count:0 size:6" took too long (529.675569ms) to execute 2022-07-29 13:09:12.203322 W | etcdserver: request "header:<ID:8694339545952555971 > put:<key:"/apisix-ew/routes/good-shop" value_size:657 prev_kv:true >" with result "size:743" took too long (1.692485533s) to execute 2022-07-29 13:09:12.205162 W | etcdserver: read-only range request "key:"/apisix-ew/phantomkey" range_end:"/apisix-ew/phantomkez" " with result "range_response_count:0 size:6" took too long (580.109896ms) to execute 2022-07-29 13:09:12.209027 W | etcdserver: read-only range request "key:"/apisix-ns/phantomkey" range_end:"/apisix-ns/phantomkez" " with result "range_response_count:0 size:6" took too long (532.260776ms) to execute 3.在各个环境当中均出现了这个问题，一个解决方案是通过重启etcd，给etcd升级配置，而后apisix貌似是恢复了。 4.现象是新增加的路由会报404 route not found；但是旧的路由仍然可以进行访问。

Environment

• APISIX version (run apisix version):
• Operating system (run uname -a):
• OpenResty / Nginx version (run openresty -V or nginx -V):
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[tokers]

1. Try to show us the route configuration, not just say the result.
2. It's not easy to troubleshoot the ETCD slow request via the error log, we also need to know the APISIX config.yaml and your ETCD cluster status, system resources and so on.

[tzssangglass]

What version of etcd are you using?

For lower versions of etcd, you can refer to: https://github.com/apache/apisix/issues/5002, you can choose a higher version of etcd, such as 3.5.x.

If it is a low version of APISIX, then it may be related to the server-info plugin and you can upgrade APISIX. ref: https://github.com/apache/apisix/issues/5723

[1024-zzw]

1.路由相关配置{ "uri": "/test-boot/", "name": "TEST-BOOT", "host": "fra-apisix-gateway.pre.com", "plugins": { "proxy-rewrite": { "regex_uri": [ "^/test-boot/(.)", "/${1}" ] } }, "plugin_config_id": "1", "upstream": { "timeout": { "connect": 6, "send": 6, "read": 6 }, "type": "roundrobin", "scheme": "http", "discovery_type": "eureka", "pass_host": "pass", "service_name": "TEST-BOOT", "keepalive_pool": { "idle_timeout": 60, "requests": 1000, "size": 320 } }, "status": 1 }

2.apisix相关配置config.yaml apiVersion: v1 data: config.yaml: |- apisix: admin_key: - key: xxx name: admin role: admin - key: xxx name: viewer role: viewer allow_admin: - 127.0.0.1/24 - 0.0.0.0/0 config_center: etcd dns_resolver_valid: 30 enable_admin: true enable_admin_cors: true enable_debug: false enable_dev_mode: false enable_heartbeat: true enable_ipv6: true enable_reuseport: true node_listen: 9080 port_admin: 9180 proxy_cache: cache_ttl: 10s zones: - cache_levels: "1:2" disk_path: /dev/shm/mycache disk_size: 1G memory_size: 50m name: mycache resolver_timeout: 5 router: http: radixtree_uri ssl: radixtree_sni ssl: enable: false enable_http2: true listen_port: 9443 ssl_ciphers: ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA ssl_protocols: TLSv1 TLSv1.1 TLSv1.2 TLSv1.3 discovery: eureka: fetch_interval: 15 host: - http://xxxx:8761 - http://xxxx:8761 prefix: /eureka/ timeout: connect: 2000 read: 5000 send: 2000 weight: 100 etcd: host: - http://172.30.xxx.xxx:2379 prefix: /apisix timeout: 30 nginx_config: error_log: /dev/stderr error_log_level: warn event: worker_connections: 65535 http: access_log: /dev/stdout access_log_format: $remote_addr\t$remote_port\t$remote_user\t[$time_local]\t$request\t$http_host\t$status\t$request_length\t$body_bytes_sent\t"$http_referer"\t"$http_user_agent"\t"$http_x_forwarded_for"\t$request_time\t$upstream_response_time\t$ssl_client_s_dn\t$ssl_cipher\t"$proxy_add_x_forwarded_for" access_log_format_escape: default client_body_timeout: 60s client_header_timeout: 60s enable_access_log: true keepalive_timeout: 60s real_ip_from: - 127.0.0.1 - 'unix:' real_ip_header: X-Real-IP send_timeout: 10s underscores_in_headers: "on" http_configuration_snippet: | client_header_buffer_size 4k; client_body_buffer_size 512k; large_client_header_buffers 4 32k; sendfile on; tcp_nopush on; tcp_nodelay on; proxy_connect_timeout 600; proxy_read_timeout 600; proxy_send_timeout 600; proxy_buffer_size 64k; proxy_buffers 4 64k; proxy_busy_buffers_size 128k; proxy_temp_file_write_size 128k; gzip on; gzip_min_length 1k; gzip_buffers 4 16k; gzip_http_version 1.0; gzip_comp_level 2; gzip_types text/plain text/javascript application/javascript application/x-javascript text/css application/xml; gzip_vary on; worker_rlimit_nofile: 365535 plugin_attr: prometheus: export_addr: ip: 0.0.0.0 port: 9091 export_uri: /apisix/prometheus/metrics metric_prefix: apisix_ plugins: - api-breaker - authz-keycloak - basic-auth - batch-requests - consumer-restriction - cors - echo - fault-injection - grpc-transcode - hmac-auth - http-logger - ip-restriction - ua-restriction - jwt-auth - kafka-logger - key-auth - limit-conn - limit-count - limit-req - node-status - openid-connect - authz-casbin - prometheus - proxy-cache - proxy-mirror - proxy-rewrite - redirect - referer-restriction - request-id - request-validation - response-rewrite - serverless-post-function - serverless-pre-function - server-info - sls-logger - syslog - tcp-logger - udp-logger - uri-blocker - wolf-rbac - zipkin - traffic-split - gzip - real-ip - public-api - forward-auth stream_plugins: - mqtt-proxy - ip-restriction - limit-conn 3.etcd版本：etcd Version: 3.4.18

[tzssangglass]

Can you find this route in etcd (not via dashboard)?

[tzssangglass]

Excluding the ETCD problem (3.4.18 is OK), I think it may be the server-info plugin, upgrade APISIX to the latest version.

[1024-zzw]

可以在etcd中找到。 目前我们通过对etcd的数据进行压缩之后，目前apisix好像是恢复正常了。

[tzssangglass]

目前我们通过对etcd的数据进行压缩之后，目前apisix好像是恢复正常了。

From experience, this was a problem caused by the server-info plugin, which has been fixed in the new version.

[github-actions[bot]]

This issue has been marked as stale due to 350 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions.

[github-actions[bot]]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.