apisix-ingress-controller
apisix-ingress-controller copied to clipboard
apache
bug: Syncing an ApisixRoute object fails due to an unknown plugin error related to opentelemetry.
Current Behavior
The sync process fails and retries repeatedly due to the unknown plugin [opentelemetry] error.
Expected Behavior
The opentelemetry plugin should be recognized and correctly handled during the sync process.
Error Logs
ingress-apisix-composite-deployment-657758f575-59zs6 ingress-apisix 2025-01-08T03:49:46+08:00 warn apisix/apisix_route.go:481 sync ApisixRoute failed, will retry {"object": {"Type":1,"Object":{"Key":"internal-services/webhook-site","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "1 error occurred:\n\t* unknown plugin [opentelemetry]\n\n"}
Steps to Reproduce
Deploy an ApisixRoute object referencing the opentelemetry plugin.
---
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
name: apisixroute-test-discovery
namespace: default
spec:
discovery:
serviceName: default/apisixroute-test-svc:http
type: kubernetes
loadbalancer:
hashOn: vars
key: remote_addr
type: roundrobin
scheme: http
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
name: apisixroute-test
namespace: default
spec:
http:
- match:
hosts:
- apisix-test.domain.net
methods:
- GET
- HEAD
- PUT
- POST
- PATCH
- DELETE
- OPTIONS
paths:
- /*
name: mainz
plugins:
- config:
sampler:
name: always_on
enable: true
name: opentelemetry
upstreams:
- name: apisixroute-test-discovery
weight: 100
...
Monitor the logs of the ingress-apisix pod.
ingress-apisix-composite-deployment-657758f575-59zs6 ingress-apisix 2025-01-08T03:49:46+08:00 error apisix/apisix_route.go:379 failed to sync ApisixRoute to apisix {"error": "1 error occurred:\n\t* unknown plugin [opentelemetry]\n\n"}
Observe the reported error regarding the unknown plugin.
The current deployed composite.yaml
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
apiVersion: v1
kind: Namespace
metadata:
name: apisix
labels:
name: apisix
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: apisix-view-clusterrole
rules:
- apiGroups:
- ""
resources:
- events
verbs:
- "*"
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- persistentvolumeclaims
- pods
- replicationcontrollers
- replicationcontrollers/scale
- serviceaccounts
- services
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- bindings
- limitranges
- namespaces/status
- pods/log
- pods/status
- replicationcontrollers/status
- resourcequotas
- resourcequotas/status
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- namespaces
verbs:
- get
- list
- watch
- apiGroups:
- apps
resources:
- controllerrevisions
- daemonsets
- deployments
- deployments/scale
- replicasets
- replicasets/scale
- statefulsets
- statefulsets/scale
verbs:
- get
- list
- watch
- apiGroups:
- autoscaling
resources:
- horizontalpodautoscalers
verbs:
- get
- list
- watch
- apiGroups:
- batch
resources:
- cronjobs
- jobs
verbs:
- get
- list
- watch
- apiGroups:
- extensions
resources:
- daemonsets
- deployments
- deployments/scale
- ingresses
- networkpolicies
- replicasets
- replicasets/scale
- replicationcontrollers/scale
verbs:
- get
- list
- watch
- apiGroups:
- policy
resources:
- poddisruptionbudgets
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
- ingresses/status
- networkpolicies
verbs:
- "*"
- apiGroups:
- metrics.k8s.io
resources:
- pods
verbs:
- get
- list
- watch
- apiGroups:
- apisix.apache.org
resources:
- apisixroutes
- apisixroutes/status
- apisixupstreams
- apisixupstreams/status
- apisixtlses
- apisixtlses/status
- apisixclusterconfigs
- apisixclusterconfigs/status
- apisixconsumers
- apisixconsumers/status
- apisixpluginconfigs
- apisixpluginconfigs/status
- apisixglobalrules
- apisixglobalrules/status
verbs:
- "*"
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- "*"
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- get
- list
- watch
- apiGroups:
- gateway.networking.k8s.io
resources:
- httproutes
- tlsroutes
- tcproutes
- gateways
- gatewayclasses
- udproutes
verbs:
- get
- list
- watch
- apiGroups:
- gateway.networking.k8s.io
resources:
- gateways/status
- gatewayclasses/status
verbs:
- get
- update
- list
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: apisix-view-serviceaccount
namespace: apisix
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: apisix-view-clusterrolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: apisix-view-clusterrole
subjects:
- kind: ServiceAccount
name: apisix-view-serviceaccount
namespace: apisix
---
apiVersion: v1
kind: ConfigMap
metadata:
name: apisix-gw-config.yaml
namespace: apisix
data:
config.yaml: >
deployment:
admin:
allow_admin:
- 127.0.0.0/24
- 0.0.0.0/0
admin_listen:
ip: 0.0.0.0
port: 9180
etcd:
host:
- "http://127.0.0.1:12379"
prefix: "/apisix"
timeout: 60
apisix:
enable_control: true
enable_reuseport: true
stream_proxy:
only: false
tcp:
- 9100
- addr: 9110
tls: true
udp:
- 9200
plugins:
- real-ip
- ai
- client-control
- proxy-control
- request-id
- zipkin
- opentelemetry
- ext-plugin-pre-req
- fault-injection
- mocking
- serverless-pre-function
- batch-requests
- cors
- ip-restriction
- ua-restriction
- referer-restriction
- csrf
- uri-blocker
- request-validation
- openid-connect
- cas-auth
- authz-casbin
- authz-casdoor
- wolf-rbac
- ldap-auth
- hmac-auth
- basic-auth
- jwt-auth
- key-auth
- consumer-restriction
- forward-auth
- opa
- authz-keycloak
- proxy-mirror
- proxy-cache
- proxy-rewrite
- workflow
- api-breaker
- limit-conn
- limit-count
- limit-req
- node-status
- gzip
- traffic-split
- redirect
- response-rewrite
- kafka-proxy
- grpc-transcode
- grpc-web
- public-api
- prometheus
- datadog
- elasticsearch-logger
- echo
- loggly
- http-logger
- splunk-hec-logging
- skywalking-logger
- google-cloud-logging
- sls-logger
- tcp-logger
- kafka-logger
- rocketmq-logger
- syslog
- udp-logger
- file-logger
- clickhouse-logger
- tencent-cloud-cls
- inspect
- aws-lambda
- azure-functions
- openwhisk
- openfunction
- serverless-post-function
- ext-plugin-post-req
- ext-plugin-post-resp
plugin_attr:
prometheus:
enable_export_server: false
opentelemetry:
resource:
service.name: APISIX
collector:
address: 127.0.0.1:4318
batch_span_processor:
drop_on_queue_full: true
batch_timeout: 10
real-ip:
source: http_x_forwarded_for
trusted_addresses:
# Broad representation of all internal AWS IPs
- "10.0.0.0/8"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ingress-apisix-composite-deployment
namespace: apisix
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: ingress-apisix-composite-deployment
template:
metadata:
labels:
app.kubernetes.io/name: ingress-apisix-composite-deployment
spec:
volumes:
- name: apisix-config-yaml-configmap
configMap:
name: apisix-gw-config.yaml
defaultMode: 420
containers:
- livenessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 15
timeoutSeconds: 2
periodSeconds: 2
successThreshold: 1
failureThreshold: 3
readinessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 10
timeoutSeconds: 2
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
name: ingress-apisix
image: apache/apisix-ingress-controller:1.7.0
command:
- /ingress-apisix/apisix-ingress-controller
- ingress
- "--log-output"
- stdout
- "--apisix-resource-sync-interval"
- 1h
- "--apisix-resource-sync-comparison=true"
- "--http-listen"
- ":8080"
- "--https-listen"
- ":8443"
- "--default-apisix-cluster-name"
- default
- "--default-apisix-cluster-base-url"
- http://127.0.0.1:9180/apisix/admin
- "--default-apisix-cluster-admin-key"
- edd1c9f034335f136f87ad84b625c8f1
- "--api-version"
- apisix.apache.org/v2
- "--ingress-status-address"
- ""
- "--disable-status-updates=false"
- "--etcd-server-enabled=true"
ports:
- name: http
containerPort: 8080
protocol: TCP
- name: https
containerPort: 8443
protocol: TCP
- name: etcd
containerPort: 12379
protocol: TCP
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
- livenessProbe:
failureThreshold: 3
initialDelaySeconds: 20
periodSeconds: 2
successThreshold: 1
tcpSocket:
port: 9080
timeoutSeconds: 2
readinessProbe:
failureThreshold: 3
initialDelaySeconds: 15
periodSeconds: 2
successThreshold: 1
tcpSocket:
port: 9080
timeoutSeconds: 1
name: apisix
image: apache/apisix:3.2.2-centos
ports:
- name: http
containerPort: 9080
protocol: TCP
- name: http-admin
containerPort: 9180
protocol: TCP
- name: https
containerPort: 9443
protocol: TCP
resources: {}
volumeMounts:
- name: apisix-config-yaml-configmap
mountPath: /usr/local/apisix/conf/config.yaml
subPath: config.yaml
restartPolicy: Always
terminationGracePeriodSeconds: 0
dnsPolicy: ClusterFirst
serviceAccountName: apisix-view-serviceaccount
serviceAccount: apisix-view-serviceaccount
securityContext: {}
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 50%
revisionHistoryLimit: 10
progressDeadlineSeconds: 600
---
apiVersion: v1
kind: Service
metadata:
name: ingress-apisix-gateway
namespace: apisix
spec:
ports:
- name: http
protocol: TCP
port: 80
targetPort: 9080
- name: https
protocol: TCP
port: 443
targetPort: 9443
selector:
app.kubernetes.io/name: ingress-apisix-composite-deployment
type: NodePort
Environment
- Ingress APISIX Version: apache/apisix:3.2.2-centos, apache/apisix:3.11.0-redhat
- Kubernetes Version: v1.30.2-eks-1552ad0
- APISIX Plugins Configured: opentelemetry
- Deployment Method: Composite Architecture
We are currently investigating the issue and noticed that the OpenTelemetry configuration is missing from the apisix-schema.json file.
Could you please confirm if this is an intentional omission or if OpenTelemetry support is expected to be included in this schema? Additionally, any guidance on the recommended approach for adding OpenTelemetry configuration would be appreciated.
Thank you for your assistance!
This issue has been marked as stale due to 90 days of inactivity. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.