Nginx instrumentation generate multi propogation header when request header count >= 20

[wmingj]

Describe your environment OpentelemetryOperator: 0.99.0 OtelWebserverMoudle: 1.0.4

We are using the nginx instrumentation moudle of otel-webserver-module in our environment. But we found the module will generate multi propogation header(traceparent and tracestate), when request header's count is larger than 20.

Steps to reproduce

1. install the latest opentelemetry-operator
2. apply the nginx instrumentation yaml: 00-install-instrumentation.yaml refer: nginx instrumentation testcase

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: nginx
spec:
  exporter:
    endpoint: http://localhost:4317
  propagators:
    - jaeger
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "0.25"
  nginx:
    attrs:
    - name: NginxModuleOtelMaxQueueSize
      value: "4096"

3. the nginx deployment: 01-install-app.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
spec:
  selector:
    matchLabels:
      app: my-nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: my-nginx
      annotations:
        #sidecar.opentelemetry.io/inject: "true"
        #instrumentation.opentelemetry.io/inject-nginx: "true"
        instrumentation.opentelemetry.io/inject-nginx: default/nginx
    spec:
      securityContext:
        runAsUser: 1000
        runAsGroup: 3000
        fsGroup: 3000
      containers:
      - name: myapp
        image: nginxinc/nginx-unprivileged:1.25.3
        imagePullPolicy: Always
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]
        # following to test lifecycle removal in cloned init container
        ports:
        - containerPort: 8765
        env:
        - name: LD_LIBRARY_PATH
          value: /opt
        volumeMounts:
          - name: nginx-conf
            mountPath: /etc/nginx/nginx.conf
            subPath: nginx.conf
            readOnly: true
        imagePullPolicy: Always
        resources:
          limits:
            cpu: "1"
            memory: 500Mi
          requests:
            cpu: 250m
            memory: 100Mi
      volumes:
      - name: nginx-conf
        configMap:
          name: nginx-conf
          items:
            - key: nginx.conf
              path: nginx.conf

---

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-conf
data:
  nginx.conf: |
    # user nginx;
    worker_processes  1;
    pid /tmp/nginx.pid;
    events {
      worker_connections  10240;
    }
    http {
      include /etc/nginx/conf.d/*.conf;
      client_body_temp_path /tmp/client_temp;
      proxy_temp_path       /tmp/proxy_temp_path;
      fastcgi_temp_path     /tmp/fastcgi_temp;
      uwsgi_temp_path       /tmp/uwsgi_temp;
      scgi_temp_path        /tmp/scgi_temp;
      server {
        listen       8765;
        server_name  localhost;
        location / {
          root   /usr/share/nginx/html;
          index  index.html index.htm;
        }

        location /api/ {
          proxy_pass http://echo.default:80/api/;
        }
      }
    }

---

4. the upstream echo server:

➜  kong cat echo-service.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app: echo
  name: echo
spec:
  ports:
  - port: 8080
    name: high
    protocol: TCP
    targetPort: 8080
  - port: 80
    name: low
    protocol: TCP
    targetPort: 8080
  selector:
    app: echo
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: echo
  name: echo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: echo
    spec:
      containers:
      - image: gcr.io/kubernetes-e2e-test-images/echoserver:2.2
        name: echo
        ports:
        - containerPort: 8080
        env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
        resources: {}

6. test:

• with over than 20 headers, key1 ~ key16 + 4 header(auto added)

 curl 'localhost:8765/api/hello' \
--header 'key1: value1' \
--header 'key2: value2' \
--header 'key3: value3' \
--header 'key4: value4' \
--header 'key5: value5' \
--header 'key6: value6' \
--header 'key7: value7' \
--header 'key8: value8' \
--header 'key9: value9' \
--header 'key10: value10' \
--header 'key11: value11' \
--header 'key12: value12' \
--header 'key13: value13' \
--header 'key14: value14' \
--header 'key15: value15' \
--header 'key16: value16'

What is the expected behavior? The request send to echo server with only one propogation header

What is the actual behavior? The request send to echo server with two propogation header:

Hostname: echo-67456bbd77-lb9vc

Pod Information:
        node name:      minikube
        pod name:       echo-67456bbd77-lb9vc
        pod namespace:  default
        pod IP: 10.244.0.34

Server values:
        server_version=nginx: 1.12.2 - lua: 10010

Request Information:
        client_address=10.244.0.37
        method=GET
        real path=/api/hello
        query=
        request_version=1
        request_scheme=http
        request_uri=http://echo.default:8080/api/hello

Request Headers:
        accept=*/*
        connection=close
        host=echo.default
        key1=value1
        key10=value10
        key11=value11
        key12=value12
        key13=value13
        key14=value14
        key15=value15
        key16=value16
        key2=value2
        key3=value3
        key4=value4
        key5=value5
        key6=value6
        key7=value7
        key8=value8
        key9=value9
        traceparent=00-59bf9c6263e207f0e0396385591bdcd5-833345068aad29be-01
        traceparent=00-59bf9c6263e207f0e0396385591bdcd5-474530dbae9782b7-01
        tracestate=
        user-agent=curl/7.81.0

Request Body:
        -no body in request-

Additional context Please review my case, thanks! I have found the root cause which is related to header iteration of ngx_list_t and would be very like to make a PR

[marcalff]

Thanks for the report. PR are always welcome.