Metrics from pulsar proxy are not scraped because /metrics when authentication is enabled

[pinkavaj]

When authentication is enabled on Pulsar, proxy enables authentication even on /metrics endpoint. This might be turned off by authenticateMetricsEndpoint configuratin option, but is enabled by default (due to security considerations).

Expected behavior

Allow setting the authenticateMetricsEndpoint option in helm chart.

[nealf]

For temporary reference, this can can be done by adding PULSAR_PREFIX_authenticateMetricsEndpoint: "false" to the proxy.configData value. To keep the default config, including what's in the values.yaml file (as of helm chart version 2.7.10), I ended up with:

proxy:
    configData:
        PULSAR_MEM: >
          -Xms64m -Xmx64m -XX:MaxDirectMemorySize=64m
        PULSAR_GC: >
          -XX:+UseG1GC
          -XX:MaxGCPauseMillis=10
          -Dio.netty.leakDetectionLevel=disabled
          -Dio.netty.recycler.linkCapacity=1024
          -XX:+ParallelRefProcEnabled
          -XX:+UnlockExperimentalVMOptions
          -XX:+AggressiveOpts
          -XX:+DoEscapeAnalysis
          -XX:ParallelGCThreads=4
          -XX:ConcGCThreads=4
          -XX:G1NewSizePercent=50
          -XX:+DisableExplicitGC
          -XX:-ResizePLAB
          -XX:+ExitOnOutOfMemoryError
          -XX:+PerfDisableSharedMem
        PULSAR_PREFIX_authenticateMetricsEndpoint: "false"

[youg9203]

        PULSAR_PREFIX_authenticateMetricsEndpoint: "false"

I need to understand, is this parameter is working? Or did you just provide the suggestion for the solution?

I'm also facing this issue now, and love to hear any other way to fix the issue.

[michaeljmarshall]

        PULSAR_PREFIX_authenticateMetricsEndpoint: "false"

I need to understand, is this parameter is working? Or did you just provide the suggestion for the solution?

I'm also facing this issue now, and love to hear any other way to fix the issue.

Yes, this is the correct solution. Whenever a configuration is not present in the conf/proxy.conf (or any of the .conf files), you must add the PULSAR_PREFIX_ to the start of the configuration value. By doing so, the configuration script will map the suffix into the .conf file.

https://github.com/apache/pulsar/blob/0c694cfdc9b11f915f1da86260ad3655c2e99a35/docker/pulsar/scripts/apply-config-from-env.py#L37

Note that in the case of functions, the prefix is actually PF_. Note that the worker config files are .yml.

https://github.com/apache/pulsar/blob/0c694cfdc9b11f915f1da86260ad3655c2e99a35/docker/pulsar/scripts/gen-yml-from-env.py#L55

You can see here that we run the bin/apply-config-from-env.py script before starting the proxy process.

https://github.com/apache/pulsar-helm-chart/blob/cee3fcfe569e2a8cc87fb5f8046b29f47f1d4e39/charts/pulsar/templates/proxy-statefulset.yaml#L184

If you'd like to avoid adding the PULSAR_PREFIX_, the best solution is to open a PR in the apache/pulsar project and add the config, in this case authenticateMetricsEndpoint to the proxy conf. Then, a prefix won't be needed for this specific configuration.

[youg9203]

Yeah, that worked, thanks