[AGENTONB-2664] Generate Certificate Authority for Agent pods

[Eokye]

What does this PR do?

A brief description of the change being made with this pull request.

Motivation

What inspired you to submit this pull request?

Additional Notes

Anything else we should know when reviewing?

Minimum Agent Versions

Are there minimum versions of the Datadog Agent and/or Cluster Agent required?

• Agent: vX.Y.Z
• Cluster Agent: vX.Y.Z

Describe your test plan

Write there any instructions and details you may have to test your PR. Add the following env vars to your agent manifest and apply:

    env:
      - name: DD_CLUSTER_TRUST_CHAIN_ENABLE_TLS_VERIFICATION
        value: "true"
      - name: DD_CLUSTER_TRUST_CHAIN_CA_CERT_FILE_PATH
        value: "/etc/datadog-agent/certificates/tls.crt"
      - name: DD_CLUSTER_TRUST_CHAIN_CA_KEY_FILE_PATH
        value: "/etc/datadog-agent/certificates/tls.key"

1. Check that volumes are mounted on pods and agent container. To be extra thorough for node-agent, you could enable all the relevant features to see if all agent containers have the correct mount path: /etc/datadog-agent/certificates from certificates:

    npm: 
      enabled: true
    liveProcessCollection:
      enabled: true
    liveContainerCollection:
      enabled: true
    otelCollector:
      enabled: true
    apm:
      enabled: true
      hostPortConfig:
         enabled: true
         hostPort: 8126
    cws:
      enabled: true

2. Get logs from node agent k logs... and see if "Successfully connected to Datadog Cluster Agent" is in the logs.
3. exec into the cluster agent pod and run

openssl s_client -connect datadog-cluster-agent:5005 \
  -CAfile /etc/datadog-agent/certificates/tls.crt \
  -showcerts

Service certificates should have Datadog Agent CA as the issuer. You should see this in the service certificates:

Server certificate
subject=O = "Datadog, Inc."
issuer=O = Datadog, CN = Datadog Agent CA

Checklist

• [x] PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
• [x] PR has a milestone or the qa/skip-qa label

[codecov-commenter]

Codecov Report

:x: Patch coverage is 3.22581% with 90 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 38.25%. Comparing base (c3c51b1) to head (bc6762b). :warning: Report is 17 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/certificates/cert_manager.go	0.00%	76 Missing :warning:
...controller/datadogagent/component/agent/default.go	14.28%	6 Missing :warning:
internal/controller/datadogagent/common/volumes.go	0.00%	3 Missing :warning:
...controller/datadogagent/controller_reconcile_v2.go	40.00%	2 Missing and 1 partial :warning:
...adogagent/component/clusterchecksrunner/default.go	0.00%	2 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2304      +/-   ##
==========================================
- Coverage   38.38%   38.25%   -0.14%     
==========================================
  Files         255      259       +4     
  Lines       21113    22561    +1448     
==========================================
+ Hits         8104     8630     +526     
- Misses      12383    13285     +902     
- Partials      626      646      +20     

Flag	Coverage Δ
unittests	38.25% <3.22%> (-0.14%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...adogagent/component/clusterchecksrunner/default.go	10.52% <0.00%> (-0.12%)	:arrow_down:
internal/controller/datadogagent/common/volumes.go	0.00% <0.00%> (ø)
...controller/datadogagent/controller_reconcile_v2.go	57.29% <40.00%> (-0.49%)	:arrow_down:
...controller/datadogagent/component/agent/default.go	8.44% <14.28%> (+1.64%)	:arrow_up:
pkg/certificates/cert_manager.go	0.00% <0.00%> (ø)

... and 15 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update c3c51b1...bc6762b. Read the comment docs.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.