[AGENTONB-2676] Refactor secret refreshes

[Eokye]

What does this PR do?

A brief description of the change being made with this pull request.

Refactor secret refreshes to remove client recreation (which doesn't need to happen). Also removed callback system since all that needs to be remade is the auth. Auth is now remade on every reconcile and passed down as a parameter, instead of a state.

NOTE: Currently apiURL is kept as a field inside the reconcilers. This might require a change once creds.go has been refactored.

Motivation

What inspired you to submit this pull request?

Additional Notes

Anything else we should know when reviewing?

Minimum Agent Versions

Are there minimum versions of the Datadog Agent and/or Cluster Agent required?

• Agent: vX.Y.Z
• Cluster Agent: vX.Y.Z

Describe your test plan

Write there any instructions and details you may have to test your PR.

1. Create a backend script that rotates keys every so often. Here's an example:

#!/bin/bash

# Predefined API keys to rotate through (Replace with actual API/APP Keys)
API_KEYS=(
  "API_KEY1"
  "API_KEY2"  
)

APP_KEYS=(
  "APP_KEY1"
  "APP_KEY2"
)

# Time-based rotation: change every minute for testing, cycle through array  
MINUTE=$(date +%M)
API_INDEX=$((MINUTE % ${#API_KEYS[@]}))
APP_INDEX=$((MINUTE % ${#APP_KEYS[@]}))

# Read JSON input from stdin (required by secret backend protocol)
read -r INPUT

# Output JSON in the required format
cat <<EOF
{"api-key":{"value":"${API_KEYS[$API_INDEX]}"},"app-key":{"value":"${APP_KEYS[$APP_INDEX]}"}}
EOF

2. Add a volume mount to the manager container with the name of the script as your path. We will be making a configmap out of it. Add volumeMount to manager.yaml

        volumeMounts:
        - name: secret-script
          mountPath: /usr/local/bin/secret/secret_backend.sh
          subPath: secret_backend.sh

and volume to pod spec

      volumes:
      - name: secret-script
        configMap:
          name: secret-backend-script
          defaultMode: 0755

3. You should have your kind cluster created already. Create a configmap from your secret_backend script. kubectl create configmap secret-backend-script --from-file=secret_backend.sh

4. Enable all custom resources (or whichever custom resource you want to test client recreation with).

5. Add API/APP key in env vars. Make sure it is encrypted:

        - name: DD_API_KEY
          value: "ENC[api-key]"
        - name: DD_APP_KEY
          value: "ENC[app-key]"

6. make deploy with this configuration in manager.yaml (feel free to change refreshInterval; it's short for testing purposes).

       - --secretBackendCommand=/usr/local/bin/secret/secret_backend.sh
       - --secretRefreshInterval=30s

if the operator fails to find the configMap, run 4. again and delete the pod.

7. Deploy your custom resources you want to manage: create, delete, update to see if they work. Also, both APIs given should be utilized (you can see this in the API keys of datadog).

Checklist

• [x] PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
• [x] PR has a milestone or the qa/skip-qa label

[codecov-commenter]

Codecov Report

:x: Patch coverage is 28.97196% with 152 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 37.34%. Comparing base (3fa031c) to head (73ae7d8).

Files with missing lines	Patch %	Lines
pkg/datadogclient/client.go	0.00%	37 Missing :warning:
internal/controller/datadogmonitor/controller.go	21.73%	15 Missing and 3 partials :warning:
...al/controller/datadoggenericresource/controller.go	26.08%	16 Missing and 1 partial :warning:
...al/controller/datadoggenericresource/synthetics.go	0.00%	16 Missing :warning:
internal/controller/setup.go	6.25%	15 Missing :warning:
internal/controller/datadogdashboard/controller.go	30.00%	13 Missing and 1 partial :warning:
internal/controller/datadogslo/controller.go	40.90%	11 Missing and 2 partials :warning:
...rnal/controller/datadoggenericresource/monitors.go	0.00%	8 Missing :warning:
...nal/controller/datadoggenericresource/notebooks.go	25.00%	6 Missing :warning:
pkg/config/creds.go	61.53%	5 Missing :warning:
... and 3 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2312      +/-   ##
==========================================
- Coverage   37.65%   37.34%   -0.31%     
==========================================
  Files         258      258              
  Lines       21710    21647      -63     
==========================================
- Hits         8174     8085      -89     
- Misses      12903    12933      +30     
+ Partials      633      629       -4     

Flag	Coverage Δ
unittests	37.34% <28.97%> (-0.31%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
internal/controller/datadogdashboard/finalizer.go	66.66% <100.00%> (ø)
...nal/controller/datadoggenericresource/finalizer.go	76.66% <100.00%> (ø)
...nternal/controller/datadoggenericresource/utils.go	89.09% <100.00%> (ø)
internal/controller/datadogmonitor/finalizer.go	67.74% <100.00%> (ø)
internal/controller/datadogmonitor_controller.go	36.84% <100.00%> (+3.50%)	:arrow_up:
internal/controller/datadogdashboard_controller.go	0.00% <0.00%> (ø)
...al/controller/datadoggenericresource_controller.go	0.00% <0.00%> (ø)
internal/controller/datadogslo_controller.go	0.00% <0.00%> (ø)
pkg/config/creds.go	71.23% <61.53%> (-11.90%)	:arrow_down:
...nal/controller/datadoggenericresource/notebooks.go	10.90% <25.00%> (ø)
... and 8 more

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 3fa031c...73ae7d8. Read the comment docs.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.