Feature Request: Better support for retries

[JavaScriptBach]

I want to use this action in my deployment pipeline to prevent unsafe pushes, the idea being we will run all synthetics with the tag blocks_prod_deploy before a code push and verify they pass before proceeding. However, I'm running into issues from flaky synthetics. Sometimes this action will fail due to a synthetic "timing out", and the linked test run will show that the first run failed but a second configured fast retry succeeded. Example:

Could we add some better handling of retries? Some ideas:

• The action respects the retry configuration of the synthetic and retries as needed before reporting failure.
• The action takes in an input letting the user choose how long to wait for each synthetic.
• The action reports the IDs of the failed synthetics as an output so the user can retry them as needed.

Right now, with none of these options implemented, I have to retry all synthetics, which is time-consuming.

[m-rousse]

Hi @JavaScriptBach, Thanks for the suggestion. Could you share the configuration you are using with the action?

[JavaScriptBach]

      - name: Checkout repo
        uses: actions/checkout@93ea575cb5d8a053eaa0ac8fa3b40d7e05a33cc8 # v3.1.0
      - name: Run Datadog Synthetics tests
        uses: DataDog/synthetics-ci-github-action@848f9e3982cfbd4c925e79a3578b0f6da97a7d97 # v0.6.0
        with:
          api_key: ${{ secrets.DATADOG_API_KEY }}
          app_key: ${{ secrets.DATADOG_APP_KEY }}
          test_search_query: tag:blocks_prod_deploy

[Drarig29]

👋 Hi! We made several changes in the last months which should have fixed this kind of behavior.