fix: try fixing E2E `crosschain_swap` failure

[ws4charlie]

Description

The PR tries to fix this E2E test crosschain_swap failure: https://github.com/zeta-chain/node/issues/2089 The failure can't be easily replicated and the reason remains unknown. Let's see if it happen again in the future after merge.

Closes: <PD-XXXX>

Type of change

• [ ] Bug fix (non-breaking change which fixes an issue)
• [ ] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [ ] This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Include instructions and any relevant details so others can reproduce.

• [ ] Tested CCTX in localnet
• [ ] Tested in development environment
• [ ] Go unit tests
• [ ] Go integration tests
• [ ] Tested via GitHub Actions

Checklist:

• [ ] I have added unit tests that prove my fix feature works

[gartnera]

The failure can't be easily replicated and the reason remains unknown. Let's see if it happen again in the future after merge.

https://github.com/zeta-chain/node/actions/runs/9169683804/job/25210551669 has a full log dump from all containers if that is useful. click "view raw logs" in the settings/gear menu for the best results

We need way to just run the e2e tests repeatedly until failure. #2264 helps a bit with that. You could do something like this:

#!/bin/bash

set -e

while true; do
  make start-e2e-test
  docker logs -f orchestrator &
  exit_code=$(docker wait orchestrator)
  if [[ $exit_code -ne 0 ]]; then
    exit $exit_code
  fi
  make stop-test
done

[lumtis]

Started some upgrade tests to see https://github.com/zeta-chain/node/actions/workflows/execute_advanced_tests.yaml

If all 5 pass, I think we can safely assume this is good

UPDATE: not solving the issue

[lumtis]

The failure can't be easily replicated and the reason remains unknown. Let's see if it happen again in the future after merge.

The issue happen very frequently with the upgrade test if this can help.

A difference you have with the upgrade test is that the local Bitcoin node will have mined lot of blocks by the time you start running the §crosschain_swap test unlike the regular test