node icon indicating copy to clipboard operation
node copied to clipboard
verified publisher icon zeta-chain

feat: enable e2e orchestrator restart

Open gartnera opened this issue 1 year ago • 1 comments

Description

Enable restarting the e2e orchestrator by:

  • always use the --setup-only option followed by running with the --skip-setup option
  • skip setup if the deployed.yml (created by setup) exists

This enables you to rerun the e2e tests just by running docker restart orchestrator

Also fix the TestDappZEVM/TestDappEVM addresses. Otherwise you'll see something like this:

setup      | ✅ setup completed in 32.235828535s
setup      | ✅ config file written in /work/deployed.yml
setup      |  --- 📜zEVM contracts ---
setup      | SystemContract: 0x91d18e54DAf4F677cB28167158d6dd21F6aB3921
setup      | ETHZRC20:       0x13A0c5930C028511Dc02665E7285134B6d11A5f4
setup      | ERC20ZRC20:     0x48f80608B672DC30DC7e3dbBd0343c5F02C738Eb
setup      | BTCZRC20:       0xd97B1de3619ed2c6BEb3860147E30cA8A7dC9891
setup      | UniswapFactory: 0x9fd96203f7b22bCF72d9DCb40ff98302376cE09c
setup      | UniswapRouter:  0x2ca7d64A7EFE2D62A725E2B35Cf7230D6677FfEe
setup      | ConnectorZEVM:  0x239e96c8f17C85c30100AC26F635Ea15f23E9c67
setup      | WZeta:          0x5F0b1a82749cb4E2278EC87F8BF6B618dC71a8bf
setup      | ZEVMSwapApp:    0x733aB8b06DDDEf27Eaa72294B0d7c9cEF7f12db9
setup      | ContextApp:     0xD28D6A0b8189305551a0A8bd247a6ECa9CE781Ca
setup      | TestDappZEVM:       0xA8D5060feb6B456e886F023709A2795373691E63
setup      |  --- 📜EVM contracts ---
setup      | ZetaEth:        0xcd01737eaa83195514601E9F6df929Bd0709DC60
setup      | ConnectorEth:   0xfdA26637E11771E5919aF5c24A363c82613d62dA
setup      | ERC20Custody:   0x03Ce5CDd0aEC673B11a672df176E0F52E9191556
setup      | ERC20:      0x486d325e1D577f1fe6f4E1633E137f91758a8362
setup      | TestDappEVM:       0x28b64929f082F7361B9d9e681AaC3436dA4Cb857
setup      | ✅ the localnet has been setup
running e2e tests...
setup      | starting E2E tests
setup      | ⚙️ setting up TSS address
setup      | ⚙️ enabling verification flags for block headers
setup      |  --- 📜zEVM contracts ---
setup      | SystemContract: 0x91d18e54DAf4F677cB28167158d6dd21F6aB3921
setup      | ETHZRC20:       0x13A0c5930C028511Dc02665E7285134B6d11A5f4
setup      | ERC20ZRC20:     0x48f80608B672DC30DC7e3dbBd0343c5F02C738Eb
setup      | BTCZRC20:       0xd97B1de3619ed2c6BEb3860147E30cA8A7dC9891
setup      | UniswapFactory: 0x9fd96203f7b22bCF72d9DCb40ff98302376cE09c
setup      | UniswapRouter:  0x2ca7d64A7EFE2D62A725E2B35Cf7230D6677FfEe
setup      | ConnectorZEVM:  0x239e96c8f17C85c30100AC26F635Ea15f23E9c67
setup      | WZeta:          0x5F0b1a82749cb4E2278EC87F8BF6B618dC71a8bf
setup      | ZEVMSwapApp:    0x733aB8b06DDDEf27Eaa72294B0d7c9cEF7f12db9
setup      | ContextApp:     0xD28D6A0b8189305551a0A8bd247a6ECa9CE781Ca
setup      | TestDappZEVM:       0x0000000000000000000000000000000000000000
setup      |  --- 📜EVM contracts ---
setup      | ZetaEth:        0xcd01737eaa83195514601E9F6df929Bd0709DC60
setup      | ConnectorEth:   0xfdA26637E11771E5919aF5c24A363c82613d62dA
setup      | ERC20Custody:   0x03Ce5CDd0aEC673B11a672df176E0F52E9191556
setup      | ERC20:      0x486d325e1D577f1fe6f4E1633E137f91758a8362
setup      | TestDappEVM:       0x28b64929f082F7361B9d9e681AaC3436dA4Cb857

<snip>

setup      | ❌ zevm mp panic: waiting cctx timeout, cctx not mined, inboundHash: 0x900d53080f3c437a9b34d6d8214a5ddfee2ba4eacc32fb4c8ec63b925d2bc9dc, stack trace goroutine 103 [running]:
github.com/zeta-chain/zetacore/cmd/zetae2e/local.zevmMPTestRoutine.func1.1()
	/go/delivery/zeta-node/cmd/zetae2e/local/zevm_mp.go:30 +0x60
panic({0x233b660, 0x4001d7ffd0})
	/usr/local/go/src/runtime/panic.go:884 +0x1f4
github.com/zeta-chain/zetacore/e2e/utils.WaitCctxsMinedByInboundHash({0x35084f8, 0x400004e780}, {0x4000edc4b0, 0x42}, {0x3530778, 0x400060dd20}, 0x1, {0x34e1f00, 0x40004c7e90}, 0x0)
	/go/delivery/zeta-node/e2e/utils/zetacore.go:62 +0x6bc
github.com/zeta-chain/zetacore/e2e/utils.WaitCctxMinedByInboundHash({0x35084f8?, 0x400004e780?}, {0x4000edc4b0, 0x42}, {0x3530778?, 0x400060dd20?}, {0x34e1f00?, 0x40004c7e90?}, 0xde0b6b3a7640000?)
	/go/delivery/zeta-node/e2e/utils/zetacore.go:28 +0x4c
github.com/zeta-chain/zetacore/e2e/runner.(*E2ERunner).WaitForMinedCCTX(0x400100aa00, {0x90, 0xd, 0x53, 0x8, 0xf, 0x3c, 0x43, 0x7a, 0x9b, ...})
	/go/delivery/zeta-node/e2e/runner/zeta.go:36 +0x1b0
github.com/zeta-chain/zetacore/cmd/zetae2e/local.zevmMPTestRoutine.func1()
	/go/delivery/zeta-node/cmd/zetae2e/local/zevm_mp.go:58 +0x248
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x5c
created by golang.org/x/sync/errgroup.(*Group).Go
	/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x9c

setup      | ❌ e2e tests failed after 6m26.432842915s

BTC tests are failing on the restart, but this is a good first step:

skipping e2e setup because it has already been completed
running e2e tests...
setup      | starting E2E tests
setup      | ⚙️ setting up TSS address
setup      | ⚙️ enabling verification flags for block headers
setup      |  --- 📜zEVM contracts ---
setup      | SystemContract: 0x91d18e54DAf4F677cB28167158d6dd21F6aB3921
setup      | ETHZRC20:       0x13A0c5930C028511Dc02665E7285134B6d11A5f4
setup      | ERC20ZRC20:     0x48f80608B672DC30DC7e3dbBd0343c5F02C738Eb
setup      | BTCZRC20:       0xd97B1de3619ed2c6BEb3860147E30cA8A7dC9891
setup      | UniswapFactory: 0x9fd96203f7b22bCF72d9DCb40ff98302376cE09c
setup      | UniswapRouter:  0x2ca7d64A7EFE2D62A725E2B35Cf7230D6677FfEe
setup      | ConnectorZEVM:  0x239e96c8f17C85c30100AC26F635Ea15f23E9c67
setup      | WZeta:          0x5F0b1a82749cb4E2278EC87F8BF6B618dC71a8bf
setup      | ZEVMSwapApp:    0x733aB8b06DDDEf27Eaa72294B0d7c9cEF7f12db9
setup      | ContextApp:     0xD28D6A0b8189305551a0A8bd247a6ECa9CE781Ca
setup      | TestDappZEVM:       0xA8D5060feb6B456e886F023709A2795373691E63
setup      |  --- 📜EVM contracts ---
setup      | ZetaEth:        0x733aB8b06DDDEf27Eaa72294B0d7c9cEF7f12db9
setup      | ConnectorEth:   0xD28D6A0b8189305551a0A8bd247a6ECa9CE781Ca
setup      | ERC20Custody:   0xff3135df4F2775f4091b81f4c7B6359CfA07862a
setup      | ERC20:      0xbD1e64A22B9F92D9Ce81aA9B4b0fFacd80215564
setup      | TestDappEVM:       0xBFF76e77D56B3C1202107f059425D56f0AEF87Ed

<snip>

zevm_mp    | ✅ completed in 41.117951979s - evm -> zevm message passing contract with failing revert
zevm_mp    | 🍾 ZEVM message passing tests completed in 5m1.017731008s
erc20      | ✅ completed in 12.073586308s - deposit ERC20 into ZEVM restricted address
erc20      | 🍾 erc20 tests completed in 5m3.565783507s
ether      | ✅ completed in 44.112256664s - deposit ZRC20 into ZEVM and call a contract
ether      | ⏳running - deposit Ether into ZEVM and call a contract that reverts; should refund
ether      | ✅ completed in 1m44.285673884s - deposit Ether into ZEVM and call a contract that reverts; should refund
ether      | ⏳running - withdraw Ether from ZEVM to restricted address
zeta       | ✅ completed in 2m30.333301442s - message passing with successful revert between external EVM chains
zeta       | ⏳running - deposit ZETA from Ethereum to ZEVM restricted address
zeta       | ✅ completed in 7.049840791s - deposit ZETA from Ethereum to ZEVM restricted address
zeta       | ⏳running - deposit ZETA from Ethereum to ZEVM
zeta       | ✅ completed in 13.081878008s - deposit ZETA from Ethereum to ZEVM
zeta       | ⏳running - deposit ZETA from Ethereum to a new ZEVM address which does not exist yet
ether      | ✅ completed in 24.081093927s - withdraw Ether from ZEVM to restricted address
ether      | 🍾 Ethereum tests completed in 7m11.938481337s
zeta       | ✅ completed in 11.072837997s - deposit ZETA from Ethereum to a new ZEVM address which does not exist yet
zeta       | 🍾 Zeta tests completed in 7m21.337073437s
bitcoin    | ✅ BTC deposited in 6m59.602731686s
setup      | ❌ bitcoin panic: timed out waiting for block header to show up in observer, stack trace goroutine 46 [running]:
github.com/zeta-chain/zetacore/cmd/zetae2e/local.bitcoinTestRoutine.func1.1()
	/go/delivery/zeta-node/cmd/zetae2e/local/bitcoin.go:32 +0x60
panic({0x233b780, 0x34daba0})
	/usr/local/go/src/runtime/panic.go:884 +0x1f4
github.com/zeta-chain/zetacore/e2e/runner.(*E2ERunner).ProveBTCTransaction(0x40014af100, 0x4001b016a0)
	/go/delivery/zeta-node/e2e/runner/bitcoin.go:425 +0x72c
github.com/zeta-chain/zetacore/e2e/runner.(*E2ERunner).DepositBTC(0x40014af100, 0x1)
	/go/delivery/zeta-node/e2e/runner/bitcoin.go:181 +0x430
github.com/zeta-chain/zetacore/cmd/zetae2e/local.bitcoinTestRoutine.func1()
	/go/delivery/zeta-node/cmd/zetae2e/local/bitcoin.go:65 +0x290
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x5c
created by golang.org/x/sync/errgroup.(*Group).Go
	/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x9c

setup      | ❌ e2e tests failed after 7m30.243773648s
e2e failed

Closes: #2232

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Include instructions and any relevant details so others can reproduce.

  • [ ] Tested CCTX in localnet
  • [x] Tested in development environment
  • [ ] Go unit tests
  • [ ] Go integration tests
  • [ ] Tested via GitHub Actions

gartnera avatar May 24 '24 17:05 gartnera