Replicable test environment

[vojtechsimetka]

Rationale: In our current tests we spawn each time new fresh network. This means we just have an empty network without any data, balances or cheques etc. Given the nature of the network, it is unrealistic to deterministically set up the network to the same state by performing actions on top of it. Therefore the approach would be to take a snapshot of the locally run environment and run tests against it.

This comes as an outcome from a testing discussion on April 7th 2021

Phases/TODO:

1. Local environment

   • [x] Create a local environment of 5 bee nodes with a local blockchain (ideally geth), deploy BZZ token, chequebook factory contract (and soon postage stamps). Make this replicable (either script or at least steps)

2. CI job for cheques tests

   • [ ] Get the bee nodes into a state that we like in terms of accounting - have some nonzero balances and several cheques.
   • [ ] Save the blockchain state and keystore and statestore of Bee
   • [ ] Create CI job that spawns the local environment and restores the state
   • [ ] Run chequebook tests against it (cashing out cheques, probably also the depositing and withdrawing)

3. CI job for feeds and other data

   • [ ] Get the bee nodes into a state that we want - probably some existing feeds and data
   • [ ] Save this state now including the localstore and use it in the CI
   • [ ] Run feed tests against it

4. Replace the current test environment

   • This may include restoring the state between different tests so that we have predictable environment

[nugaon]

The orchestration of the test Bee clients along with generating its required keys are going to happen by swarm-cli as proposed in Bee ticket #1557. identity and docker commands will be used for that, the remaining implementation of the latter is going to start soon #17. In order to start the implementation of the command we need more information about swarm.key of Bee, the discussions about it has been already started. The test environment processes will have a different repository to keep all assets of it (smart contract codes, snapshots of Bee, etc.) there. This repository could be fetched by the CI and could be used to create test environment.

[nugaon]

Currently #297 PR solved the flaky tests and created a faster environment by the bee-factory project with supporting the followings:

• build and publish Bee images manually with the corresponding Blockchain service easily by running Bee Factory's publish workflow
  • it also handles different versions of the same Bee version which are distinguished by its image tag retrieved from bee version command on image build. After publish it can be used by providing BEE_VERSION and BLOCKCHAIN_VERSION with this special tag for using the desired environment the same way as so far.
  • if new parameters are needed for a Bee version, blockchain contract changes are needed or anything else, then in Bee Factory you can create a branch with the modifications and run the image build against that.
  • freely choose the Bee image version that the build will be based on.
• every feature branch/PR can use different and specific version of Bee environment in need.
• In order to use the built environments on your local machine, you can pull the images the same way as the CI does currently.
• it is possible to build any individual Bee environment with any arbitrary state on your computer if you go throught the Bee Factory's build process.

The last point is not solved in the CI image build workflow, because:

• its maintenance and the Docker image interpretations would be more cumbersome
• you can still build any arbitrary state on your local machine in need
• CI jobs that utilize Bee Factory images have to separate their tests that needs to have pre-defined state from those which does not need
  • it raises the complexity level of the test config
• it can be solved by writing traffic generation scripts before tests that set the environment to the desired state
  • every test run will produce different state (e.g. swarm-test-worker-1 will have different state than the previous run), but with pre-built images you would have the same problem just on every new image build (statestore can change between Bee versions and Bee Factory shouldn't maintain this and create transition from one state structure to another), thereby your auto tests shouldn't rely exact state checking (e.g. one worker node has 5 checks every time).
• The image build process wouldn't be scalable anymore since you need to run the Bee docker images before pushing the images, which is not the case at the moment.

[nugaon]

@ethersphere/js ?

[vojtechsimetka]

I would still love to have images with state in it so that we always have the same environment to test against (and sure from time to time this will need to recreate it as the Bee evolves and has new potentially incompatible internal state). But we would have what we always wanted - predictable non-empty state against which we run tests in all of our repos (bee-js, swarm-cli, bee-dashboard...). This would also make it much simpler locally, you could just pull the images and before each test restore the state.

That being said if we go the way of generating the state before running the tests, I would still advocate to save the state between tests. So the workflow would be:

1. generate the state we want and save it
2. locally this state could then be restored between test runs, on CI this could also potentially be stored in a cache similar to how node_modules are
3. if needed, this state could also be restored between different tests (I don't think we need it now)