oceanprotocol
Need integration tests for all Ocean-supported networks
Motivation
In solving #923 "ocean.py has no remote tests", I created a test for a single remote network (Mumbai). The test file is tests/integration/remote/test_mumbai.py.
But we really should have tests (local & CI) to ensure that ocean.py works on all Ocean-supported remote networks, both test and production. Currently that's about ten networks. It's simply good hygiene.
There's a second reason: it's possible that ocean.py doesn't work on some of these networks! For example, in making the remote test work for Mumbai, I had to solve a new issue (#920). There will likely be issues for some of the other networks too.
TODO
What to do: generalize test_mumbai.py to all the Ocean-supported networks.
My opinion is all tests in ocean.py should be local, with barge. And we should create a new QA repository where we add all the remote tests. The bloating with ocean.py tests is insane, where this single repo has the burden to test almost everything in our stack, from contracts to fees, provider and aquarius behaviour, compute results, local and remote networks etc. Let's consider a strategy that does not bloat ocean.py with tests that do not concern its functionality.
Perhaps create a new ocean.py issue to kick that off. (Issues after that will be part of the new repo)
Update: I just created the issue. #969 "Move heavy/slow integration tests into their own github repo"
Re
My opinion is all tests in ocean.py should be local, with barge. And we should create a new QA repository where we add all the remote tests. The bloating with ocean.py tests is insane, where this single repo has the burden to test almost everything in our stack, from contracts to fees, provider and aquarius behaviour, compute results, local and remote networks etc. Let's consider a strategy that does not bloat ocean.py with tests that do not concern its functionality.
I agree with all of this except for the first sentence.
ocean.py was basically not working at all on anything but local. This shouldn't be a surprise: it wasn't being tested there; nor were there READMEs there. Yet it means that wasn't able to perform its most basic duties on any production environment!
The solution:
- inside ocean.py, have at least some basic_ tests to make sure ocean.py works on remote networks. test_mumbai.py is just about right for this! If test fails, that's great news, because it's the "canary in the coal mine" that ocean.py is having trouble beyond ganache. Eg it was this test that showed ocean.py couldn't find Ocean addresses beyond Ganache - a critical bug if there ever was one.
- ocean.py has more (not fewer) READMEs for people using remote networks. It walks them from local to remote in an easy way. This minimizes the friction to them using ocean.py in their actual production use case. Good news, we're already there. (And testing needs to get buffed up.)
- put the heavier, more extensive tests into a dedicated CI testing repo (see issue # 969). This includes the remote README tests.
Update: I've added the relevant comments into the new issue: #969. Let's have further discussion there.
Given how much of a pain it's been to test polygon & mumbai - due to unreliability of the networks themelves,
Which leads me to expect:
- testing for more networks reliably will be high cost to us
- and will only have marginal benefit
Therefore: let's drop this issue for now. If it becomes a priority to do so at some point, it will happen organically.