Flaky `query` test

[joepio]

The db::test::queries sometimes fails in the authorized subjects step. If I run the test locally, it seems to work most of the time - both when I run it as a single test, as well as when I run it as part of --all.

However, the fact that at least once with the same code I got a fail, means the test is flaky.

---- db::test::queries stdout ----
thread 'db::test::queries' panicked at 'assertion failed: `(left == right)`
  left: `0`,
 right: `1`: authorized subjects', lib/src/db/test.rs:358:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Edit: here it happened again: https://github.com/joepio/atomic-data-rust/runs/5523317495?check_suite_focus=true#step:6:135

For some context: the queries test contains a bunch of tests for queries. In the failing line, I expect one resource to appear, since only one resource has public read rights, which was set in the start of the test. Most of the time, the results do return one resource, but sometimes 0. This only happens when I run the test as part of a set of other tests, which share a DB instance.

[jonassmedegaard]

I have experienced this exact failure several times now.

From your description it sounds like the code is "too asyncronous" - i.e. that ressource might not have completed being stored by the time you query for it.

Perhaps a dirty hack is to check if at most one ressource appears.

In the perl world, there is a convention to tag tests that either require internet or are very slow or are flaky, and skip those by default. The convention then says to enable them when environment flag EXTENDED_TESTING is set.

[joepio]

Thanks for sharing your thoughts on this. I'll add a sleep to see if that removes the failing test, that should give clarity on whether this is the issue.

In the perl world, there is a convention to tag tests that either require internet or are very slow or are flaky, and skip those by default. The convention then says to enable them when environment flag EXTENDED_TESTING is set.

I'd like to prevent this, as the test is important enough to be executed always (at least in the CI). But adding #[ignore] and setting the CI test to not ignore is actually pretty reasonable.

[joepio]

I've switched to nextest which allows for auto-retry, which helps to let the CI still succeed in this flaky test. Not a real fix for this issue, though.

I tried to give it a non-shared Db, but that did not prevent the issue.

    let store = Db::init_temp("db_queries").unwrap();

This seemed to be the obvious solution, unfortunately.