Deleting _users renders instance unusable

[kocolosk]

Description

There is a bug on main right now where the _users DB cannot be recreated once it has been deleted, and the cluster is rendered completely inoperable until CouchDB restarts. That's the reason for d8eea48 (and, I believe, for Paul's exclusion of other tests in that module). I've called that out separately so we can revert that commit once the fix is found.

NB: the _users DB does get recreated automatically when you restart the instance, so this issue is maybe less severe than it would otherwise seem.

Steps to Reproduce

1. Start an instance building from main i.e. 3e05ca5
2. Delete _users via the API
3. Try to make any request e.g. _all_dbs with server admin credentials. You'll see a 404 response

Expected Behaviour

Requests w/ server admin credentials should not be dependent on _users, and in particular a server admin should be able to recreate the _users DB.

Additional Context

Originally reported in #3818

[big-r81]

Hi, is this still a problem? I can't reproduce this.

Steps I done

1. Start a server, e.g (./dev/run -a a:a)
2. Delete _users db
3. curl _all_dbs with curl 'http://a:a@localhost:15984/_all_dbs'

Result

curl 'http://a:a@localhost:15984/_all_dbs'
["jwtdb","jwttestdb"]

But we getting this error in the logs:

[error] 2022-06-24T10:54:13.245132Z [email protected] emulator -------- Error in process <0.1055.0> on node '[email protected]' with exit value:
{database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,426}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,401}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,430}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,100}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,39}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,212}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,158}]}]}

[big-r81]

Closing this, can't reproduce with latest main.