Query Planner selecting `_all_docs` even though valid indexes exist

[michaeldelago]

Description

Tuesday afternoon, the Query Planner stopped automatically selecting indexes for our _find queries. This was confirmed by sending the same queries to the _explain endpoint. This appears to have happened without any any changes to the software using CouchDB or the design documents that should have been used.

To resolve this, we created copies of the design documents, without changing any of the content inside of the document, except for the _id, appended the suffix "-new". Once doing this, the issue almost immediately resolved itself, and the _explain endpoint would show that it was using the non-suffixed design document.

Things that we tried, that failed:

• Restarting CouchDB nodes, one at a time
• Restarting microservice accessing CouchDB Cluster
• Restarted virtual machines hosting CouchDB, one at a time

Steps to Reproduce

Issue happened seemingly at random, with no software elease, configuration, or design document changes.

Expected Behaviour

The Query Planner in CouchDB should have selected the proper index, as documented

Your Environment

5 Node CouchDB cluster running in docker containers, deployed via in house Ansible-playbooks. config.ini on each node:

[couch_httpd_auth]
cookie_domain = REDACTED

[couchdb]
uuid = REDACTED

[log]
level = info

[chttpd]
admin_only_all_dbs = true

[smoosh.ratio_dbs]
concurrency= 2

[smoosh.ratio_views]
concurrency= 2

[smoosh.slack_dbs]
concurrency= 2

[smoosh.slack_views]
concurrency= 2

[mango]
default_limit = 1000

[admins]
REDACTED

[couch_httpd_auth]
secret = REDACTED

Cluster Information:

{
  "couchdb": "Welcome",
  "version": "3.1.1",
  "git_sha": "ce596c65d",
  "uuid": "1dd6dbe0b018660d2754ad88c2006b67",
  "features": [
    "search",
    "access-ready",
    "partitioned",
    "pluggable-storage-engines",
    "reshard",
    "scheduler"
  ],
  "vendor": {
    "name": "The Apache Software Foundation"
  }
}

Database Information

{
  "db_name": "DATABASE_NAME",
  "purge_seq": "0-g1AAAAB9eJzLYWBgYMpgTmFQSc4vTc5ISXIoLc4vsiiA8oyM9EqLU_VKk1MK9PJSS3JAqvNYgCRDA5D6DwRZiQx4tBtiak9kSKqH6MsCALbNKe0",
  "update_seq": "870085-g1AAAACDeJzLYWBgYMpgTmFQSc4vTc5ISXIoLc4vsiiA8oyM9EqLU_VKk1MK9PJSS3JAqvNYgCRDA5D6DwRZSQxsy9_jMcEQ04REhqR6qNZ517IA2sctBQ",
  "sizes": {
    "file": 895381990,
    "external": 1838490411,
    "active": 875249258
  },
  "props": {},
  "doc_del_count": 455,
  "doc_count": 379055,
  "disk_format_version": 8,
  "compact_running": false,
  "cluster": {
    "q": 2,
    "n": 3,
    "w": 2,
    "r": 2
  },
  "instance_start_time": "0"
}

• CouchDB version used: 3.1.1, using the ibmcom/couchdb3 image on DockerHub
• Browser name and version: N/A, access is done over NodeJS HTTP Client Libraries
• Operating system and version: Red Hat Enterprise Linux 8.3

Additional Context

When using the use_index parameter, it worked without issues.

In the documentation, it does state that "It’s good practice to specify indexes explicitly in your queries". This is something we're looking to get situated, though many of our microservices rely on the Query Planner to pre-select their index.