datahike icon indicating copy to clipboard operation
datahike copied to clipboard

Better test coverage for different database configurations

Open jsmassa opened this issue 4 years ago • 2 comments

Problem

In many of the tests only the default configuration for datahike is being tested.

Solution

There should be at least additionally proper testing for

  • {:attribute-refs? true}
  • {:index :datahike.index/persistent-set} (or in general non-default indices)

The best case scenario would be to enable testing of all config combinations:

  • :index [ :datahike.index/hitchhiker-tree :datahike.index/persistent-set]}
  • :backend [:mem :file]
  • :keep-history? [true false]
  • :attribute-refs? [true false]
  • :schema-flexibility [:read :write]

Consequences

  • The attribute-refs test folder in datahike.tests should be made obsolete and if at all replaced by a single attribute-refs test namespace testing only specific features

Difficulties

  1. It's a quite tedious task to port all the namespaces, so:
    • port namespaces one by one, step by step
    • start with the most important namespaces that test core functionalities
  2. Databases with {:schema-flexibility :write} yield more datoms than schema-on-read databases, so the entities of produced datoms will differ, plus they need a schema to be added to the database
    • a schema has to be thought of for tests not working with one yet
  3. Databases with {:attribute-refs? true} have even more datoms, they need a schema to work and their produced datoms look different than databases using attribute references directly
    • potentially use query/pull outputs for comparisons instead of raw datom listings
  4. Testing all configurations will potentially take a long time
    • kaocha can be used to configure the tests to run a fast subset of configurations

Tips

  • The attribute-refs test folder contains some ideas on
    • how to port schema-on-read test to schema-on-write tests
    • how to port tests to compare datoms with references, although there should be a better way

jsmassa avatar Apr 26 '22 14:04 jsmassa

Blocked by #519

jsmassa avatar Apr 27 '22 12:04 jsmassa

PR #503 introduces a new way of running tests by setting the default configuration form the tests.edn test configuration file. This can be used to increase the test coverage for any valid configuration

jsmassa avatar May 30 '22 07:05 jsmassa