Extend query parameters by adding query clauses

[micheleriva]

Is your feature request related to a problem? Please describe. As for now, Lyra is capable of indexing documents with searchable and non-searchable fields.

For instance, given the following schema, we index the following fields:

import { lyra } from '@nearfom/lyra';

const movieDB = new Lyra({
  schema: {
    // searchable fields
    title: 'string',
    director: 'string',
    plot: 'string',

    // non searchable
    year: 'number',
    isFavorite: 'boolean'
  }
});

Even though numbers and booleans are non-searchable fields, we should start using them for performing queries using the where keyword. An example could be:

const result = await movieDB.search({
  term: 'love',
  limit: 10,
  offset: 5,
  where: {
    year: { '>=': 1990 },
    isFavorite: true
  }
});

as a first iteration, we could go using AND only (so, in the above example, WHERE year >= 1990 AND isFavorite = true). In the future, we might want to support AND, OR, CONTAINS, etc.

[DanieleFedeli]

How can this be done? Do we first retrieve the documents and then apply the conditions or we are also indexing non-searchable fields?

[micheleriva]

@DanieleFedeli I think we could store non-searchable parameters in reverse indexes and then access these properties in constant time during the search. I am currently working on the stemming issue, but I can try to illustrate my proposal with a simple, non-working, demonstrative PR

[micheleriva]

TL;DR

[DanieleFedeli]

I am working on that, by now I have only created the type for the where clause.

[DanieleFedeli]

For now I am using these data structures for indexing non-searchable fields:

• Boolean: HashMap -> Key: ${PropName}_${booleanValue} Value: Set which contains ids
• Number: HashMap -> Key: ${PropName} value: another HashMap (Key -> numeric value, Value -> Set which contains ids)

Imho the boolean approach is ok but I personally don't like the numeric one. For exact numbers, it works well with just two reads in O(1) but when we have to do a range query it is a bit hacky.

Do you have any suggestions on that?

[DanieleFedeli]

My PR is almost ready, I will try to do some benchmark

[micheleriva]

@DanieleFedeli quick follow-up: we're planning a plugin system, and we'll probably move the query clauses as a separate plugin; as for now I'll keep your PR opened as it might get merged as a plugin

[nicksav]

once where clause comes, facets filters will become available too?

[DanieleFedeli]

I am building this as a plugin https://github.com/DanieleFedeli/lyra-advanced-query-plugin For the moment it is pretty raw and It is not published to npm yet.

[micheleriva]

Released in v.0.4.9