search-replica

search-replica copied to clipboard

Search Replica

Replicates PostgreSQL primary database into Elasticsearch/OpenSearch index read-only replica.

Does not rely on additional database queries or external queues, using exclusively logical replication protocol, allowing almost realtime consistent synchronisation without external dependencies.

Focused on speed and efficiency.

Docs | DockerHub | Try it

• Consistent and fault tolerant, without dependencies
  Thanks to PostgreSQL replication slots mechanisms.
• Initial (re)indexing
  Using COPY command
• Uses native PG protocol
  Both Text and Binary form.
• Full DB types support
  json fields, composite types, arrays, enums... Except of arrays of composite types.
• Native Parent/Child join Including document _id and routing control
• Limited denormalization & document modifications. check inlining
• Bulk requests Data is flushed to Elasticsearch/OpenSearch in bulk.

Configuration

ConfTags

You can

• set routing and document _id fields;
• rename or skip fields;
• define parent/child join field;
• inline rows as object into parent document;
• set custom inlining script;
• ~~set templated fields~~ (planned)
• ~~json-path names~~ (planned)

Using COMMENTs in your database schema. Check syntax and description

Env Config

Notes

• The script is single threaded* (not a bottleneck)... Separate goroutine is used to make ES requests.
• Links between Database <-> Schema <-> Table <-> Column, shoudld be considered read only, and safe for multithread use... (not yet)
• It's fast. All the the efforts shuld be towards readability, reliability and functionality.

Known Limitations:

• No 1:1 inlines (yet)
• Delete document deletes all inlines (AKA DELETE CASCADE), and they can not be restored.