connect icon indicating copy to clipboard operation
connect copied to clipboard
verified publisher icon redpanda-data

Investigate elastic/opensearch forks

Open Jeffail opened this issue 4 years ago • 8 comments

The dust is starting to settle now since the opensearch fork so it's probably worth spending some time understanding what's needed in order to continue to support both offerings. We're currently using https://github.com/olivere/elastic which has been fantastic (more info about future direction here: https://github.com/olivere/elastic/issues/1533) but we could also consider moving to the official client libraries as we only use a small fraction of ES endpoints.

Official libraries:

  • https://github.com/elastic/go-elasticsearch
  • https://github.com/opensearch-project/opensearch-go

Questions:

  • Do we need a forked opensearch component?

Jeffail avatar Nov 29 '21 12:11 Jeffail

Hey @Jeffail

Some people ( me too ) want to run their own Elastic equivalent.

There is a golang equivalent that matches the Elastic API closely https://github.com/prabhatsharma/zinc For collection they recommend https://github.com/fluent/fluent-logger-golang for logging into it

Its getting alot of attention. I guess tons of people want an efficient alternative.

It works well. It's based on bluge: https://github.com/blugelabs/bluge ( thankyou @mschoch ).

gedw99 avatar Jan 05 '22 15:01 gedw99

@Jeffail I would be interested in seeing a dedicated output for OpenSearch built against the aforementioned opensearch-go. We could ask the coordinators of https://opensearch.org/events/ to allow you to present, showing off Benthos and its OpenSearch integration.

ryn9 avatar Jan 05 '22 20:01 ryn9

@Jeffail I can contribute in creating a dedicated opensearch output sink. Let me know.

arunx2 avatar Aug 15 '22 03:08 arunx2

@Jeffail I can contribute in creating a dedicated opensearch output sink. Let me know.

Go ahead if you have cycles to spare. All new contributors are welcome! Happy to help you get it in shape for merging

mihaitodor avatar Aug 15 '22 13:08 mihaitodor

@Jeffail I can contribute in creating a dedicated opensearch output sink. Let me know.

Go ahead if you have cycles to spare. All new contributors are welcome! Happy to help you get it in shape for merging

I'm on it. Thanks!

arunx2 avatar Aug 16 '22 01:08 arunx2

The official Go Client Library for OpenSearch - https://github.com/opensearch-project/opensearch-go

ryn9 avatar Aug 16 '22 22:08 ryn9

@Jeffail I can contribute in creating a dedicated opensearch output sink. Let me know.

Go ahead if you have cycles to spare. All new contributors are welcome! Happy to help you get it in shape for merging

I'm on it. Thanks!

@Jeffail here is the PR. https://github.com/benthosdev/benthos/pull/1388

arunx2 avatar Aug 17 '22 13:08 arunx2