clp icon indicating copy to clipboard operation
clp copied to clipboard
verified publisher icon y-scope

possible to store in S3 and query via Athena

Open nagarajatantry opened this issue 3 years ago • 4 comments

Request

I would like to know if it is possible to store CLP compressed data in S3 and query it through Athena using SQL?

Possible implementation

NA

nagarajatantry avatar Nov 03 '22 03:11 nagarajatantry

Hi @tannaga,

Thanks for your interest. We have support for storing to S3 in our cloud, but we haven't open-sourced it yet. We are also working on a connector for Presto/Athena. We'll let you know when a prototype is available to try out.

Out of curiosity (if you can share), how much data are you looking to query and what kind of queries do you typically run?

kirkrodrigues avatar Nov 06 '22 03:11 kirkrodrigues

Thank you! Looking forward to the connector. If there is anyway I can contribute, please let me know.

CLP compression and search on compressed data seems promising. So I wanted to explore that for logs first and then expand it for other data type like metrics/traces. We have terabytes of logs per day. Was planning to experiment with S3/Athena.

nagarajatantry avatar Nov 07 '22 19:11 nagarajatantry

Any update?

gerilya avatar Mar 24 '24 15:03 gerilya

Hey @gerilya, we're working on adding support to CLP to allow it to read and write from S3. We hope to have something you can try in about a month or sooner. We haven't yet prioritized the plugin for Athena/Presto, but we'll be considering our priorities for the next quarter soon.

To guide our development, could you share what kind of use cases you have that you'd like to use CLP with S3 and/or Athena/Presto?

kirkrodrigues avatar Mar 25 '24 20:03 kirkrodrigues