Implement API that imports Prometheus blocks into Cortex.

[edith007]

Problem statement

There might be a lot of cases where the users might already be using Prometheus and later decides to switch to Cortex. Cortex will take care of the metrics which will be generated after the switch, but at present there is no way to backfill the old Prometheus TSDB blocks to Cortex. This proposal is to add the ability in cortex to fetch the Prometheus TSDB blocks and backfill the data.

Detailed Solution

We can create a new service called Importer which will be responsible for bringing in the old Prometheus TSDB blocks to Cortex.

How Importer will work

The importer service will have two API endpoints: import and import/status/{id}

import:

• It will take a link to where the old Prometheus TSDB blocks are stored
• The API will return an ID. This ID can be used to get the status of the import operation
• Cortex will download this block from the link received and upload to the long term storage

import/status/{id}:

• This endpoint can be used to check the status of a particular import operation.
• It will return status as SUCCESS, FAILED or ONGOING.

How will Cortex import the old TSDB blocks

Fixes #4956

[edith007]

@alanprot I am researching two things:

1. How can we fetch the old Prometheus blocks
2. How to convert them to the format in which cortex stores its time series data
3. Once converted, how to push to the object store... I am sure there are things that one should keep in mind while backfilling data, which I might not be thinking at the moment... Can you suggest some pointers??

[edith007]

Notion Page: https://neon-system-ca8.notion.site/Cortex-Proposal-90419399709d4073ae5726c3a0ce3b68

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.