Akumuli icon indicating copy to clipboard operation
Akumuli copied to clipboard

Akumuli crashing on query

Open sptrakesh opened this issue 5 years ago • 2 comments

I am running via the latest Docker image (akumuli:0.8.80-skylake). The following query via the Grafana plugin causes the container to crash.

spt_akumuli.1.sullcoizlnqg@docker-desktop    | 2020-06-05 23:02:06,816 http [INFO] Cursor 140025295594008 created
spt_akumuli.1.sullcoizlnqg@docker-desktop    | 2020-06-05 23:02:06,816 http [INFO] Cursor 140025295594008 started
spt_akumuli.1.sullcoizlnqg@docker-desktop    | 2020-06-05 23:02:06,817 Main [INFO] Parsing query:
spt_akumuli.1.sullcoizlnqg@docker-desktop    | 2020-06-05 23:02:06,818 Main [INFO] {
spt_akumuli.1.sullcoizlnqg@docker-desktop    |     "select": "request.counter",
spt_akumuli.1.sullcoizlnqg@docker-desktop    |     "range": {
spt_akumuli.1.sullcoizlnqg@docker-desktop    |         "from": "20200605T230206.779",
spt_akumuli.1.sullcoizlnqg@docker-desktop    |         "to": "20200605T200206.779"
spt_akumuli.1.sullcoizlnqg@docker-desktop    |     },
spt_akumuli.1.sullcoizlnqg@docker-desktop    |     "where": "",
spt_akumuli.1.sullcoizlnqg@docker-desktop    |     "order-by": "series",
spt_akumuli.1.sullcoizlnqg@docker-desktop    |     "apply": "",
spt_akumuli.1.sullcoizlnqg@docker-desktop    |     "limit": "1000000",
spt_akumuli.1.sullcoizlnqg@docker-desktop    |     "pivot-by-tag": [
spt_akumuli.1.sullcoizlnqg@docker-desktop    |         "resource"
spt_akumuli.1.sullcoizlnqg@docker-desktop    |     ]
spt_akumuli.1.sullcoizlnqg@docker-desktop    | }
spt_akumuli.1.sullcoizlnqg@docker-desktop    | 
spt_akumuli.1.sullcoizlnqg@docker-desktop    | 2020-06-05 23:02:06,819 Main [ERROR] MergeIterator - broken invariant
spt_akumuli.1.sullcoizlnqg@docker-desktop    | 2020-06-05 23:02:06,820 Main [ERROR] Terminating (core dumped)

Akumuli logs also show this error:

2020-06-05 23:09:30,277 Main [INFO] 18446744073709550400 Trying to open tree, repair status - OK, addr: 20537
2020-06-05 23:09:30,278 Main [ERROR] Empty rescue points list found, leaf-node data was lost
2020-06-05 23:09:30,278 Main [ERROR] Repair needed, id=1145

sptrakesh avatar Jun 05 '20 23:06 sptrakesh

How your tags look like? Is 'resource' the only tag in the series name?

Lazin avatar Jun 08 '20 19:06 Lazin

No, there are additional tags such as method, resource, ipaddress, status and mimeType.

Sorry, an OS update wiped out the old logs. I was able to trace the issue back further to what looked like a database corruption (probably caused when writing the series). The query causing the crash seemed to be an after effect of that corruption.

Will create another issue if I run into it again. Until then probably only option is close this issue.

sptrakesh avatar Jun 08 '20 20:06 sptrakesh