Scribery
Playback issue with Elasticsearch
We are trying to play back (tlog-play) recording that's been sent to our ELK infrastructure, but are unable to do so. The error message we get is:
Invalid reply received from HTTP server
Failed reading the source at message #0
The same API call used with curl works just fine. Is there a way of debugging this more verbosely? I was unable to see any options for debugging with tlog-play. Here is a JSON entry that's been sent to ES:
{
"_shards" : {
"skipped" : 0,
"failed" : 0,
"successful" : 2,
"total" : 2
},
"hits" : {
"hits" : [
{
"_index" : "jumphost-tlog-2018.28",
"_score" : 1,
"_source" : {
"in_txt" : "",
"out_bin" : [],
"in_bin" : [],
"ver" : "2.2",
"logstash_processor" : "log-processor-prod02.domain.com",
"tags" : [
"beats_input_codec_plain_applied",
"_dateparsefailure"
],
"offset" : 69534455,
"prospector" : {
"type" : "log"
},
"pipeline_processor" : "jumphost-processor",
"pos" : 176827466,
"@version" : "1",
"pipeline_receiver" : "beats-receiver-5044",
"timestamp" : "2018-07-11T13:56:47.434149+02:00",
"logowner" : [
"usit-gsd"
],
"user" : "rafael-drift",
"session" : 2300,
"application" : "jumphost-tlog",
"out_txt" : "-------. 1 root root 95890 Jul 10 09:25 cron\\r\\n-rw-------. 1 root root 49627 Jun 17 03:19 cron-20180617\\r\\n-rw-------. 1 root root 49198 Jun 24 03:39 cron-20180624\\r\\n-rw-------. 1 root root 49573 Jul 1 03:50 cron-20180701\\r\\n-rw-------. 1 root root 140699 Jul 8 03:48 cron-20180708\\r\\n-rw-r--r--. 1 root root 63199 Jun 5 11:14 dmesg\\r\\n-rw-------. 1 root root 243861 Jul 11 13:55 elk-rsyslog.log\\r\\n-rw-------. 1 root root 69534455 Jul 11 13:56 elk-tlog.log\\r\\ndrwx------. 2 root root 22 Jul 11 11:03 \\u001B[38;5;27mfilebeat\\u001B[0m\\r\\n-rw-------. 1 root root 4973 Jul 2 13:22 grubby\\r\\n-rw-r--r--. 1 root root 193 Jun 5 11:02 grubby_prune_debug\\r\\n-rw-r--r--. 1 root root 301928 Jul 11 11:40 lastlog\\r\\n-rw-------. 1 root root 1167 Jul 10 02:16 maillog\\r\\n-rw-------. 1 root root 0 Jun 10 03:16 maillog-20180617\\r\\n-rw-------. 1 root root 6354 Ju",
"@timestamp" : "2018-07-11T11:56:54.841Z",
"beat" : {
"version" : "6.3.1",
"hostname" : "host01.domain.com",
"name" : "host01.domain.com"
},
"logstash_receiver" : "log-receiver-prod01.domain.com",
"term" : "xterm-256color",
"rec" : "cd4263b8165144e2952752446d11b521-125c-128595f5",
"id" : 21805,
"input" : {
"type" : "log"
},
"timing" : "=259x70>1882",
"host" : "host01.domain.com",
"source" : "/var/log/elk-tlog.log"
},
"_type" : "doc",
"_id" : "AWSJPIwLhYv-cRdMJshG"
}
],
"max_score" : 1,
"total" : 30469
},
"took" : 16,
"timed_out" : false
}
Any help/hint to further debug this and find a solution is highly appreciated.
Hi @fossxplorer,
I reformatted your JSON sample for readability, and don't see anything in it which could have triggered this error. So, I would recommend capturing the actual HTTP traffic between Elasticsearch and tlog, looking at it, and tracing the execution through the source code which can possibly return this error (TLOG_RC_ES_JSON_READER_REPLY_INVALID).
Thanks for your feedback. We did some simple debugging prior to filing the issue here, but will do more debugging based on your suggestions. Thanks.
