tstack
How does lnav custom log format works?
Hi,
I am new here and I don't know if this is the place to ask but I realy need some help with custom logs in lnav. I tried to search for my situation but didn't find anything usefull or I did not know where to look for.
I have a big custom log format on my nginx server because it is easier to find the problem when something brokes and now I want to try lnav for my log analisys but I can not figure out how the log format works.
This is a line from my log: 127.0.0.1 - - [14/Oct/2021:15:15:01 +0300] "GET / HTTP/2.0" 301 166 "http://site.ro/" "Optimus Cache Prime/2.7-site (https://example.com/projects/12/)[email protected]" "GET" "https" "site.ro" "/" "HTTP/2.0" "0.000" "2603966" "1" "d6ad43la777e108otherstufb5b1f7da9a09" "127.0.0.1" "-" "-" "-" "-" "-" "-" "-" "site.ro" "TLSv1.2" "ECDHE-RSA-AES256-GCM-SHA384" "NONE" "." "efbdc9542b4a38ff42a04686otherstufhere07b1f8dda8003823" "-" "ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA:AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:DES-CBC3-SHA" "X25519:prime256v1:secp384r1:secp521r1" "-" "-" "-" "-" "-" "-" "-" "-" "-" "text/html" "-" "0"
This is the log format from nginx: '$CLIENT_IP - $remote_user [$time_local] "$request" ''$status $body_bytes_sent "$http_referer" ''"$http_user_agent"'' "$request_method"'' "$scheme"'' "$host"'' "$request_uri"'' "$server_protocol"'' "$request_time"'' "$connection"'' "$connection_requests"'' "$request_id"'' "$remote_addr"'' "$http_x_real_ip"'' "$http_x_forwarded_for"'' "$http_cf_connecting_ip"'' "$http_cf_ipcountry"'' "$http_cf_ray"'' "$http_cf_request_id"'' "$http_cf_visitor"'' "$ssl_server_name"'' "$ssl_protocol"'' "$ssl_cipher"'' "$ssl_client_verify"'' "$ssl_session_reused"'' "$ssl_session_id"'' "$ssl_client_fingerprint"'' "$ssl_ciphers"'' "$ssl_curves"'' "$ssl_early_data"'' "$upstream_addr"'' "$upstream_bytes_received"'' "$upstream_bytes_sent"'' "$upstream_status"'' "$upstream_connect_time"'' "$upstream_header_time"'' "$upstream_response_time"'' "$upstream_response_length"'' "$sent_http_content_type"'' "$sent_http_x_ws"'' "$ADMIN_LOGGED_IN"';
If you could point me in the right direction, i could adapt my format for lnav or if someone has a script to convert my format into lnav format's that would be also grate.
I am new here and I don't know if this is the place to ask but I realy need some help with custom logs in lnav.
Welcome, this is a fine place to ask questions.
I have a big custom log format on my nginx server
You can create a format file to tell lnav how to interpret your log files. Here's the documentation on defining a new format file:
https://docs.lnav.org/en/latest/formats.html#defining-a-new-format
Basically, you'll be writing a JSON file that contains a definition of your format. The main parts of the definition are a regular expression that matches your log lines and a description of the values being captured.
Here's the format file for the default apache/nginx log format, to give you an idea of what you're trying to make:
https://github.com/tstack/lnav/blob/master/src/formats/access_log.json
I might have some more time over the weekend to help out more.
That is super. I haven't wrote any code in Java script but I will give it a try and see how it works.
The regular expression might be the hard part and I know I am not good at all for this but, I,m 20 years old, I do have time to learn it good :))
Thank you for your explanation and I will write back if I can not finish this.
The regular expression might be the hard part and I know I am not good at all for this but, I,m 20 years old, I do have time to learn it good :))
Try out some websites that help with constructing regexes. Here's a link to one that I've preloaded with a start of the regex, you'll need to continue fleshing it out to capture the rest of the values in your example string:
https://regex101.com/r/qR0ZFa/1/
I think I was able to construct the regular expression on the site you gave me. Now I just need to make the json and if I understood right, the json will only have the declaration of the fields that I use in the regex so that lnav can use them.
Could you verify if the regex looks right? I think it was too easy to be right :)) I think that this is the link:
https://regex101.com/r/TdbB42/1/
Looks about right to me. The following is a start at the format definition. Save it to a file with a .json extension and use lnav -i path/to/format.json to install it. You'll still need to flesh out the value section to add the captured values that I missed.
{
"$schema": "https://lnav.org/schemas/format-v1.schema.json",
"micu_nginx_log": {
"title": "Micu nginx log",
"description": "abc",
"multiline": false,
"regex": {
"std": {
"pattern": "^(?<CLIENT_IP>[\\w\\.:\\-]+)\\s+[\\w\\.\\-]+\\s+(?<remote_user>\\S+)\\s+\\[(?<timestamp>[^\\]]+)\\] \"(?:\\-|(?<request>\\w+) (?<cs_uri_stem>[^ \\?]+)(?:\\?(?<cs_uri_query>[^ ]*))? (?<cs_version>[\\w\\/\\.]+))\" (?<status>\\d+) (?<body_bytes_sent>\\d+|-)(?: \"(?<http_referer>[^\"]+)\" \"(?<http_user_agent>[^\"]+)\")?\\s* \"(?<request_method>[^\"]+)\" \"(?<scheme>[^\"]+)\" \"(?<host>[^\"]+)\" \"(?<request_uri>[^\"]+)\" \"(?<server_protocol>[^\"]+)\" \"(?<request_time>[^\"]+)\" \"(?<connection>[^\"]+)\" \"(?<connection_requests>[^\"]+)\" \"(?<request_id>[^\"]+)\" \"(?<remote_addr>[^\"]+)\" \"(?<http_x_real_ip>[^\"]+)\" \"(?<http_x_forworded_for>[^\"]+)\" \"(?<http_cf_connecting_ip>[^\"]+)\" \"(?<http_cf_ipcountry>[^\"]+)\" \"(?<http_cf_ray>[^\"]+)\" \"(?<http_cf_request_id>[^\"]+)\" \"(?<http_cf_visitor>[^\"]+)\" \"(?<ssl_server_name>[^\"]+)\" \"(?<ssl_protocol>[^\"]+)\" \"(?<ssl_cipher>[^\"]+)\" \"(?<ssl_client_verify>[^\"]+)\" \"(?<ssl_session_reused>[^\"]+)\" \"(?<ssl_session_id>[^\"]+)\" \"(?<ssl_client_fingerprint>[^\"]+)\" \"(?<ssl_ciphers>[^\"]+)\" \"(?<ssl_curves>[^\"]+)\" \"(?<ssl_early_data>[^\"]+)\" \"(?<upstream_addr>[^\"]+)\" \"(?<upstream_bytes_received>[^\"]+)\" \"(?<upstream_bytes_sent>[^\"]+)\" \"(?<upstream_status>[^\"]+)\" \"(?<upstream_connect_time>[^\"]+)\" \"(?<upstream_header_time>[^\"]+)\" \"(?<upstream_response_time>[^\"]+)\" \"(?<upstream_response_length>[^\"]+)\" \"(?<sent_http_content_type>[^\"]+)\" \"(?<sent_http_x_ws>[^\"]+)\" \"(?<ADMIN_LOGGED_IN>[^\"]+)\""
}
},
"level-field": "sc_status",
"level": {
"error": "^[^123].*"
},
"opid-field": "c_ip",
"value": {
"ADMIN_LOGGED_IN": {
"kind": "integer",
"foreign-key": true
},
"body_bytes_sent": {
"kind": "integer"
},
"connection": {
"kind": "string",
"identifier": true
},
"connection_requests": {
"kind": "integer"
},
"cs_uri_query": {
"kind": "string"
},
"cs_uri_stem": {
"kind": "string",
"identifier": true
},
"cs_version": {
"kind": "string",
"identifier": true
},
"CLIENT_IP": {
"kind": "string",
"collate": "ipaddress",
"identifier": true
},
"host": {
"identifier": true
}
},
"sample": [
{
"line": "127.0.0.1 - - [14/Oct/2021:15:15:01 +0300] \"GET / HTTP/2.0\" 301 166 \"http://site.ro/\" \"Optimus Cache Prime/2.7-site (https://example.com/projects/12/)[email protected]\" \"GET\" \"https\" \"site.ro\" \"/\" \"HTTP/2.0\" \"0.000\" \"2603966\" \"1\" \"d6ad43la777e108otherstufb5b1f7da9a09\" \"127.0.0.1\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\" \"site.ro\" \"TLSv1.2\" \"ECDHE-RSA-AES256-GCM-SHA384\" \"NONE\" \".\" \"efbdc9542b4a38ff42a04686otherstufhere07b1f8dda8003823\" \"-\" \"ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA:AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:DES-CBC3-SHA\" \"X25519:prime256v1:secp384r1:secp521r1\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\" \"text/html\" \"-\" \"0\""
}
]
}
}
Well, that should be it. Thank you verry much for all your help. This showld be easy to fill in.
Just thank you! You are awesome!
Hi, I am still working on this log format but I can not figure out what is going on. I can see that lnav is finding my format but it can not be used.
I actually do not know what is wrong here but I will attach the log format json and also the error that I get.
{
"$schema": "https://lnav.org/schemas/format-v1.schema.json",
"micu_nginx_log": {
"title": "Micu nginx log",
"description": "Distinctlogv2",
"multiline": false,
"regex": {
"std": {
"pattern": "^(?<CLIENT_IP>[\\w\\.:\\-]+)\\s+[\\w\\.\\-]+\\s+(?<remote_user>\\S+)\\s+\\[(?<timestamp>[^\\]]+)\\] \"(?:\\-|(?<request>\\w+) (?<cs_uri_stem>[^ \\?]+)(?:\\?(?<cs_uri_query>[^ ]*))? (?<cs_version>[\\w\\/\\.]+))\" (?<status>\\d+) (?<body_bytes_sent>\\d+|-)(?: \"(?<http_referer>[^\"]+)\" \"(?<http_user_agent>[^\"]+)\")?\\s* \"(?<request_method>[^\"]+)\" \"(?<scheme>[^\"]+)\" \"(?<host>[^\"]+)\" \"(?<request_uri>[^\"]+)\" \"(?<server_protocol>[^\"]+)\" \"(?<request_time>[^\"]+)\" \"(?<connection>[^\"]+)\" \"(?<connection_requests>[^\"]+)\" \"(?<request_id>[^\"]+)\" \"(?<remote_addr>[^\"]+)\" \"(?<http_x_real_ip>[^\"]+)\" \"(?<http_x_forworded_for>[^\"]+)\" \"(?<http_cf_connecting_ip>[^\"]+)\" \"(?<http_cf_ipcountry>[^\"]+)\" \"(?<http_cf_ray>[^\"]+)\" \"(?<http_cf_request_id>[^\"]+)\" \"(?<http_cf_visitor>[^\"]+)\" \"(?<ssl_server_name>[^\"]+)\" \"(?<ssl_protocol>[^\"]+)\" \"(?<ssl_cipher>[^\"]+)\" \"(?<ssl_client_verify>[^\"]+)\" \"(?<ssl_session_reused>[^\"]+)\" \"(?<ssl_session_id>[^\"]+)\" \"(?<ssl_client_fingerprint>[^\"]+)\" \"(?<ssl_ciphers>[^\"]+)\" \"(?<ssl_curves>[^\"]+)\" \"(?<ssl_early_data>[^\"]+)\" \"(?<upstream_addr>[^\"]+)\" \"(?<upstream_bytes_received>[^\"]+)\" \"(?<upstream_bytes_sent>[^\"]+)\" \"(?<upstream_status>[^\"]+)\" \"(?<upstream_connect_time>[^\"]+)\" \"(?<upstream_header_time>[^\"]+)\" \"(?<upstream_response_time>[^\"]+)\" \"(?<upstream_response_length>[^\"]+)\" \"(?<sent_http_content_type>[^\"]+)\" \"(?<sent_http_x_ws>[^\"]+)\" \"(?<ADMIN_LOGGED_IN>[^\"]+)\""
}
},
"level-field": "sc_status",
"level": {
"error": "^[^123].*"
},
"opid-field": "c_ip",
"value": {
"ADMIN_LOGGED_IN": {
"kind": "integer",
"foreign-key": true
},
"body_bytes_sent": {
"kind": "integer"
},
"connection": {
"kind": "string",
"identifier": true
},
"connection_requests": {
"kind": "integer"
},
"cs_uri_query": {
"kind": "string"
},
"cs_uri_stem": {
"kind": "string",
"identifier": true
},
"cs_version": {
"kind": "string",
"identifier": true
},
"CLIENT_IP": {
"kind": "string",
"collate": "ipaddress",
"identifier": true
},
"host": {
"identifier": true
},
"timestamp": {
"kind": "string"
},
"request": {
"kind": "string"
},
"status": {
"kind": "string"
},
"http_referer": {
"kind": "string"
},
"http_user_agent": {
"kind": "string"
},
"request_method": {
"kind": "string"
},
"scheme": {
"kind": "string"
},
"request_uri": {
"kind": "string"
},
"ssl_cipher": {
"kind": "string"
},
"server_protocol": {
"kind": "string"
},
"request_time": {
"kind": "integer"
},
"request_id": {
"kind": "string"
},
"remote_addr": {
"kind": "string"
},
"http_x_real_ip": {
"kind": "string"
},
"http_x_forwarded_for": {
"kind": "string"
},
"http_cf_connecting_ip": {
"kind": "string"
},
"http_cf_ipcountry": {
"kind": "string"
},
"http_cf_ray": {
"kind": "string"
},
"http_cf_request_id": {
"kind": "integer"
},
"http_cf_visitor": {
"kind": "string"
},
"ssl_server_name": {
"kind": "string"
},
"ssl_protocol": {
"kind": "string"
},
"ssl_client_verify": {
"kind": "string"
},
"ssl_session_reused": {
"kind": "string"
},
"ssl_session_id": {
"kind": "string"
},
"ssl_client_fingerprint": {
"kind": "string"
},
"ssl_ciphers": {
"kind": "string"
},
"ssl_curves": {
"kind": "string"
},
"ssl_early_data": {
"kind": "string"
},
"upstream_addr": {
"kind": "string"
},
"upstream_bytes_received": {
"kind": "integer"
},
"upstream_bytes_sent": {
"kind": "integer"
},
"upstream_status": {
"kind": "string"
},
"upstream_connect_time": {
"kind": "integer"
},
"upstream_header_time": {
"kind": "integer"
},
"upstream_response_time": {
"kind": "integer"
},
"upstream_response_length": {
"kind": "integer"
},
"sent_http_content_type": {
"kind": "string"
},
"sent_http_x_ws": {
"kind": "string"
}
},
"sample": [
{
"line": "127.0.0.1 - - [14/Oct/2021:15:15:01 +0300] \"GET / HTTP/2.0\" 301 166 \"http://site.ro/\" \"Optimus Cache Prime/2.7-site (https://example.com/projects/12/)[email protected]\" \"GET\" \"https\" \"site.ro\" \"/\" \"HTTP/2.0\" \"0.000\" \"2603966\" \"1\" \"d6ad43la777e108otherstufb5b1f7da9a09\" \"127.0.0.1\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\" \"site.ro\" \"TLSv1.2\" \"ECDHE-RSA-AES256-GCM-SHA384\" \"NONE\" \".\" \"efbdc9542b4a38ff42a04686otherstufhere07b1f8dda8003823\" \"-\" \"ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA:AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:DES-CBC3-SHA\" \"X25519:prime256v1:secp384r1:secp521r1\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\" \"text/html\" \"-\" \"0\""
}
]
}
}
and this is the error:
`2021-10-27T15:27:14.128 D log_format_loader.cc:82 format regex: micu_nginx_log/std = ^(?<CLIENT_IP>[\w\.:\-]+)\s+[\w\.\-]+\s+(?<remote_user>\S+)\s+\[(
?<timestamp>[^\]]+)\] "(?:\-|(?<request>\w+) (?<cs_uri_stem>[^ \?]+)(?:\?(?<cs_uri_query>[^ ]*))? (?<cs_version>[\w\/\.]+))" (?<status>\d+) (?<body_byt
es_sent>\d+|-)(?: "(?<http_referer>[^"]+)" "(?<http_user_agent>[^"]+)")?\s* "(?<request_method>[^"]+)" "(?<scheme>[^"]+)" "(?<host>[^"]+)" "(?<request_
uri>[^"]+)" "(?<server_protocol>[^"]+)" "(?<request_time>[^"]+)" "(?<connection>[^"]+)" "(?<connection_requests>[^"]+)" "(?<request_id>[^"]+)" "(?<remo
te_addr>[^"]+)" "(?<http_x_real_ip>[^"]+)" "(?<http_x_forworded_for>[^"]+)" "(?<http_cf_connecting_ip>[^"]+)" "(?<http_cf_ipcountry>[^"]+)" "(?<http_cf
_ray>[^"]+)" "(?<http_cf_request_id>[^"]+)" "(?<http_cf_visitor>[^"]+)" "(?<ssl_server_name>[^"]+)" "(?<ssl_protocol>[^"]+)" "(?<ssl_cipher>[^"]+)" "(?
<ssl_client_verify>[^"]+)" "(?<ssl_session_reused>[^"]+)" "(?<ssl_session_id>[^"]+)" "(?<ssl_client_fingerprint>[^"]+)" "(?<ssl_ciphers>[^"]+)" "(?<ssl
_curves>[^"]+)" "(?<ssl_early_data>[^"]+)" "(?<upstream_addr>[^"]+)" "(?<upstream_bytes_received>[^"]+)" "(?<upstream_bytes_sent>[^"]+)" "(?<upstream_s
tatus>[^"]+)" "(?<upstream_connect_time>[^"]+)" "(?<upstream_header_time>[^"]+)" "(?<upstream_response_time>[^"]+)" "(?<upstream_response_length>[^"]+)
" "(?<sent_http_content_type>[^"]+)" "(?<sent_http_x_ws>[^"]+)" "(?<ADMIN_LOGGED_IN>[^"]+)"
2021-10-27T15:27:14.129 I log_format_loader.cc:509 found format: micu_nginx_log
2021-10-27T15:27:14.129 W log_format_loader.cc:567 Format collision, format 'access_log' matches sample from 'micu_nginx_log'
2021-10-27T15:27:14.130 E pcrepp.hh:422 pcre err -8
2021-10-27T15:27:14.130 E pcrepp.hh:422 pcre err -8
2021-10-27T15:27:14.133 I log_format_loader.cc:614 Format order:
2021-10-27T15:27:14.133 I log_format_loader.cc:618 block_log
2021-10-27T15:27:14.133 I log_format_loader.cc:618 choose_repo_log
2021-10-27T15:27:14.133 I log_format_loader.cc:618 dpkg_log
2021-10-27T15:27:14.133 I log_format_loader.cc:618 engine_log
2021-10-27T15:27:14.133 I log_format_loader.cc:618 error_log
2021-10-27T15:27:14.133 I log_format_loader.cc:618 fsck_hfs_log
2021-10-27T15:27:14.133 I log_format_loader.cc:618 glog_log
2021-10-27T15:27:14.133 I log_format_loader.cc:618 micu_nginx_log
2021-10-27T15:27:14.133 I log_format_loader.cc:618 openam_log
2021-10-27T15:27:14.133 I log_format_loader.cc:618 openamdb_log
2021-10-27T15:27:14.133 I log_format_loader.cc:618 openstack_log
2021-10-27T15:27:14.133 I log_format_loader.cc:618 page_log
2021-10-27T15:27:14.133 I log_format_loader.cc:618 snaplogic_log
2021-10-27T15:27:14.133 I log_format_loader.cc:618 strace_log
2021-10-27T15:27:14.133 I log_format_loader.cc:618 syslog_log
2021-10-27T15:27:14.133 I log_format_loader.cc:618 tcf_log
2021-10-27T15:27:14.133 I log_format_loader.cc:618 tcsh_history
2021-10-27T15:27:14.133 I log_format_loader.cc:618 uwsgi_log
2021-10-27T15:27:14.133 I log_format_loader.cc:618 vdsm_log
2021-10-27T15:27:14.133 I log_format_loader.cc:618 vmk_log
2021-10-27T15:27:14.133 I log_format_loader.cc:618 vmw_log
2021-10-27T15:27:14.133 I log_format_loader.cc:618 access_log
2021-10-27T15:27:14.134 D log_vtab_impl.cc:147 creating log format table: generic_log = 0x2532658
2021-10-27T15:27:14.134 D log_vtab_impl.cc:147 creating log format table: block_log = 0x2530698
2021-10-27T15:27:14.135 D log_vtab_impl.cc:147 creating log format table: choose_repo_log = 0x25325c8
2021-10-27T15:27:14.135 D log_vtab_impl.cc:147 creating log format table: dpkg_log = 0x2533d68
2021-10-27T15:27:14.135 D log_vtab_impl.cc:147 creating log format table: engine_log = 0x2532968
2021-10-27T15:27:14.135 D log_vtab_impl.cc:147 creating log format table: error_log = 0x2538268
2021-10-27T15:27:14.135 D log_vtab_impl.cc:147 creating log format table: fsck_hfs_log = 0x2532448
2021-10-27T15:27:14.135 D log_vtab_impl.cc:147 creating log format table: glog_log = 0x253ded8
2021-10-27T15:27:14.135 E log_format.cc:1324 failed postcondition `0'
2021-10-27T15:27:14.135 E lnav_log.cc:272 Received signal: 6
lnav[0x453e29]
lnav[0x4543ae]
lnav[0x46024e]
lnav[0x49237f]
lnav[0x492886]
/lib64/libsqlite3.so.0(+0x3b361)[0x7ff49a64d361]
/lib64/libsqlite3.so.0(+0x6bd14)[0x7ff49a67dd14]
/lib64/libsqlite3.so.0(sqlite3_step+0x2bf)[0x7ff49a685d1f]
/lib64/libsqlite3.so.0(sqlite3_exec+0x10a)[0x7ff49a67b81a]
lnav[0x491f23]
lnav[0x40a417]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7ff498b93555]
lnav[0x40f3a1]
2021-10-27T15:27:14.149 I lnav_log.cc:159 uname:
2021-10-27T15:27:14.149 I lnav_log.cc:160 sysname=Linux
2021-10-27T15:27:14.149 I lnav_log.cc:161 nodename=varnish-master-distinct1.distinct.ro
2021-10-27T15:27:14.149 I lnav_log.cc:162 machine=x86_64
2021-10-27T15:27:14.149 I lnav_log.cc:163 release=5.4.78-2-pve
2021-10-27T15:27:14.149 I lnav_log.cc:164 version=#1 SMP PVE 5.4.78-2 (Thu, 03 Dec 2020 14:26:17 +0100)
2021-10-27T15:27:14.149 I lnav_log.cc:165 Environment:
2021-10-27T15:27:14.149 I lnav_log.cc:166 HOME=/root
2021-10-27T15:27:14.149 I lnav_log.cc:167 LANG=ro_RO.UTF-8
2021-10-27T15:27:14.149 I lnav_log.cc:168 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
2021-10-27T15:27:14.149 I lnav_log.cc:169 TERM=xterm
2021-10-27T15:27:14.149 I lnav_log.cc:170 TZ=(null)
2021-10-27T15:27:14.149 I lnav_log.cc:171 Process:
2021-10-27T15:27:14.149 I lnav_log.cc:172 pid=25584
2021-10-27T15:27:14.149 I lnav_log.cc:173 ppid=19097
2021-10-27T15:27:14.149 I lnav_log.cc:174 pgrp=25584
2021-10-27T15:27:14.149 I lnav_log.cc:175 uid=0
2021-10-27T15:27:14.149 I lnav_log.cc:176 gid=0
2021-10-27T15:27:14.149 I lnav_log.cc:177 euid=0
2021-10-27T15:27:14.149 I lnav_log.cc:178 egid=0
2021-10-27T15:27:14.149 I lnav_log.cc:180 cwd=/var/log/nginx
2021-10-27T15:27:14.149 I lnav_log.cc:181 Executable:
2021-10-27T15:27:14.149 I lnav_log.cc:182 version=lnav 0.7.3
Ohh and this is what I get right after I am typing the lnav command:
[[email protected] nginx]# lnav access.log
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/$schema:unexpected data, expecting one of the following data types --
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/$schema:accepted paths --
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/$schema: ^/\w+/regex/[^/]+/pattern$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/$schema: ^/\w+/(json|convert-to-local-time|epoch-timestamp)$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/$schema: ^/\w+/timestamp-divisor$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/$schema: ^/\w+/(file-pattern|level-field|timestamp-field|body-field|url|url#|title|description|timestamp-format#)$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/$schema: ^/\w+/level/(trace|debug\d*|info|stats|warning|error|critical|fatal)$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/$schema: ^/\w+/value/\w+/(kind|collate|unit/field)$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/$schema: ^/\w+/value/\w+/(identifier|foreign-key|hidden)$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/$schema: ^/\w+/value/\w+/unit/scaling-factor/.*$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/$schema: ^/\w+/value/\w+/action-list#
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/$schema: ^/\w+/action/\w+/label
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/$schema: ^/\w+/action/\w+/capture-output
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/$schema: ^/\w+/action/\w+/cmd#
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/$schema: ^/\w+/sample#/line$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/$schema: ^/\w+/line-format#/(field|default-value)$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/$schema: ^/\w+/line-format#/min-width$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/$schema: ^/\w+/line-format#$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/multiline:unexpected data, expecting one of the following data types --
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/multiline:accepted paths --
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/multiline: ^/\w+/regex/[^/]+/pattern$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/multiline: ^/\w+/(json|convert-to-local-time|epoch-timestamp)$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/multiline: ^/\w+/timestamp-divisor$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/multiline: ^/\w+/(file-pattern|level-field|timestamp-field|body-field|url|url#|title|description|timestamp-format#)$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/multiline: ^/\w+/level/(trace|debug\d*|info|stats|warning|error|critical|fatal)$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/multiline: ^/\w+/value/\w+/(kind|collate|unit/field)$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/multiline: ^/\w+/value/\w+/(identifier|foreign-key|hidden)$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/multiline: ^/\w+/value/\w+/unit/scaling-factor/.*$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/multiline: ^/\w+/value/\w+/action-list#
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/multiline: ^/\w+/action/\w+/label
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/multiline: ^/\w+/action/\w+/capture-output
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/multiline: ^/\w+/action/\w+/cmd#
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/multiline: ^/\w+/sample#/line$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/multiline: ^/\w+/line-format#/(field|default-value)$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/multiline: ^/\w+/line-format#/min-width$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/multiline: ^/\w+/line-format#$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/opid-field:unexpected data, expecting one of the following data types --
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/opid-field:accepted paths --
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/opid-field: ^/\w+/regex/[^/]+/pattern$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/opid-field: ^/\w+/(json|convert-to-local-time|epoch-timestamp)$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/opid-field: ^/\w+/timestamp-divisor$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/opid-field: ^/\w+/(file-pattern|level-field|timestamp-field|body-field|url|url#|title|description|timestamp-format#)$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/opid-field: ^/\w+/level/(trace|debug\d*|info|stats|warning|error|critical|fatal)$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/opid-field: ^/\w+/value/\w+/(kind|collate|unit/field)$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/opid-field: ^/\w+/value/\w+/(identifier|foreign-key|hidden)$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/opid-field: ^/\w+/value/\w+/unit/scaling-factor/.*$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/opid-field: ^/\w+/value/\w+/action-list#
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/opid-field: ^/\w+/action/\w+/label
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/opid-field: ^/\w+/action/\w+/capture-output
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/opid-field: ^/\w+/action/\w+/cmd#
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/opid-field: ^/\w+/sample#/line$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/opid-field: ^/\w+/line-format#/(field|default-value)$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/opid-field: ^/\w+/line-format#/min-width$
warning:/root/.lnav//formats/installed/micu_nginx_log.json:/micu_nginx_log/opid-field: ^/\w+/line-format#$
==== GURU MEDITATION ====
Unfortunately, lnav has crashed, sorry for the inconvenience.
You can help improve lnav by sending the following file to [email protected] :
/root/.lnav/crash/crash-2021-10-27-16-41-14.42566.log
=========================
It looks like you're using a really old version of lnav:
2021-10-27T15:27:14.149 I lnav_log.cc:182 version=lnav 0.7.3
v0.7.3 is from 2015. Can you please try a more recent version, like the latest beta:
https://github.com/tstack/lnav/releases/tag/v0.10.1-beta1
I was able to open lnav without crashing on the 0.8.2 version but the company does not want to upgrade from the 0.7.3 version.
I sow the Releases but I was not able to identify exactly what is different on the version that worked. Could you help me with another hint here?
Maybe I can modify the log format so that it meets the requirements of lnav 0.7.3.
Same here lnav 0.7.3 is in CentOS-7 release repo and compiling from source is not an option at the moment/server.
I also try to write a custom format and lnav 0.7.3 crashes with 2022-04-19T11:21:29.849 E pcrepp.hh:422 pcre err -8 2022-04-19T11:21:29.849 E pcrepp.hh:422 pcre err -8
Maybe related to #352 as it has the same error message, i.e. pcre err -8 and my custom logformat (weblogic) also contains larger Java stack traces.
@tstack this /multiline:unexpected data, expecting one of the following data types -- is raised because "multiline" : true / false, is not a recognized JSON Path in lnav-0.7.3. This may have been misleading.
I had the /usr/bin/lnav on the path and not lnav-0.10.1. Sorry for the noise
~~Is there an option to get more Debug output on pcrepp.hh:422 pcre err -8 ?~~
@MicuAlexandru so this opid-field:unexpected data, expecting one of the following data types -- means opid-field is not recognized by the JSON logformat.json parser either.
@stefan123t I was able to take out multiline and opit-field from the logformat.json but still no luck. now I get something about Format collision, format 'access_log' matches sample from 'micu_nginx_log'
I think that this has something to do with lnav 0.7.3 not working with nested JSON.
Have you found a fix for this?
@MicuAlexandru sorry I do not parse JSON logs. Only things I parse are some weblogic logs which have <> field syntax spanning multiple lines (especially the message / body with stack traces). It is a bit awkward to set up a RegEx for that, but keep in mind that lnav does require you to only match the first line of a multiline block in order to detect the right logfile format. So you have to make your match pattern as general as possible, eventually having several optional field matching groups (?<group>blablapattern)? indicated by the optional ? at the end of the group.
Please reconsider if you are able to install the current "static" binary of lnav-0.10.1 as it has a lot more updates, e.g. JSON logfile support which I am not aware of, as I do not use it. But also the above unrecognized patterns in your micu_nginx_log.json may have been improved with some valuable feedback. Just give it a try on a machine with some excerpts from your logs and then try to adapt the working 0.10.1 micu_nginx_log.json back to 0.7.3 as a possible approach to narrow down your issue.