tstack
Support attaching a format to a json string field
There are scenarios where I want to use journald to store logs but don't control the logging output format of a service (in this case glog). It would be really cool if we could tell lnav to parse the MESSAGE field of the journald json output using one of the existing formats so it can be recognized and colorized/indexed. Does this work already today or should I open a PR to add this?
There’s some very basic support for this, but it still needs work. Can you provide some example log files that demonstrate what you want.
See #571
Can you provide some example log files that demonstrate what you want.
I'm not the original poster, but I'd also be interested in this. I have Keycloak and Filebeat both configured to output JSON log messages to journald. Example output (_HOSTNAME and _CMDLINE redacted) :
# journalctl -f -o json
…
{ "__CURSOR" : "s=38f25861caa643f6a95cae52951e3708;i=4532b;b=329d0974dc8c4c2e97f8888ec3f86c0e;m=7833006dd26;t=602b9f24b4e29;x=1409856c57e18800", "__REALTIME_TIMESTAMP" : "1691847517490729", "__MONOTONIC_TIMESTAMP" : "8260027866406", "_BOOT_ID" : "329d0974dc8c4c2e97f8888ec3f86c0e", "PRIORITY" : "6", "SYSLOG_FACILITY" : "3", "_CAP_EFFECTIVE" : "0", "_AUDIT_LOGINUID" : "0", "_MACHINE_ID" : "367f56be83f6488f989ec087164d4971", "_HOSTNAME" : "example.com", "_TRANSPORT" : "stdout", "SYSLOG_IDENTIFIER" : "java", "_UID" : "1001", "_GID" : "1001", "_COMM" : "(java)", "_EXE" : "/usr/lib/jvm/java/bin/java", "_CMDLINE" : "/bin/java … io.quarkus.bootstrap.runner.QuarkusEntryPoint start …", "_STREAM_ID" : "5d255c5ba6f448d08a8b44984ce42daf", "_PID" : "76020", "MESSAGE" : "{\"timestamp\":\"2023-08-12T09:38:37.49-04:00\",\"sequence\":9727,\"loggerClassName\":\"org.jboss.logging.Logger\",\"loggerName\":\"org.keycloak.events.jpa.JpaEventStoreProvider\",\"level\":\"DEBUG\",\"message\":\"Cleared 0 expired events in all realms\",\"threadName\":\"Timer-0\",\"threadId\":27,\"mdc\":{},\"ndc\":\"\",\"hostName\":\"example.com\",\"processName\":\"QuarkusEntryPoint\",\"processId\":76020}" }
{ "__CURSOR" : "s=38f25861caa643f6a95cae52951e3708;i=4532e;b=329d0974dc8c4c2e97f8888ec3f86c0e;m=78335acdace;t=602b9f7f1134c;x=8ff1679a7747cc64", "__REALTIME_TIMESTAMP" : "1691847612240716", "__MONOTONIC_TIMESTAMP" : "8260122630862", "_BOOT_ID" : "329d0974dc8c4c2e97f8888ec3f86c0e", "PRIORITY" : "6", "SYSLOG_FACILITY" : "3", "_UID" : "0", "_GID" : "0", "_CAP_EFFECTIVE" : "0", "_AUDIT_LOGINUID" : "0", "_MACHINE_ID" : "367f56be83f6488f989ec087164d4971", "_HOSTNAME" : "example.com", "_TRANSPORT" : "stdout", "SYSLOG_IDENTIFIER" : "filebeat", "_COMM" : "(filebeat)", "_EXE" : "/usr/share/filebeat/bin/filebeat", "_CMDLINE" : "/usr/share/filebeat/bin/filebeat …", "_SYSTEMD_UNIT" : "filebeat.service", "_STREAM_ID" : "e61a50faf7c04ece9fda9119d190e790", "_PID" : "41203", "_SYSTEMD_INVOCATION_ID" : "055bd474e93e44f9b3dabec69a0253a4", "MESSAGE" : "{\"log.level\":\"info\",\"@timestamp\":\"2023-08-12T09:40:12.240-0400\",\"log.logger\":\"input.filestream\",\"log.origin\":{\"file.name\":\"filestream/input.go\",\"file.line\":336},\"message\":\"Reader was closed. Closing. Path='/var/log/munin-node/munin-node.log'\",\"service.name\":\"filebeat\",\"id\":\"muninnode\",\"source_file\":\"filestream::muninnode::native::64224-51969\",\"path\":\"/var/log/munin-node/munin-node.log\",\"state-id\":\"native::64224-51969\",\"ecs.version\":\"1.6.0\"}" }
…
When browsing with lnav, I'd like to be able to parse the JSON MESSAGE text and show just a few of the fields, so they fit in better with typical journald log messages (where the MESSAGE text is not JSON).
Current lnav output, where it requires horizontal scrolling to see the interesting parts of the JSON log messages:
There’s some very basic support for this, but it still needs work.
@tstack, could you give an example of the existing basic support?
Edit: I found https://github.com/tstack/lnav/blob/917270e293e47cd1234ac60af3756a8e983ccd65/test/formats/jsontest/format.json#L28 which uses rewriter to modify field values, so I was thinking I might be able to use that to reformat JSON-in-JSON log lines. But I haven't been able to get it to work — in lnav 0.11.2, the output doesn't match the expected output in https://github.com/tstack/lnav/blob/917270e293e47cd1234ac60af3756a8e983ccd65/test/expected/test_json_format.sh_a06b3cdd46b387e72d6faa4cce648b8b11ae870b.out#L9-L10 — the rewriter directive seems to have no effect.
As documented, rewriter directive works only when pretty-printing the message (shift+P hotkey)
