tstack
Format for multiline log files
Hi!
I'm trying to define a format for a log file that looks like this:
!@,ACSTRNGeneratorBeanTF,2017-06-20 00:53:11.671,joviancut4a,jupnms,-1,java,unknownFile,noline:
ACSTRNGeneratorBean:sendMessage() got error response for ESN 10000058 with error code: 111 and error msg: Anothe test
!@,trnBacklogEJBTF,2017-06-20 00:53:11.673,joviancut4a,jupnms,-1,java,unknownFile,noline:
TRNBacklogEJB.updateTRNRetryCount(): ENTERING for ESN: 10000058
with the lines starting with !@ having a timestamp and the following lines having the trace message.
So I defined this format (I tried to be as generic as possible):
{
"trace_log" : {
"title" : "Trace Log Format",
"description" : "The log format used for traces",
"url" : "",
"regex" : {
"basic" : {
"pattern" : "^!@(.*),(?<timestamp>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}\\.\\d{3}),.*:(\\n(?<message>(.*\\n)*))?$"
}
},
"timestamp-format" : [
"%Y-%m-%d %H:%M:%S.%L"
],
"value" : {
"message" : {
"kind" : "string"
}
},
"sample" : [
{
"line" : "!@,acsTRNGeneratorEJBTF,2017-06-20 00:53:11.591,joviancut4a,jupnms,-1,java,unknownFile,noline:\nAfter sleeping500ms the retrieved ACS Terminal state is 2\n"
}
]
}
}
But I'm noticing that the regex only matches with the first line (that starts with !@) and not the second. Is there a way to group lines such that every line containing a timestamp and all subsequent lines (until another timestamp is reached) will be considered one entity or how would I modify my regex to reflect this?
Also, is there a way to omit parts of a line from the file viewer completely (I want to omit the !@ from every line)? Or would that be done by hiding the fields?
Thanks!
I made a small tweak, not sure if it actually does what you want:
{
"trace_log" : {
"title" : "Trace Log Format",
"description" : "The log format used for traces",
"url" : "",
"regex" : {
"basic" : {
"pattern" : "^!@(.*),(?<timestamp>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}\\.\\d{3}),(?<tags>[^:]*):\\n?(?<body>(?:.|\\n)*)$"
}
},
"timestamp-format" : [
"%Y-%m-%d %H:%M:%S.%L"
],
"value" : {
"tags" : {
"kind" : "string"
}
},
"sample" : [
{
"line" : "!@,acsTRNGeneratorEJBTF,2017-06-20 00:53:11.591,joviancut4a,jupnms,-1,java,unknownFile,noline:\nAfter sleeping500ms the retrieved ACS Terminal state is 2\n"
}
]
}
}
You would want to name the capture 'body' instead of 'message' since that is the builtin name used in a few places, like the automatic extraction functionality.
Also, is there a way to omit parts of a line from the file viewer completely (I want to omit the !@ from every line)? Or would that be done by hiding the fields?
Not at this time, unfortunately. There is functionality to hide fields, but the hidden fields are replaced with an ellipsis or not at all if the field is shorter than the ellipsis, which it is in this case.
Thank you! I was also wondering if I could group lines such that if I have this:
!@,ACSTRNGeneratorBeanTF,2017-06-20 00:53:11.671,joviancut4a,jupnms,-1,java,unknownFile,noline:
ACSTRNGeneratorBean:sendMessage() got error response for ESN 10000058 with error code: 111 and error msg: Anothe test
!@,trnBacklogEJBTF,2017-06-20 00:53:11.673,joviancut4a,jupnms,-1,java,unknownFile,noline:
TRNBacklogEJB.updateTRNRetryCount(): ENTERING for ESN: 10000058
it would show up like this on the file viewer:
!@,ACSTRNGeneratorBeanTF,2017-06-20 00:53:11.671,joviancut4a,jupnms,-1,java,unknownFile,noline:
ACSTRNGeneratorBean:sendMessage() got error response for ESN 10000058 with error code: 111 and error msg: Anothe test
!@,trnBacklogEJBTF,2017-06-20 00:53:11.673,joviancut4a,jupnms,-1,java,unknownFile,noline:
TRNBacklogEJB.updateTRNRetryCount(): ENTERING for ESN: 10000058
Can that be done or is that based more on the format of the log file itself?
Sorry, I'm not sure what you mean by 'group lines'. I'm not seeing much of a difference in the example.
But, what is displayed is based on whatever is in the log file. There is only a little bit of manipulation that lnav can do when displaying the log messages, like hiding fields.
If I understand @plav567 right, he wants some pre-processing of the logfile before it is displayed,
e.g. something that indents everything not starting with !@, and/or
adding an empty new line after/before every block.
!@,ACSTRNGeneratorBeanTF,2017-06-20 00:53:11.671,joviancut4a,jupnms,-1,java,unknownFile,noline:
ACSTRNGeneratorBean:sendMessage() got error response for ESN 10000058 with error code: 111 and error msg: Anothe test
!@,trnBacklogEJBTF,2017-06-20 00:53:11.673,joviancut4a,jupnms,-1,java,unknownFile,noline:
TRNBacklogEJB.updateTRNRetryCount(): ENTERING for ESN: 10000058
