"lnav logfile" and "cat logfile | lnav" not the same for larger file sizes

[dardhal]

lnav version Latest 0.13-rc4 compiled from source

Describe the bug It is at least my expectation that running "lnav" against a log file is the same as running "lnav" for the contents of the log file being piped through "stdin".

Obviously, if I have to open a given file, I would not feed it to "lnav" using "cat file | lnav", but sometimes I can to do some good'old UNIX text processing of the original logs, for example, to not feed "lnav" with tons of entries I am not interested in (I am talking here input files weighing tens of GiB and having hundreds of millions of entries), so that I leave "lnav" to do what it is most useful to me, like real-time complex filtering and the SQL interface.

Also the server I need to do this processing is a shared one, and I would like to keep lnav's process memory footprint at a minimum, as processing those amounts of log data from lnav alone can easily have 5 - 6 GiB RAM resident, and this is a 64 GiB VM I have to share with many others doing other stuff.

I have noticed when the amounts of data resulting from UNIX piping fed into lnav exceed some amount, "lnav" is dropping lines coming in from "stdin". Don't know if this is format or log entries specific, but in my small testing (have a log file which I do "head" for to create a "test-file.log" for testing) :

• I am hitting the limit when lnav starts to misbehave at some point in between 340,000 and 350,000 log lines (33724220 bytes big for the former, 34720433 for the latter)
• When loading up a test log file with the first 340,000 lines of the original file (cat test-file.log | lnav), I get shown all 340,000 lines on the "lnav" interface, and the files section looks like this :

  ◆ stdin       10.0MB 
  ◆ stdin.1     10.0MB 
  ◆ stdin.2     10.0MB 
+ ◆ stdin.3      9.3MB 

• When the test file is created for the first 350,000 log lines in the original log file, doing "cat test-file.log | lnav" only shows me 263,506 lines, and the files section is showing like this :

+ ◆ stdin.1     10.0MB 
  ◆ stdin.2     10.0MB 
  ◆ stdin.3     10.0MB 
  ◆ stdin.4    486.3KB 

In between all attempts I made sure to do a "lnav -m piper clean", just in case.

No errors are shown or printed, some of the lines coming in from the stdin seem to just be dropped silently.

To Reproduce As described above.

Assuming this is some sort of issue with lnav "piper" processing, I made a change to my configuration, so that the piper max size was increased doubled from default (:config /tuning/piper/max-size 20971520), and next the test with 350,000 lines coming in from stdin was OK. Same effect was achieved with increasing the piper "rotations" configuration from 4 to 10.

Of course, this makes "lnav" behave as expected on increasingly large log data coming in from "stdin", as much as you increase the combination of rotations and max-size allowed for each one.

Don't know if this is expected behavior or not, but silently dropping input data does not look good. There may be use cases beyond mine which may be affected as well ("lanv -e" for example), so it may be worth a check.

I am certainly surprised "lnav" behaves like this, as it was my understanding it would insert each log line processed in the SQLite tables, hence that lots of data comes in through "stdin" shouldn't be making a real difference compared to reading from file, but I may be wrong.

Thanks a lot .

[tstack]

Don't know if this is expected behavior or not,

It is expected behavior since lnav cannot know how much data will be coming in. So, it needs to be rotated and dropped at some point, otherwise it would fill the disk.

but silently dropping input data does not look good.

No, it doesn't. I'll try to figure out a way to make the behavior more clear. Sorry this was a surprise.

I am certainly surprised "lnav" behaves like this, as it was my understanding it would insert each log line processed in the SQLite tables

The SQLite log tables are virtual tables that are backed by the log files themselves. lnav is not inserting the log messages into tables in a SQLite database file.

[dardhal]

Nothing to be sorry @tstack , this is an incredible piece of software and I kind of hope by making this tool as useful as possible at my current employer, you may get some well deserved recognition and compensation.

It's good to know this is not a defect but as per design, I just need to keep our internal docs updated with a warning and the way to accept more data being piped in by increasing the number of piper max size and rotations.

Obviously, there will be a human factor (people not reading the docsa), so if when "lnav" does the ingesting from stdin it has to eventually drop data, some kind of warning being printed on the interface would be great (for the longer term).