tstack
Switch from using SQLite to using DuckDB
Is your feature request related to a problem? Please describe. When using LNAV to do after incident analytics I often need to export out data from LNAV to CSV and load it into a CSV and load that into duck db to do further querying. The magical parsing of properties that LNAV does is amazing, but the query side is lacking due to inherent limitations of SQLite.
Describe the solution you'd like Basically replace SQLite with DuckDB which contains more features for analytics workloads. It is built to be embedded and more aimed at analytics workloads.
Describe alternatives you've considered I know that this is a tall order, but I do think it would be an amazing upgrade.
I have to say, it's pretty rare for me to do analytical queries in lnav. So, I haven't felt the limitations. I'd be really interesting in hearing some examples of what you're doing and some example queries. I have tried to add various functions and PRQL to make analysis easier. But, since I don't use it in anger much, I don't see the gaps.
At this point, SQLite is woven throughout lnav. So, it's not practical to replace. It goes beyond exposing log data through vtables. A lot of the lnav internals are exposed through SQLite vtables as well so that operations can be scripted. Also, since the log tables are directly backed by the log files, things are inherently row-based and not columnar like DuckDB would want.
I could explore making it easier to export log data to DuckDB. Maybe even making it possible to write/run DuckDB queries inside of lnav.
Thanks for taking the time to write this up.
