tstack
sql handling of long TEXT, eg log_body
lnav version v0.13.1
Describe the bug I was confused about the data I was seeing in SQL summaries that included log_body. The text seemed to be broken when there were long lines. It looks like maybe this is a feature to display the first part and then the end of message when length exceeds a certain value. I looked for mention of this in docs but failed. It does seem quite a good idea in some ways, though in some cases it was creating text shorted than available screen width for the column. I'm also unclear if selects would work if the search string is in the 'ellipsed' part of the text, but suspect it is just a display trick. The problem was compounded by the fact that the ... was hardly visible on my screen until I just pasted it from plain text view. It looked more like a "-". (Many small pixels.)
In SQL results: : (CoreData) [com.apple.coredata:error] fault: Unable to cr⋯eServerEndpointFactory 0x14a60c290 -newEndpoint returned nil
From field: Family: (CoreData) [com.apple.coredata:error] fault: Unable to create token NSXPCConnection. NSXPCStoreServerEndpointFactory 0x14a60c290 -newEndpoint returned nil
To Reproduce ;select log_body from all_logs where log_body like '%0x14a60c290%';
Maybe if string shortening is done a flag or note could be shown??
Yes, column widths are limited to 120 cells and that doesn't seem to be documented. I'll use this bug to track updating the docs.
I'm also unclear if selects would work if the search string is in the 'ellipsed' part of the text, but suspect it is just a display trick.
The limit is only for display and doesn't affect the raw data. Note that if you want a copy of the cell contents, you can press p to open the overlay, press CTRL+] to change the focus to the overlay, and then press c to copy the full contents.
The problem was compounded by the fact that the ... was hardly visible on my screen until I just pasted it from plain text view. It looked more like a "-". (Many small pixels.)
Any special characters in the display should be highlighted with a yellow color. The following is a screenshot that shows long log_body columns where the ellipsis is highlighted in yellow along with a line-feed (\n) and a tab (\t):
Always so much more, great. You won't be seeing this depth from AI code for some time I suspect.
I think I used :config to /ui/theme sometime and so have probably lost some of those basic features. I have dug around a bit in the settings folder and lifted a new default setting files from /configs but still seem to not have basic interface as default. For instance I have dark red text on back as default which is very hard to read. If I use the : config /ui/theme I can get better visibility but loose the sort of detail on showing special colours as you have above. How do I return to the plain default set up? (I tried :config default) For legibility now I am using "theme": "modus-operandi", (iTerm2, MacOS)
Also noticing on third line of display: ┌Filtering the log data using "logType == 16" [This might arise from log stream --predicate 'messageType == error' |lnav - but I do not see it in plain output to head in terminal] But I fail to see where this is arising, I have not set any filters AFAIK, again this is log stream output from macOS. Nor have I found how to investigate it.
Probably a different point but I notice the fist line of these logs is a header: Timestamp Thread Type Activity PID TTL Is it possible or sensible to make use of this in some way? The logs get seen as "generic_log" and perhaps for generic logs a quick check for a header row could be worthwhile -OR- provide the ability to set a line as a header that is the displayed at the top of screen while the content scrolls in the region below?
I know this is somewhat at odds with the collection of multiple log inputs, and might not be very good UI to have multiple header rows flashing in and out as a user scrolls down lines from different logs... With the unified logs I am >95% just looking at the one source.
Just to push a little further re macOS... how far are we from being able to recognise and handle/colour field and patterns from unified logging? Does the header line help? Is there a best existing start point that I could take and make some attempt with?
Sorry if I am being a bit dense, have a cold/flu that is slowing me and keeping me from normal activities!
IF you are considering value of unified logging work then the biggest other problem is the extended multi-line log: 2025-11-26 14:06:03.859783+0000 0x7ed Error 0x0 142 0 usbmuxd: [com.apple.usbmux:bonjour] _SendAttachNotification sending attach for device 'fc:66:cf:cd:52:c9@fe80::fe66:cfff:fecd:52c9-supportsRP-24._apple-mobdev2._tcp.local.': { AttachedInterfaceIndex = 6; FullEscapedServiceName = "fc:66:cf:cd:52:c9@fe80::fe66:cfff:fecd:52c9-supportsRP-24._apple-mobdev2._tcp.local."; InterfaceIndex = 6; PortNumber = 32498; SockaddrStorage = {length = 128, bytes = 0x1c1e0000 00000000 fe800000 00000000 ... 00000000 00000000 }; TXTRecord = ""; UniqueDeviceID = "00008101-0014091636D8001E"; WiFiMACAddress = "fc:66:cf:cd:52:c9"; } (_GetAddrInfoReplyReceivedCallback matched).
Many people seem stymied by the mass of data from unified logging, so I think a tool to help would be valued!
How do I return to the plain default set up? (I tried :config default)
You can set the default theme with:
:config /ui/theme default
The screenshot I posted is the "night-owl" theme in a macOS Terminal.app window.
┌Filtering the log data using "logType == 16"
That seems to come from log stream --predicate 'messageType == error'
Probably a different point but I notice the fist line of these logs is a header:
Timestamp Thread Type Activity PID TTLIs it possible or sensible to make use of this in some way?
I'd have to build in some code to recognize columnar formats like this. There's some existing code that handles the self-describing "bro" and "w3c" formats. So, it's possible.
But, it would be easier to just use log stream --style=ndjson" since lnav can work with JSON-lines streams. I'll look into getting one added for the upcoming release, shouldn't be too hard.
I've pushed a change that adds a macosuni_log format that should work with the output of log stream --style=ndjson. I just took a guess as to what values to capture and put in the line-format since I don't make use of the service. Feel free to make suggestions of what might be more useful.
Brilliant! I have spent an hour not not yet seen anything I could improve - together with the p-key detail view and SQL I can do so much more than before - and the display line is used economically too. For anyone who has wanted to look at unified log output on macOS I highly recommend. [ALL I needed was to install the macosuni_log.json format file....] 10/10 ***** and then log stream --style=ndjson |lnav