New fields in a search result are incorrectly shown as type = unknown

[drewmiranda-gl]

When viewing the search page, and refreshing the search to show the newest messages (NOT a full browser page refresh), new messages are displayed, however, if a field that has never been seen by graylog appears in any of these new messages, graylog incorrectly says the type = unknown

If you do a FULL browser page refresh you do get the correct field mapping.

Expected Behavior

New fields are shown with the correct type. For numeric fields this disallows using aggregations/charts that require int fields.

Current Behavior

New fields show as unknown type, its impossible to use the chart feature or add them to an aggregation on the page without fulling refreshing the entire page.

Possible Solution

Steps to Reproduce (for bugs)

1. Open a search page that displays messages
2. Send a message to the graylog cluster that contains a field never before seen by the cluster
3. Click the green search button to update the search result

Context

I was putting together log parsing from a text file and was super confused why i couldn't get the field type to display as long/int despite using to_long via a pipeline rule. I then noticed ALL new fields showed as type unknown which is very unusual. After about 30 min of troubleshooting i realized what was going on.

Your Environment

• Graylog Version: Graylog 4.3.5+32fa802
• Java Version: 11.0.16
• Elasticsearch Version: Opensearch 1.3.4
• MongoDB Version: 4.4.15
• Operating System: Ubuntu Server 20.04 LTS
• Browser version: Chrome 104

Please let me know if you have any questions or wish to discuss! Feel free to zoom/slack me :)

[linuspahl]

It looks like this is related to the fields list being outdated and we need to ensure it is being refetched.

[dennisoelkers]

@drewmiranda-gl: When a message has just been ingested and the frontend did not refresh its field types, this kind of behavior is expected. It would be hard to push the information about message fields which have just been ingested for the first time out to all clients with our current infrastructure. But, I would be surprised if a full-page refresh is really necessary, just triggering another search execution should also fetch new field types.