Allow inputs to automatically clean fields

[drewmiranda-gl]

What?

Graylog disallows message fields that do not adhere to a specific pattern: https://github.com/Graylog2/graylog2-server/blob/496233da162a439a1e1cb6b63b6485bad7720c5a/graylog2-server/src/main/java/org/graylog2/plugin/Message.java#L219

When a user sets/creates fields, such as via a pipeline and using set_field or set_fieldsthey have access to theclean_fields` argument.

However, when using some inputs, such as beats, the user cannot control how fields are delivered to the graylog input.

This issue is asking to have a checkbox, configured on the input's configuration page, that, when enabled, will do the same thing that clean_fields does.

Why?

The user does not have the ability to correct invalid field names that arrive to the graylog input already invalid.

[tellistone]

for anyone else ignorant of this function

[drewmiranda-gl]

Adding additional context, i noticed these in my forwarder.log of messages coming directly from a beat agent:

INFO  [Message] Ignoring invalid or reserved key winlogbeat_winlog_event_data_Product Name for message 2d8b7471-9d60-11ef-a7ba-bc2411cce508
INFO  [Message] Ignoring invalid or reserved key winlogbeat_winlog_event_data_Scan ID for message 2d8b7471-9d60-11ef-a7ba-bc2411cce508
INFO  [Message] Ignoring invalid or reserved key winlogbeat_winlog_event_data_Scan Type Index for message 2d8b7471-9d60-11ef-a7ba-bc2411cce508

[chalfling]

This also a problem with JSON Extractor. I have a PanOS source in JSON with "/" in the name of fields, like: "tunnel_id/imsi" and I got several "Ignoring invalid or reserved key" erros until Graylog freezes and don't process anything more.

[miwent]

This also a problem with JSON Extractor. I have a PanOS source in JSON with "/" in the name of fields, like: "tunnel_id/imsi" and I got several "Ignoring invalid or reserved key" erros until Graylog freezes and don't process anything more.

@chalfling Are you parsing the JSON logs in a pipeline rule then creating fields with set_fields()? The issue is probably when Graylog is attempting to convert the map generated by the JSON parser into fields. The set_fields() function has had a parameter clean_fields added recently, if you add the parameter clean_fields: true to set_fields() it might address this, converting the illegal field name characters to underscores.

[chalfling]

This also a problem with JSON Extractor. I have a PanOS source in JSON with "/" in the name of fields, like: "tunnel_id/imsi" and I got several "Ignoring invalid or reserved key" erros until Graylog freezes and don't process anything more.

@chalfling Are you parsing the JSON logs in a pipeline rule then creating fields with set_fields()? The issue is probably when Graylog is attempting to convert the map generated by the JSON parser into fields. The set_fields() function has had a parameter clean_fields added recently, if you add the parameter clean_fields: true to set_fields() it might address this, converting the illegal field name characters to underscores.

Hello @miwent !

I'm not using pipeline. I'm using Default JSON Extractor from Message Filter Chain's Processor.

[miwent]

@chalfling I'd recommend switching from the input extractors to using pipelines to process the messages, it will give you more flexibility and control, and using the clean fields parameter you'll be able to address the illegal field names derived from your JSON.