fluent-bit out_cloudwatch_logs: log_key option demands map formatted log

Signed-off-by: Matthew Fala [email protected]

Problem Summary

Fluent Bit crashes upon receiving long log without expected log_key (out_cloudwatch_logs) res_large

Expected log_key usage:

map dbg_map

Note that map size is 2, since there are 2 keys.

Before fix:

regular dbg_regular Note that map size is incorrectly describing character count 28, though provided a string. Fluent bit does not crash

Screen Shot 2021-10-04 at 3 37 03 PM Screen Shot 2021-10-04 at 3 36 42 PM res_large Note that map size is incorrectly describing character count 262116. kv + j evaluates to kv + 262116 which is out of the memory bounds of Fluent Bit. Fluent bit crashes

Fix description

Add a type check sentinel at the start of the msgpack map key evaluator code. If msgpack log object is not a map, then pass over the log and continue to the next.

After fix:

regular dbg_regular The log is now type checked, and passed over if not map type.

large dbg_large res_large The large log which previously crashed fluent bit no longer crashes fluent bit.

Test Configuration

[SERVICE]
     Grace 30
     Log_Level debug

# Provide entry point for logs
[INPUT]
     Name http
     host 0.0.0.0
     port 8888

[OUTPUT]
     Name cloudwatch_logs
     Match *
     log_stream_prefix test
     log_group_name fluent_replay_all
     auto_create_group true
     region us-west-2
     log_key <log_key>

Tests

The following logs were sent to Fluent Bit by Postman through HTTP input plugin. The log_key configuration option was set to "key" and omitted, and the results of each log on Cloud Watch and Fluent Bit is recorded below

Description	log_key	Value	Result
Long map	"key"	{"key": "xxx...xx(long log)"}	map[key] value truncated and sent to CW
Short map	"key"	{"key": "xxxx"}	map[key] value sent to CW
Long string	"key"	"xxx...xx(long log)"	Nothing sent to CW, Fluent Bit does not fail
Short string	"key"	"xxxx"	Nothing sent to CW, Fluent Bit does not fail
Long map	(none)	{"key": "xxx...xx(long log)"}	Truncated and sent as string to CW
Short map	(none)	{"key": "xxxx"}	Sent as string to CW as string
Long string	(none)	"xxx.xx(long log)"	Truncated and sent to CW
Short string	(none)	"xxxx"	Sent to CW

Enter [N/A] in the box, if an item is not applicable to your change.

Testing Before we can approve your change; please submit the following in a comment:

[x] Example configuration file for the change
[x] Debug log output from testing the change

[N/A] Attached Valgrind output that shows no leaks or memory corruption was found

Documentation

[N/A] Documentation required for this feature

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.