fluent-bit
fluent-bit copied to clipboard
fluent
out_http: default the output format to JSON.
Summary
Default out_http to JSON format so it works with in_http. Setting the output format to msgpack is still possible by setting it manually.
Enter [N/A] in the box, if an item is not applicable to your change.
Testing Before we can approve your change; please submit the following in a comment:
- [x] Example configuration file for the change
- [x] Debug log output from testing the change
- [x] Attached Valgrind output that shows no leaks or memory corruption was found
Documentation
- [ ] Documentation required for this feature
Backporting
- [ ] Backport to latest stable release.
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.
Here is a valgrind log:
$ valgrind ./bin/fluent-bit -i dummy -o http -p host=localhost -p port=8889 -f 1
==997298== Memcheck, a memory error detector
==997298== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==997298== Using Valgrind-3.22.0 and LibVEX; rerun with -h for copyright info
==997298== Command: ./bin/fluent-bit -i dummy -o http -p host=localhost -p port=8889 -f 1
==997298==
Fluent Bit v3.0.0
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
____________________
< Fluent Bit v2.2.2 >
-------------------
\
\
\ __---__
_- /--______
__--( / \ )XXXXXXXXXXX\v.
.-XXX( O O )XXXXXXXXXXXXXXX-
/XXX( U ) XXXXXXX\
/XXXXX( )--_ XXXXXXXXXXX\
/XXXXX/ ( O ) XXXXXX \XXXXX\
XXXXX/ / XXXXXX \__ \XXXXX
XXXXXX__/ XXXXXX \__---->
---___ XXX__/ XXXXXX \__ /
\- --__/ ___/\ XXXXXX / ___--/=
\-\ ___/ XXXXXX '--- XXXXXX
\-\/XXX\ XXXXXX /XXXXX
\XXXXXXXXX \ /XXXXX/
\XXXXXX > _/XXXXX/
\XXXXX--__/ __-- XXXX/
-XXXXXXXX--------------- XXXXXX-
\XXXXXXXXXXXXXXXXXXXXXXXXXX/
""VXXXXXXXXXXXXXXXXXXV""
[2024/02/16 19:12:23] [ info] [fluent bit] version=3.0.0, commit=bf767458b2, pid=997298
[2024/02/16 19:12:23] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/02/16 19:12:23] [ info] [output:http:http.0] worker #1 started
[2024/02/16 19:12:23] [ info] [output:http:http.0] worker #0 started
[2024/02/16 19:12:23] [ info] [cmetrics] version=0.6.6
[2024/02/16 19:12:23] [ info] [ctraces ] version=0.4.0
[2024/02/16 19:12:23] [ info] [input:dummy:dummy.0] initializing
[2024/02/16 19:12:23] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2024/02/16 19:12:23] [ info] [sp] stream processor started
[2024/02/16 19:12:25] [ info] [output:http:http.0] localhost:8889, HTTP status=201
[2024/02/16 19:12:26] [ info] [output:http:http.0] localhost:8889, HTTP status=201
[2024/02/16 19:12:27] [ info] [output:http:http.0] localhost:8889, HTTP status=201
[2024/02/16 19:12:28] [ info] [output:http:http.0] localhost:8889, HTTP status=201
[2024/02/16 19:12:29] [ info] [output:http:http.0] localhost:8889, HTTP status=201
[2024/02/16 19:12:30] [ info] [output:http:http.0] localhost:8889, HTTP status=201
[2024/02/16 19:12:31] [ info] [output:http:http.0] localhost:8889, HTTP status=201
[2024/02/16 19:12:32] [ info] [output:http:http.0] localhost:8889, HTTP status=201
[2024/02/16 19:12:33] [ info] [output:http:http.0] localhost:8889, HTTP status=201
^C[2024/02/16 19:12:33] [engine] caught signal (SIGINT)
[2024/02/16 19:12:33] [ warn] [engine] service will shutdown in max 5 seconds
[2024/02/16 19:12:33] [ info] [input] pausing dummy.0
[2024/02/16 19:12:33] [ info] [output:http:http.0] localhost:8889, HTTP status=201
[2024/02/16 19:12:34] [ info] [engine] service has stopped (0 pending tasks)
[2024/02/16 19:12:34] [ info] [input] pausing dummy.0
[2024/02/16 19:12:34] [ info] [output:http:http.0] thread worker #0 stopping...
[2024/02/16 19:12:34] [ info] [output:http:http.0] thread worker #0 stopped
[2024/02/16 19:12:34] [ info] [output:http:http.0] thread worker #1 stopping...
[2024/02/16 19:12:34] [ info] [output:http:http.0] thread worker #1 stopped
==997298==
==997298== HEAP SUMMARY:
==997298== in use at exit: 0 bytes in 0 blocks
==997298== total heap usage: 3,068 allocs, 3,068 frees, 3,986,697 bytes allocated
==997298==
==997298== All heap blocks were freed -- no leaks are possible
==997298==
==997298== For lists of detected and suppressed errors, rerun with: -s
==997298== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
Here is a debug log:
$ ./bin/fluent-bit -v -i dummy -o http -p host=localhost -p port=8889 -f 1
Fluent Bit v3.0.0
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
____________________
< Fluent Bit v2.2.2 >
-------------------
\
\
\ __---__
_- /--______
__--( / \ )XXXXXXXXXXX\v.
.-XXX( O O )XXXXXXXXXXXXXXX-
/XXX( U ) XXXXXXX\
/XXXXX( )--_ XXXXXXXXXXX\
/XXXXX/ ( O ) XXXXXX \XXXXX\
XXXXX/ / XXXXXX \__ \XXXXX
XXXXXX__/ XXXXXX \__---->
---___ XXX__/ XXXXXX \__ /
\- --__/ ___/\ XXXXXX / ___--/=
\-\ ___/ XXXXXX '--- XXXXXX
\-\/XXX\ XXXXXX /XXXXX
\XXXXXXXXX \ /XXXXX/
\XXXXXX > _/XXXXX/
\XXXXX--__/ __-- XXXX/
-XXXXXXXX--------------- XXXXXX-
\XXXXXXXXXXXXXXXXXXXXXXXXXX/
""VXXXXXXXXXXXXXXXXXXV""
[2024/02/16 19:16:02] [ info] Configuration:
[2024/02/16 19:16:02] [ info] flush time | 1.000000 seconds
[2024/02/16 19:16:02] [ info] grace | 5 seconds
[2024/02/16 19:16:02] [ info] daemon | 0
[2024/02/16 19:16:02] [ info] ___________
[2024/02/16 19:16:02] [ info] inputs:
[2024/02/16 19:16:02] [ info] dummy
[2024/02/16 19:16:02] [ info] ___________
[2024/02/16 19:16:02] [ info] filters:
[2024/02/16 19:16:02] [ info] ___________
[2024/02/16 19:16:02] [ info] outputs:
[2024/02/16 19:16:02] [ info] http.0
[2024/02/16 19:16:02] [ info] ___________
[2024/02/16 19:16:02] [ info] collectors:
[2024/02/16 19:16:02] [ info] [fluent bit] version=3.0.0, commit=bf767458b2, pid=999168
[2024/02/16 19:16:02] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2024/02/16 19:16:02] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/02/16 19:16:02] [ info] [cmetrics] version=0.6.6
[2024/02/16 19:16:02] [ info] [ctraces ] version=0.4.0
[2024/02/16 19:16:02] [ info] [input:dummy:dummy.0] initializing
[2024/02/16 19:16:02] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2024/02/16 19:16:02] [debug] [dummy:dummy.0] created event channels: read=21 write=22
[2024/02/16 19:16:02] [debug] [http:http.0] created event channels: read=23 write=24
[2024/02/16 19:16:02] [ info] [output:http:http.0] worker #0 started
[2024/02/16 19:16:02] [ info] [sp] stream processor started
[2024/02/16 19:16:02] [ info] [output:http:http.0] worker #1 started
[2024/02/16 19:16:03] [debug] [input chunk] update output instances with new chunk size diff=36, records=1, input=dummy.0
[2024/02/16 19:16:04] [debug] [task] created task=0x7ffff00222f0 id=0 OK
[2024/02/16 19:16:04] [debug] [output:http:http.0] task_id=0 assigned to thread #0
[2024/02/16 19:16:04] [debug] [input chunk] update output instances with new chunk size diff=36, records=1, input=dummy.0
[2024/02/16 19:16:04] [debug] [upstream] KA connection #49 to localhost:8889 is connected
[2024/02/16 19:16:04] [debug] [http_client] not using http_proxy for header
[2024/02/16 19:16:04] [ info] [output:http:http.0] localhost:8889, HTTP status=201
[2024/02/16 19:16:04] [debug] [upstream] KA connection #49 to localhost:8889 is now available
[2024/02/16 19:16:04] [debug] [out flush] cb_destroy coro_id=0
[2024/02/16 19:16:04] [debug] [task] destroy task=0x7ffff00222f0 (task_id=0)
[2024/02/16 19:16:05] [debug] [task] created task=0x7ffff0022640 id=0 OK
[2024/02/16 19:16:05] [debug] [output:http:http.0] task_id=0 assigned to thread #1
[2024/02/16 19:16:05] [debug] [input chunk] update output instances with new chunk size diff=36, records=1, input=dummy.0
[2024/02/16 19:16:05] [debug] [upstream] KA connection #50 to localhost:8889 is connected
[2024/02/16 19:16:05] [debug] [http_client] not using http_proxy for header
[2024/02/16 19:16:05] [ info] [output:http:http.0] localhost:8889, HTTP status=201
[2024/02/16 19:16:05] [debug] [upstream] KA connection #50 to localhost:8889 is now available
[2024/02/16 19:16:05] [debug] [out flush] cb_destroy coro_id=0
[2024/02/16 19:16:05] [debug] [task] destroy task=0x7ffff0022640 (task_id=0)
[2024/02/16 19:16:06] [debug] [task] created task=0x7ffff00228b0 id=0 OK
[2024/02/16 19:16:06] [debug] [output:http:http.0] task_id=0 assigned to thread #0
[2024/02/16 19:16:06] [debug] [input chunk] update output instances with new chunk size diff=36, records=1, input=dummy.0
[2024/02/16 19:16:06] [debug] [upstream] KA connection #49 to localhost:8889 has been assigned (recycled)
[2024/02/16 19:16:06] [debug] [http_client] not using http_proxy for header
[2024/02/16 19:16:06] [ info] [output:http:http.0] localhost:8889, HTTP status=201
[2024/02/16 19:16:06] [debug] [upstream] KA connection #49 to localhost:8889 is now available
[2024/02/16 19:16:06] [debug] [out flush] cb_destroy coro_id=1
[2024/02/16 19:16:06] [debug] [task] destroy task=0x7ffff00228b0 (task_id=0)
[2024/02/16 19:16:07] [debug] [task] created task=0x7ffff0022aa0 id=0 OK
[2024/02/16 19:16:07] [debug] [output:http:http.0] task_id=0 assigned to thread #1
[2024/02/16 19:16:07] [debug] [input chunk] update output instances with new chunk size diff=36, records=1, input=dummy.0
[2024/02/16 19:16:07] [debug] [upstream] KA connection #50 to localhost:8889 has been assigned (recycled)
[2024/02/16 19:16:07] [debug] [http_client] not using http_proxy for header
[2024/02/16 19:16:07] [ info] [output:http:http.0] localhost:8889, HTTP status=201
[2024/02/16 19:16:07] [debug] [upstream] KA connection #50 to localhost:8889 is now available
[2024/02/16 19:16:07] [debug] [out flush] cb_destroy coro_id=1
[2024/02/16 19:16:07] [debug] [task] destroy task=0x7ffff0022aa0 (task_id=0)
^C[2024/02/16 19:16:07] [engine] caught signal (SIGINT)
[2024/02/16 19:16:07] [debug] [task] created task=0x7ffff0022c50 id=0 OK
[2024/02/16 19:16:07] [debug] [output:http:http.0] task_id=0 assigned to thread #0
[2024/02/16 19:16:07] [ warn] [engine] service will shutdown in max 5 seconds
[2024/02/16 19:16:07] [ info] [input] pausing dummy.0
[2024/02/16 19:16:07] [debug] [upstream] KA connection #49 to localhost:8889 has been assigned (recycled)
[2024/02/16 19:16:07] [debug] [http_client] not using http_proxy for header
[2024/02/16 19:16:07] [ info] [output:http:http.0] localhost:8889, HTTP status=201
[2024/02/16 19:16:07] [debug] [upstream] KA connection #49 to localhost:8889 is now available
[2024/02/16 19:16:07] [debug] [out flush] cb_destroy coro_id=2
[2024/02/16 19:16:07] [debug] [task] destroy task=0x7ffff0022c50 (task_id=0)
[2024/02/16 19:16:08] [ info] [engine] service has stopped (0 pending tasks)
[2024/02/16 19:16:08] [ info] [input] pausing dummy.0
[2024/02/16 19:16:08] [ info] [output:http:http.0] thread worker #0 stopping...
[2024/02/16 19:16:08] [ info] [output:http:http.0] thread worker #0 stopped
[2024/02/16 19:16:08] [ info] [output:http:http.0] thread worker #1 stopping...
[2024/02/16 19:16:08] [ info] [output:http:http.0] thread worker #1 stopped
@pwhelan thanks. would you mind to open a PR with docs mentioning this breaking change in the new config ?
