fluent-bit icon indicating copy to clipboard operation
fluent-bit copied to clipboard

filter_modify: Add data type condition checker to `Modify` filter.

Open zihengCat opened this issue 5 years ago • 3 comments

Summary

It would be good if Fluent Bit supports strong data type condition check when applied in structured log stream. I implemented a data type condition checker for Modify filter.

Check data type using lua filter isn't clear enough for end-user and costs more performance overhead.

Usage

Modify filter supports two new conditions. See modify_filter_example.conf below.

Condition Parameter 1 Parameter 2 Description
Key_Value_Type_Matches STRING:KEY STRING:VALUE Is true if KEY exists and its value data type is VALUE.
Key_Value_Type_Does_Not_Match STRING:KEY STRING:VALUE Is true if KEY exists and its value data type is not VALUE.

Parameter 2 data type string literal could be:

  • null
  • bool
  • number
  • int
  • float
  • str
  • array
  • map
[INPUT]
    Name dummy
    Tag  test
    dummy {"nullType": null, "boolType": true, "intType": 123, "floatType": 3.14159, "strType": "awesome!", "arrayType": [1, 2, 3], "mapType": {"k": "v"}}

[OUTPUT]
    Name  stdout
    Match *

[FILTER]
    Name  modify
    Match *

    Condition Key_Value_Type_Matches             nullType          null
    Condition Key_Value_Type_Matches             boolType          bool
    Condition Key_Value_Type_Matches             intType           number
    Condition Key_Value_Type_Matches             intType           int
    Condition Key_Value_Type_Matches             floatType         float
    Condition Key_Value_Type_Matches             strType           str
    Condition Key_Value_Type_Matches             arrayType         array
    Condition Key_Value_Type_Matches             mapType           map

    Condition Key_Value_Type_Does_Not_Match      mapType           array

    Rename nullType nullTypeMatches
    Rename mapType mapTypeDoesNotMatch

modify_filter_example.conf

Unit Tests

Here comes unit testing results.

13/31 Test #13: flb-rt-filter_kubernetes .........   Passed  193.00 sec
      Start 14: flb-rt-filter_parser
14/31 Test #14: flb-rt-filter_parser .............   Passed   20.00 sec
      Start 15: flb-rt-filter_modify
15/31 Test #15: flb-rt-filter_modify .............   Passed   55.00 sec
      Start 16: flb-rt-core_engine
16/31 Test #16: flb-rt-core_engine ...............   Passed   20.00 sec

Addresses

#2530


Enter [N/A] in the box, if an item is not applicable to your change.

Testing Before we can approve your change; please submit the following in a comment:

  • [x] Example configuration file for the change
  • [x] Debug log output from testing the change
  • [x] Attached Valgrind output that shows no leaks or memory corruption was found

Documentation

  • [x] Documentation required for this feature

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

zihengCat avatar Sep 11 '20 10:09 zihengCat

thanks for this PR.

  • is it nil a configuration value ?, if so, can it be null ? , that's more representative for end-users
  • does number covers int and float ?

edsiper avatar Sep 28 '20 22:09 edsiper

@edsiper Fine.

  • Change nil to null.
  • number covers both int and float, corresponding to JSON number data type.

zihengCat avatar Sep 29 '20 03:09 zihengCat

Is there any chance this might eventually be merged?

philomory avatar Jan 08 '23 21:01 philomory