buffering_storage - No application log when storage.total_limit_size is reached for storage.type filesystem

[sirwio]

Bug Report

Describe the bug In previoius version an error log was created when the storage.total_limit_size condition was reached for output plugins with storage.type as filesystem. In e.g. fluent-bit 1.6 the log was generated as:

/*
             * The worst scenerio is that we cannot find a space by dropping some
             * old chunks for the incoming chunk. We need to adjust the routes_mask
             * of the incoming chunk to not flush to that output instance.
             */
            flb_error("[input chunk] no enough space in filesystem to buffer "
                      "chunk %s in plugin %s", flb_input_chunk_get_name(ic), o_ins->name);

and since 1.9 the wording was somewhat re-phrased:

            flb_error("[input chunk] chunk %s would exceed total limit size in plugin %s",
                      flb_input_chunk_get_name(ic), o_ins->name);

In commit 00f94c8 - input_chunk: changed the chunk release strategy this part of the code was refactored and it appears as if there is now no log created when this condition happens.

A fluent-bit application log is not created if the storage.total_limit_size is reached when the storage.type is filesystem.

To Reproduce fluent-bit.conf

[SERVICE]
    Flush 1
    Log_Level info
    Storage.path buffer
    Storage.sync full
    ## Configs below is just to avoid clutter due to retries.
    scheduler.base 240
    scheduler.cap   360

[INPUT]
    Name dummy
    Storage.type filesystem
    Log_Level info
    Rate 1000
    Samples 10000
    Dummy {“message”:” Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec elit neque, suscipit quis iaculis ac, commodo sed tellus. Aliquam arcu eros, ultrices et nisl quis, dignissim mollis quam. Nulla auctor non libero vitae euismod. Nunc sit amet nisi magna. Phasellus eget congue tellus. Praesent molestie condimentum velit, in efficitur lacus pharetra eu. Fusce in lacus vel mi varius molestie ac id turpis. Fusce blandit odio massa. Duis arcu purus, pharetra in rhoncus finibus, faucibus sit amet sapien. Praesent nunc est, scelerisque nec tincidunt et, egestas quis justo. Fusce iaculis urna non leo porta, in auctor tortor consectetur.”}
    
[OUTPUT]
    Name tcp
    Match *
    Retry_Limit False
    Log_Level warning
    storage.total_limit_size 2M

Run fluent-bit with:

./fluent-bit -c fluent-bit.conf

Expected behavior Expect an application log providing information that the configured filesystem space is not sufficient to backbuffer all logs. The application is with the above dummy message expected after about the fourth send attempt to upstream.

Your Environment fluent-bit 2.2 running on Ubuntu 22.04.3 with configuration file listed above.

Additional context This functionallity is essential to be able to trigger an alert if this condition occurs. It is both an indication that some logs might have been lost in the audit pipeline as well as a hint that the configuration of fluent-bit can be tweaked to handle the load. Preferably this audit shall not be error but rather info.

[sirwio]

This broke in the fixing of issue #7729 @leonardo-albertovich Can you take a look and comment if I have misunderstood something.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.