Remove corrupted input chunk on flush

[bgaifullin]

The error CIO_CORRUPTED does not handle in flush method and fluent-bit stops processing logs if it meets corrupted file. The patch tries to handle this case by silently removing corrupted chunk.

Addresses #4278

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing Before we can approve your change; please submit the following in a comment:

• [ ] Example configuration file for the change
• [ ] Debug log output from testing the change

• [ ] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• [ ] Attached local packaging test output showing all targets (including any new ones) build.

Documentation

• [ ] Documentation required for this feature

Backporting

• [ ] Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[JeffLuoo]

I'm kinda confused as the CIO_CORRUPTED is checked when the function sb_segregate_chunks is executed: https://sourcegraph.com/github.com/fluent/fluent-bit@0336dc63d3358db77c1f35dd15dc87ad368d2edb/-/blob/plugins/in_storage_backlog/sb.c?L354. Do you mind sharing your chunk to reproduce the issue?

Edit: Went back and found that you're referring to a different error than I thought about could not segregate backlog chunk but https://github.com/fluent/fluent-bit/issues/4278#issuecomment-1209024100.

[bgaifullin]

Unfortunately I cannot share chunk. I can try to collect another kind of information, if you need.

[tarruda]

There was a recent chunkio refactor (#5753 for details). @bgaifullin can you check if this issue is still relevant?

[bgaifullin]

Ok, I will check. Thank you.

[bgaifullin]

Seems like version 1.9.9 works fine. Thank you

[github-actions[bot]]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.