Feature Request - batch logging in FluentLogger->post() function

[aacanakin]

Hello, I want to be able to write batch logs (50 items) atomically. Currently, I'm posting logs in foreach loop in batch requests. However, if td agent stops unexpectedly, I need to write the complete request(50 items) to csv. Is it possible in the current code base ? Does fluentd interface support batch logging ?

Thanks Aras

[chobie]

AFAIK, fluentd doesn't have such interface. in_tail might solve your concerns. I recommend to use in_tail and FileLogger if you can switch fluentd configuration.

http://docs.fluentd.org/articles/in_tail

Basically, PHP is poor about back ground task like sending metrics data. It's PHP architecture design. But it can improve with C extension. I'll plan to add async logger this year.

[disq]

Yes, this does improve performance. curl is always faster than file_get_contents.

[cosmo0920]

FYI: For now, you can use Embulk for batch operations. Embulk is designed for batch operations.

[repeatedly]

This issue seems to request micro-batch forward, not Embulk like huge batching.

in_forward accepts multiple events in one request so supporting this is good for some users. http://docs.fluentd.org/articles/in_forward#protocol