globocom
Request support for Byte size-based flushing
Feature Request
Description:
I would like to request support for byte size-based flushing in the buffer package. Currently, the package supports flushing based on a timeout or when the buffer is full based on the number of items. However, there are use cases where it's essential to flush the buffer when it reaches a certain byte size limit.
Use Case:
Imagine a scenario where we're buffering log entries to be written to the buffer and sent to a remote server. It's important to flush the buffer when it reaches a certain size to avoid memory overflows or to ensure that log entries are sent in manageable chunks.
Proposed Solution:
I suggest adding a new configuration option to the Options struct to specify a byte size limit. When this limit is reached, the buffer should automatically trigger a flush. Users should be able to set this byte size limit when creating a new Buffer instance.
Example:
// Create a new buffer with byte size-based flushing.
buffer := buffer.New(buffer.WithByteSizeLimit(1024)) // Flush when the buffer reaches 1 KB.
// Push data into the buffer.
buffer.Push([]byte("Some log entry"))
// The buffer will automatically flush when it reaches 1 KB in size.
I'm not sure if that is doable since the buffer holds an array of items that can be anything (currently interface{}). I don't now how to get the overall size in bytes without resorting to unsafe or reflect. Also, even if it was possible to safely get that number it wouldn't be practicable to flush the buffer partially (e.g. at exactly 1KB).
Did you think about a working implementation?
It can be done by keeping a counter of size before the item is added to the buffer. If the (new item size + existing size) exceeds the limit, flush existing items, add a new item to the buffer and reset the existing size. Doesn't have to be precisely (1KB) its a max limit.
What do you have in mind to get the size in bytes of each item in order to keep a counter?