apache
Issue #4033: Added a flush function to run flush without commit
Descriptions of the changes in this PR:
Added a sync method for AppendOnlyWriter
Motivation
We are testing using Distributedlog as HBase WAL storage. For HBase, there are three different durability levels, async wal, sync wal, fsync wal. The features of these three kinds of wal are:
async wal: the main thread only invokes append (write method of appendonlywriter) and returns.
sync wal: the main thread returns after confirming that the data is already arrived at the storage service through network.
fsync wal: the main thread returns after confirming that the data is already persistently stored.
But currently, the AppendOnlyWriter only provides a method force, which provides a stronger durability than fsync wal (the client will wait until the data is visible to other clients).
Stronger durability means poorer performance. The purpose to provide three different WALs is to make it possible for clients to make a trade off between durability and performance.
Changes
Added a new method flush of AppendOnlyStreamWriter, which just invoke the underlying flush#BKLogSegmentWriter and to wait the future completed or not by a boolean parameter.
Master Issue: #4033
Can you please provide more details for "(Explain: why you're making that change, what is the problem you're trying to solve)".
After the change, the AppendOnlyWriter will be a little bit more inflexible.
The new methods I added in this issue is just for the implementations of the above different kinds of durability. For async wal, we can just inovke write method to write data into buffer and return. For sync wal, we can directly invoke flush and do not wait the future be completed. Finally, for fsync wal, the main thread just wait the future be completed.
I have tested the method in our test environment. It will boost the performance of the whole WAL system.
BTW, I am not so proficient at BK and DL. If the change is not so good, do we have other ways to support the above kinds of WALs?
Added a UT in TestAppendOnlyStreamWriter. The flush works as expected locally.
