dledger icon indicating copy to clipboard operation
dledger copied to clipboard

DLedger should support TransientStorePool

Open humkum opened this issue 3 years ago • 2 comments

FEATURE REQUEST

We found that there would be frequent "pagecache busy" with the growth of tps. And RocketMQ create a transientStorePool by using DirectByteBuffer to insulate write and read in Master-Slave mode. 
So I guess it would work better if dledger support transientStorePool to reduce jitter caused by pagecache busy.
There are some of my opinions:
1. Leader node uses directByteBuffer to write and follower nodes still use mappedByteBuffer to aviod data loss in some cases.
2. Initialize transientStorePool when the broker role changes to "leader", and release these memory space while the broker role change to "follower" from "leader".
3. If the broker role changed to "leader" from "follower", it could still use mappedBytebuffer to write the uncompleted file, and use directByteBuffer to write next file.
I'm looking forward to your suggestion. 

humkum avatar Jul 01 '22 06:07 humkum

我理解在leader节点不会直接将数据落盘,接受到来自客户端的请求之后写入directByteBuffer,然后异步的向follower同步日志,以及异步的将directByteBuffer中的数据写入到mmapfile。 我的理解对吗?

akkw avatar Oct 17 '22 12:10 akkw

我理解在leader节点不会直接将数据落盘,接受到来自客户端的请求之后写入directByteBuffer,然后异步的向follower同步日志,以及异步的将directByteBuffer中的数据写入到mmapfile。 我的理解对吗?

需要成功向 1/2 以上节点同步完成后才可以返回成功

humkum avatar Jan 12 '24 09:01 humkum