rocks-queue-java icon indicating copy to clipboard operation
rocks-queue-java copied to clipboard

The queue does not work, if I add more than 255 elements

Open Hixon10 opened this issue 4 years ago • 2 comments

Hello,

To reproduce this case, you need to perform the following steps:

  1. create queue
  2. enqueue 500 elements
  3. close the application (it helps to reset your in-memory pointer com.me.rocks.queue.RocksQueue#head
  4. start the application (in this moment, your queue load in-memory pointer from the disk)
  5. consume 500 elements
  6. you will notice, that you receive the wrong elements

The reason, that you use BigEndian order for bytes, but uint64add merge operator assumes to have LITTLE_ENDIAN - https://github.com/facebook/rocksdb/pull/7243 and https://github.com/facebook/rocksdb/blob/main/utilities/merge_operators/uint64add.cc

I don't know how to fix it. Java Rocksdb API does not allow to create a new merge operator. Also, we cannot use LITTLE_ENDIAN, because it breaks the queue order.

I saw golang implementation of the queue. They created the own merge operator, but we cannot do it. https://github.com/mijia/rocksq/blob/master/merge.go

Hixon10 avatar Oct 11 '21 15:10 Hixon10

@Hixon10 Thanks for your feedback. I had not noticed this issue before. And yes it might depend rocks java to add the merge operator API in the future.

artiship avatar Oct 14 '21 12:10 artiship

@artiship I asked about it - https://groups.google.com/g/rocksdb/c/iGLOaBZqBkY

Hixon10 avatar Oct 14 '21 12:10 Hixon10