openmessaging
MmapFileList.prepend方法中这段代码可能抛异常，没影响么？
// allocate不一定>=8个字节吧？可能抛异常，没影响么 ByteBuffer byteBuffer = ByteBuffer.allocate(mappedFile.getFileSize() - mappedFile.getWrotePosition()); byteBuffer.putInt(BLANK_MAGIC_CODE); byteBuffer.putInt(mappedFile.getFileSize() - mappedFile.getWrotePosition());
if (blank < MIN_BLANK_LEN) {
logger.error("Blank {} should ge {}", blank, MIN_BLANK_LEN);
return -1;
} else {
// allocate不一定超过8个字节把，应该加判断。
ByteBuffer byteBuffer = ByteBuffer.allocate(mappedFile.getFileSize() - mappedFile.getWrotePosition());
byteBuffer.putInt(BLANK_MAGIC_CODE);
byteBuffer.putInt(mappedFile.getFileSize() - mappedFile.getWrotePosition());
if (mappedFile.appendMessage(byteBuffer.array())) {
//need to set the wrote position
mappedFile.setWrotePosition(mappedFile.getFileSize());
} else {
logger.error("Append blank error for {}", storePath);
return -1;
}
Hi @bobbyz007 the MmapFileList#prepend method paramter len, least value is 48. append like this
java -jar target\DLedger.jar append -d ""
so if we want append the data to DLedger, the mappedFileSizeForEntryData value least value is 48. when we first to append message. default value is mapped 1G
(mappedFile.getFileSize() - mappedFile.getWrotePosition() == 48 ) > 8
then the next time execute will get newleast file to append.
So that's not going to happen.
@mxsm If you append empty string firstly , the outer if condition could not be satisfied(except len + blank > 1G).
if (len + blank > mappedFile.getFileSize() - mappedFile.getWrotePosition()) {
My question is When invoking append(...) method multiple times, how could you make sure:
(mappedFile.getFileSize() - mappedFile.getWrotePosition() ) > 8 ?
Put it another way: Every entry's size is 48 bytes +body. After appending multple entries, is it impossible there is just left less than 8 bytes in 1G mmap file?
@mxsm If you append empty string firstly , the outer if condition could not be satisfied(except len + blank > 1G).
if (len + blank > mappedFile.getFileSize() - mappedFile.getWrotePosition()) {My question is When invoking append(...) method multiple times, how could you make sure:
(mappedFile.getFileSize() - mappedFile.getWrotePosition() ) > 8 ?Put it another way: Every entry's size is 48 bytes +body. After appending multple entries, is it impossible there is just left less than 8 bytes in 1G mmap file?
If mappedFile.getFileSize() - mappedFile.getWrotePosition() is greater than or equal to len+blank, len can still be stored, and the remaining length is at least 8bytes, the next time to execute this method will enter the if condition. and mappedFile.getFileSize() - mappedFile.getWrotePosition() >= 8 is true. The above illustration is just an example.You can focus on the condition that this critical value is equal to
@mxsm Thx got it. Another point： Would it be better to allocate 'byteBuffer' variable just 8 bytes as BLANK_MAGIC_CODE is only a mark indicating no entry behind it:
ByteBuffer byteBuffer = ByteBuffer.allocate(8);
Allocating a temp buffer varialbe 'byteBuffer' of size 'mappedFile.getFileSize() - mappedFile.getWrotePosition()' rather than 8 bytes sounds unnecessary especially if it's a huge number , that may cause memory issue. because wrotePosition has been correccted if mappedFile.appendMessage(...) is true.
ByteBuffer byteBuffer = ByteBuffer.allocate(mappedFile.getFileSize() - mappedFile.getWrotePosition());
byteBuffer.putInt(BLANK_MAGIC_CODE);
byteBuffer.putInt(mappedFile.getFileSize() - mappedFile.getWrotePosition());
if (mappedFile.appendMessage(byteBuffer.array())) {
//need to set the wrote position
mappedFile.setWrotePosition(mappedFile.getFileSize());
} else {
@mxsm Thx got it. Another point： Would it be better to allocate 'byteBuffer' variable just 8 bytes as BLANK_MAGIC_CODE is only a mark indicating no entry behind it:
ByteBuffer byteBuffer = ByteBuffer.allocate(8);Allocating a temp buffer varialbe 'byteBuffer' of size 'mappedFile.getFileSize() - mappedFile.getWrotePosition()' rather than 8 bytes sounds unnecessary especially if it's a huge number , that may cause memory issue. because wrotePosition has been correccted if mappedFile.appendMessage(...) is true.
ByteBuffer byteBuffer = ByteBuffer.allocate(mappedFile.getFileSize() - mappedFile.getWrotePosition()); byteBuffer.putInt(BLANK_MAGIC_CODE); byteBuffer.putInt(mappedFile.getFileSize() - mappedFile.getWrotePosition()); if (mappedFile.appendMessage(byteBuffer.array())) { //need to set the wrote position mappedFile.setWrotePosition(mappedFile.getFileSize()); } else {
I think is good idea for
ByteBuffer byteBuffer = ByteBuffer.allocate(8);
I think the else condition can keep. Because some other unanticipated reasons cause put data into file failure.
今天仔细想了一下，感觉这段代码好像没有什么问题，我们一起来讨论一下。我的思路是这样的。
我们先把条件稍微移动一下项，可以写成blank > fileSize - (wrotePosition + len)，这个表达式的意思可以理解为，在某一次写入时，文件长度 - （已写入的字节长度 + 本次写入的字节长度 ）可以表示为本次写入后文件的剩余长度。接下来我举例。
假设有一个长度为20Byte的文件 第一次写入2Byte内容 计算条件 20 - (0 + 2) = 12 Byte > MIN_BLANK_LEN ,写入后wrotePosition 值为 2，剩余长度为 18,条件不成立 接下来再次写入10字节内容 计算条件 20 - （2 + 10） = 8 Byte = MIN_BLANK_LEN, 写入后wrotePosition为12, 剩余长度8, 条件不成立。 再次写入1字节 计算条件 20 - （12 + 1） = 7 Byte < MIN_BLANK_LEN, 条件成立，此时会将长度为MIN_BLANK_LEN的文件结尾标志写入文件 这个假设可以看出文件并不会写入益出。
再来一个假设
假设有一个长度为20Byte的文件 第一次写入2Byte内容 计算条件 20 - (0 + 2) = 12 Byte > MIN_BLANK_LEN ,写入后wrotePosition 值为 2，剩余长度为 18,条件不成立 接下来再次写入11字节内容 计算条件 20 - （2 + 11） = 7 Byte < MIN_BLANK_LEN,条件成立，会直接写入一个文件结尾标志，此时文件剩余长度为18，同样不会造成文件写入溢出。
通过这两个假设以及计算条件移项blank > fileSize - (wrotePosition + len) 我觉得这个方法并没有潜在的问题。我们可以继续讨论一下。
计算条件移项后可以理解为在文件大小确定且不变的情况下，本次写入之后的剩余长度一定是大于8字节的，如果某一次写入之后文件长度小于8 就会将文件结尾标志写入文件中，然后去尝试创建新的文件。在新的文件中将bytes写入，所以文件不会发生溢出的情况
