[ISSUE#4824] Support consumequeue, indexfile multiple directories storage mode

[shengminw]

ISSUE#4824

In RIP-7 https://github.com/apache/rocketmq/wiki/RIP-7-Multiple-Directories-Storage-Support, CommitLog File has supported multi-directories storage mode. In this case, the commitlog can be stored on multiple disks, but consumequeue/index is still stored on one disk. However, the size of consumequeue/index can reach tens of Gbs, or even 100 Gbs, and the space used can't be easily ignored. The current commitLog multipath needs to be configured independently, as follows:

storePathRootDir=/rmqstore1
storePathCommitLog=/rmqstore1/commitlog:/rmqstore2/commitlog

Therefore, I optimized the multi-path mode, and the improvements are as follows: ● Add multi-path mode for consumequeue.
● Add multi-path mode for indexfile. ● Optimize the method to get the commitlog path. It is not necessary to specify storePathCommitLog separately, and generate the storePathCommitLog from storePathRootDir. ● Add a Timed thread to check path capacity. When select store path, it will try to avoid writing to a nearly full disk. ● Loading metadata in multi-path. By default, the config file is stored in the first path, and the config bak file is stored in the second path. For example: storePathRootDir = /rmqstore0:/rmqstore1:/rmqstore2, config dir = /rmqstore0/config, config bak dir = /rmqstore1/config. At startup, scan all paths and load the config file from the latest path of dataversion.（This ensures that when the path is changed, the latest version can be loaded） ● Loading checkpoint file and abort file in multi-path. The loading path of the checkpoint file and the abort file is the same as that of the config file.

Chinese Version / 中文版本

在RIP-7 https://github.com/apache/rocketmq/wiki/RIP-7-Multiple-Directories-Storage-Support 当中，CommitLog文件已经支持了多文件路径的存储模式。 在这个情况下，commitLog可以被存储至多块磁盘当中，但是consumequeue和index依然存储于单盘上。然而consumequeue 和 index的文件大小也可能达到数十G或者上百G的大小。这样的存储空间并不能被忽略。 当前的commitLog的多路径需要被额外配置，如下所示：

storePathRootDir=/rmqstore1
storePathCommitLog=/rmqstore1/commitlog:/rmqstore2/commitlog

因此，我对多路径模式做了一定的优化，改进主要如下： ● consumequeue的存储增加多路径模式。 ● indexfile的存储增加多路径模式。 ● 优化commitlog获取文件路径的方式。不必要单独设置storePathCommitLog，storePathCommitLog可以从storePathRootDir根目录下直接生成。 ● 增加一个定时线程检查路径的容量。当选择写入路径的时候，尽量避免写入至快满的磁盘中。 ● 多路径下的元数据加载。默认将config文件夹写入至第1个路径中，config文件夹的bak文件写入至第2个路径。例如：storePathRootDir = /rmqstore0:/rmqstore1:/rmqstore2, config dir = /rmqstore0/config, config bak dir = /rmqstore1/config。在启动的时候，扫描所有路径，选择dataversion最新的所在路径作为初始加载路径。（这样能确保路径被更改时，能够从最新的路径下加载） ● 加载checkpoint文件和abort文件。checkpoint文件和abort文件的加载路径与config路径保持一致。

[codecov-commenter]

Codecov Report

Merging #4825 (d6ae39a) into develop (de7cd0d) will increase coverage by 0.12%. The diff coverage is 60.38%.

@@              Coverage Diff              @@
##             develop    #4825      +/-   ##
=============================================
+ Coverage      43.26%   43.39%   +0.12%     
- Complexity      7741     7844     +103     
=============================================
  Files            995      995              
  Lines          69065    69435     +370     
  Branches        9143     9206      +63     
=============================================
+ Hits           29883    30133     +250     
- Misses         35441    35507      +66     
- Partials        3741     3795      +54     

Impacted Files	Coverage Δ
...cketmq/broker/offset/LmqConsumerOffsetManager.java	74.35% <0.00%> (-1.96%)	:arrow_down:
...apache/rocketmq/store/timer/TimerMessageStore.java	53.96% <0.00%> (-0.22%)	:arrow_down:
...apache/rocketmq/store/queue/BatchConsumeQueue.java	64.89% <7.40%> (-3.37%)	:arrow_down:
...cketmq/broker/offset/ConsumerOrderInfoManager.java	8.49% <33.33%> (+0.35%)	:arrow_up:
...c/main/java/org/apache/rocketmq/common/MixAll.java	43.38% <39.13%> (-1.06%)	:arrow_down:
...java/org/apache/rocketmq/common/ConfigManager.java	40.35% <40.90%> (-19.11%)	:arrow_down:
...e/rocketmq/store/config/StorePathConfigHelper.java	55.88% <52.17%> (-7.76%)	:arrow_down:
...org/apache/rocketmq/store/DefaultMessageStore.java	53.21% <54.43%> (-0.19%)	:arrow_down:
...a/org/apache/rocketmq/broker/BrokerController.java	48.00% <59.45%> (+4.15%)	:arrow_up:
...cketmq/broker/schedule/ScheduleMessageService.java	56.58% <66.66%> (+0.14%)	:arrow_up:
... and 62 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[coveralls]

Coverage decreased (-0.09%) to 49.139% when pulling f52caeee88b8644cc3eba9f2062e210c5980fd7c on shengminw:multi-route into 8dd07c84697d3709f1f4bef5d78089b7d171b6cd on apache:develop.

[coveralls]

Coverage increased (+0.009%) to 49.234% when pulling 8d955033188a8f9950563f4ddcf26d0b7c2b669c on shengminw:multi-route into 8dd07c84697d3709f1f4bef5d78089b7d171b6cd on apache:develop.

[zongtanghu]

Can you fix the codes' conflicts firstly! @shengminw

[shengminw]

Can you fix the codes' conflicts firstly! @shengminw

Sorry, this conflict was newly created when the develop branch was merged yesterday. I have fixed this conflict.

[aaron-ai]

We intend to add CI pipelines for more platforms recently besides Linux, there are too many unit tests could not pass in other platforms especially Windows.

It would be awesome if you could concern more about cross-platform issues.

By the way, if you don't have windows device for testing now, Github Action is a good choice. You can refer to https://github.com/aaron-ai/rocketmq/pull/1 for support.

[shengminw]

@aaron-ai thanks for your suggestion, I will check the compatible with windows platform.

[github-actions[bot]]

This PR is stale because it has been open for 365 days with no activity. It will be closed in 3 days if no further activity occurs. If you wish not to mark it as stale, please leave a comment in this PR.

[github-actions[bot]]

This PR was closed because it has been inactive for 3 days since being marked as stale.