python-rocksdb icon indicating copy to clipboard operation
python-rocksdb copied to clipboard

How to optimize the speed when I import a large data into pyrocksdb?

Open xiaolinpan opened this issue 5 years ago • 2 comments

When I import a large data into rocksdb, the speed is very slow. How can I optimize the speed?thank you!

xiaolinpan avatar Nov 25 '20 08:11 xiaolinpan

Replace the hard disk with an SSD, and provide your rocksdb status log, search for some optimization solutions about delayed write and write blocking,If you are importing existing data to rocksdb, you can try to create an sst file to import

Jamiejoin avatar Nov 28 '20 14:11 Jamiejoin

If you are doing an initial bulk load, insert the data in order, sorted by key. Sorting the data up front makes for much less work when loading.

You can also create the SST files yourself- https://github.com/facebook/rocksdb/wiki/Creating-and-Ingesting-SST-files

ckingdev avatar Dec 19 '20 00:12 ckingdev