mdawsonuk
Program consumes all memory on DB of size comparable to RAM and blows up on Bitcoin blockchain transaction index
LevelDB Dumper version
e750a27ff58443ecc410b5c16abbdc539d617387
To Reproduce
To obtain the test data, you have to install Bitcoin and fetch a full node with txindex, which can take 24 hours, on Ubuntu 23.10:
snap install bitcoin-core
bitcoin-core.daemon -txindex=1
Also the exact data layout is not deterministic, but hopefully that won't matter :crossed_fingers:
After that the leveldb is located at /snap/bitcoin-core/common/.bitcoin/indexes/txindex and running:
LevelDBDumper -q -t json -d ~/snap/bitcoin-core/common/.bitcoin/indexes/txindex -o . -f btc.json
consumes more and more memory until after about 30 minutes it takes up everything and the kernel kills it.
The size of all files in the leveldb database directory is ~ 46.2 GiB, so it does not feel very reasonable that it should blow up the 64 GB RAM of my Lenovo ThinkPad P14s. Maybe there is some memory that could be freed but isn't?
The question is, is it possible to read the input files bit by bit, and output as we go, or does it absolutely require reading everything into memory.
