fork-off-substrate
fork-off-substrate copied to clipboard
Polkadot state JSON not parsable
When I run npm start with a Polkadot archive node locally, it results in:
node:buffer:593
slice: (buf, start, end) => buf.utf8Slice(start, end),
^
Error: Cannot create a string longer than 0x1fffffe8 characters
at Object.slice (node:buffer:593:37)
at Buffer.toString (node:buffer:811:14)
at Object.readFileSync (node:fs:497:41)
at main (/home/vados/fork-off-substrate/index.js:123:31)
at processTicksAndRejections (node:internal/process/task_queues:96:5) {
code: 'ERR_STRING_TOO_LONG'
}
NodeJS only allows for strings to be 512 MiB, currently the state of Polkadot seems to be larger that that.
Example file.
There are some async parsing libs, but they are slow and I dont know how to write the state back to disk with them either...