fork-off-substrate icon indicating copy to clipboard operation
fork-off-substrate copied to clipboard

Polkadot state JSON not parsable

Open ggwpez opened this issue 3 years ago • 0 comments

When I run npm start with a Polkadot archive node locally, it results in:

node:buffer:593
    slice: (buf, start, end) => buf.utf8Slice(start, end),
                                    ^
Error: Cannot create a string longer than 0x1fffffe8 characters
    at Object.slice (node:buffer:593:37)
    at Buffer.toString (node:buffer:811:14)
    at Object.readFileSync (node:fs:497:41)
    at main (/home/vados/fork-off-substrate/index.js:123:31)
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  code: 'ERR_STRING_TOO_LONG'
}

NodeJS only allows for strings to be 512 MiB, currently the state of Polkadot seems to be larger that that.
Example file.

There are some async parsing libs, but they are slow and I dont know how to write the state back to disk with them either...

ggwpez avatar Mar 18 '22 11:03 ggwpez