How to stream ArrayBuffer

[FirstVertex]

Hi Keyang! Thanks so much for such great lib as csvtojson!

When we are picking a file in browser with <input> we can use readAsArrayBuffer() to get the contents efficiently into memory. But then, I didn't find any example how to use csvtojson's readFromStream() with ArrayBuffer.

I had to write a little class who allows ArrayBuffer to be treated as a stream. I wanted to share it with you and your users, in case someone else could benefit.

ArrayBufferStream class

import * as Stream from 'stream';

class ArrayBufferStream extends Stream.Readable {
    constructor(private _arrayBuffer: ArrayBuffer, opts?: Stream.ReadableOptions) {
        super(opts);
    }

    private _chunk_size = 1024 * 16;
    private _pointer = 0;

    _read(size: number) {
        size = size || this._chunk_size;

        if (this._pointer + size >= this._arrayBuffer.byteLength) {
            size = this._arrayBuffer.byteLength - this._pointer;
        }

        const from = this._pointer;
        this._pointer += size;

        // we use setTimeout here to yield to the UI thread, prevent it from locking up
        setTimeout(() => {
            if (size) {
                this.push(Buffer.from(this._arrayBuffer, from, size));
            } else {
                this.push(null);
            }
        }, 0);
    }
}

Usage

csv().fromStream(new ArrayBufferStream(arrayBuffer))