BUG: crashes VS Code

[morganics]

When right clicking and opening Sand Dance via VS Code (even on a relatively small dataset - 150mb), I just get this.

System information;

Version: 1.39.2 (system setup) Commit: 6ab598523be7a800d7f3eb4d92d7ab9a66069390 Date: 2019-10-15T15:35:18.241Z Electron: 4.2.10 Chrome: 69.0.3497.128 Node.js: 10.11.0 V8: 6.9.427.31-electron.0 OS: Windows_NT x64 10.0.17134

[danmarshall]

Thank you for posting. I'm not sure if 150mb is small or not. Can you characterize this data set for me? I'd like to know the file type, number of rows, number of columns, and types of data in the columns.

[morganics]

Sure, 29 columns, CSV file type (utf8 encoded), around 216mb when loaded in to Pandas (though file size is 152mb). 12 string cols, 6 floats and 9 ints. Some fields have missing values. It loads quickly in VS Code (just loading the file itself), but crashes when viewing in the Sanddance plugin. There are 1.04m rows. As far as I can tell, memory use doesn't increase significantly when loading. I'll reduce the filesize further as maybe that's still the issue. Any way to debug?

[morganics]

Just to add - have tried with 100k rows (15mb file) (it works!) but 500k says 'loading' (76mb) which ends in a blank screen (no crash message). As it's only possible to graph a limited number of dimensions at once, it seems incorrect to try and preload the whole file. Also, it would be good perhaps to have a more graceful error message if the file is too large.

[danmarshall]

See #141 for more scenarios which cause a similar crash.

We will need a strategy for solving this My current thought is:

1. Columns: Only load a small number of columns into the initial working data. Other column will be shown in the pickers but not loaded until picked. Unused columns can be omitted.
2. Rows: above 500k, load progressively,(with a load more... button ?) and gracefully catch crashes if possible. With the unused columns unloaded, this may also help alleviate this issue.

[stej]

Using SandDance in VS Code, version v1.2.12.

My CSV file has 260k rows. 13 string cols, 2 int cols. The file itself has 63MB. After I removed 3 string cols, it works until I select "Stacks" with both string axes. Then it crashes sometimes.

JS error:

webviewElement.ts:341 embedded page crashed
(anonymous) @ webviewElement.ts:341
dispatchEvent @ web-view-impl.ts:124
dispatchEvent @ guest-view-internal.ts:59
(anonymous) @ guest-view-internal.ts:77
emit @ events.js:203
onMessage @ init.ts:59
bootstrap-window.js:212 [uncaught exception]: TypeError: Cannot read property 'forEach' of undefined
onUnexpectedError @ bootstrap-window.js:212
(anonymous) @ bootstrap-window.js:51
emit @ events.js:203
./lib/renderer/init.ts.window.onerror @ init.ts:188
n.globals.onerror @ errorTelemetry.ts:20
bootstrap-window.js:215 TypeError: Cannot read property 'forEach' of undefined
    at dispatchEvent (guest-view-internal.ts:55)
    at EventEmitter.<anonymous> (guest-view-internal.ts:77)
    at EventEmitter.emit (events.js:203)
    at Object.onMessage (init.ts:59)
onUnexpectedError @ bootstrap-window.js:215
(anonymous) @ bootstrap-window.js:51
emit @ events.js:203
./lib/renderer/init.ts.window.onerror @ init.ts:188
n.globals.onerror @ errorTelemetry.ts:20
guest-view-internal.ts:55 Uncaught TypeError: Cannot read property 'forEach' of undefined
    at dispatchEvent (guest-view-internal.ts:55)
    at EventEmitter.<anonymous> (guest-view-internal.ts:77)
    at EventEmitter.emit (events.js:203)
    at Object.onMessage (init.ts:59)
dispatchEvent @ guest-view-internal.ts:55
(anonymous) @ guest-view-internal.ts:77
emit @ events.js:203
onMessage @ init.ts:59

System info

SandDanceExplorer version: 1.6.5
SandDanceReact version: 1.1.7
SandDance version: 1.6.5
WebGL enabled: Yes
WebGL2 enabled: Yes

[stej]

What's the reason for this crash? Also it would be quite useful not to limit lines to ~500k (if that's still true), as I have a lot of unused memory and would be great also analyze bigger files.