systemds icon indicating copy to clipboard operation
systemds copied to clipboard

[SYSTEMDS-???] Frame text and csv write without meta

Open Baunsgaard opened this issue 2 years ago • 1 comments

This commit removes the #meta part of the writing and reading csv files for frames. This change removes the branching of reading every line of csv files that improve the I/O time. It was not an issue before, because the IO was significantly slower, but now with the disk speed on high Raid 0 or fast nvme ssds it matters.

Baunsgaard avatar Jan 05 '24 17:01 Baunsgaard

@mboehm7

The CSV readers contain code to manage a '#META' tag this slows down the reading, and I wanted to remove it in this PR. The spark instructions also handle these in some cases and i removed those as well. I think i have misunderstood the use of these, can you maybe elaborate on the use of the meta tag?

Baunsgaard avatar Jan 08 '24 12:01 Baunsgaard