systemds
systemds copied to clipboard
[SYSTEMDS-???] Frame text and csv write without meta
This commit removes the #meta part of the writing and reading csv files for frames. This change removes the branching of reading every line of csv files that improve the I/O time. It was not an issue before, because the IO was significantly slower, but now with the disk speed on high Raid 0 or fast nvme ssds it matters.
@mboehm7
The CSV readers contain code to manage a '#META' tag this slows down the reading, and I wanted to remove it in this PR. The spark instructions also handle these in some cases and i removed those as well. I think i have misunderstood the use of these, can you maybe elaborate on the use of the meta tag?