jayvee icon indicating copy to clipboard operation
jayvee copied to clipboard

[BUG] `LocalFileExtractor` emits wrong error messag if the input file is too big

Open jrentlez opened this issue 1 year ago • 0 comments

Steps to reproduce

  1. Execute this model (big_file.csv is larger than 2GB):
pipeline Pipeline {

	Extractor
		-> ToTextFile
		-> ToCSV
		-> ToTable
		-> Loader;


	block Extractor oftype LocalFileExtractor {
		filePath: "./big_file.csv";
	}

	block ToTextFile oftype TextFileInterpreter { }

	block ToCSV oftype CSVInterpreter {
		enclosing: '"';
	}

	block ToTable oftype TableInterpreter {
		header: true;
		columns: [
			"C1" oftype integer,
			"C2" oftype text,
			"C3" oftype boolean,
		];
	}

	block Loader oftype SQLiteLoader {
		table: "Data";
		file: "./Data.sqlite";
	}
}

Description

  • Expected: The interpreter should fail with the message that the file is too big
  • Actual: The interpreter fails with a "file not found" message:
[Pipeline] Overview:
        Blocks (5 blocks with 1 pipes):
         -> Extractor (LocalFileExtractor)
                 -> ToTextFile (TextFileInterpreter)
                         -> ToCSV (CSVInterpreter)
                                 -> ToTable (TableInterpreter)
                                         -> Loader (SQLiteLoader)

        error: File './big_file.csv' not found.
        $In /home/jonas/Code/uni/hiwi/jayvee/pipeline.jv:14:2
        14 |     block Extractor oftype LocalFileExtractor {
           |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        15 |         filePath: "./big_file.csv";
           | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        16 |     }
           | ^^^^^

jrentlez avatar Jul 30 '24 14:07 jrentlez