jvalue
[BUG] JavaScript heap out of memory
Steps to reproduce
Use following pipeline:
pipeline TestPipeline {
Extractor -> TextFileInterpreter->RangeSelector->CSVInterpreter->ColumnDeleter->TableInterpreter->Loader;
block Extractor oftype HttpExtractor {
url: "https://derwebmaster.eu/wp-content/uploads/2024/02/star0000-1.csv";}
block TextFileInterpreter oftype TextFileInterpreter {encoding: "utf8";}
block RangeSelector oftype TextRangeSelector {lineFrom: 1;}
block CSVInterpreter oftype CSVInterpreter {enclosing: '';delimiter: ",";}
block ColumnDeleter oftype ColumnDeleter {delete: [];}
block TableInterpreter oftype TableInterpreter {
header: false;
columns: ["col1" oftype integer,"col2" oftype integer,"col3" oftype integer,"col4" oftype integer,"col5" oftype integer,"col6" oftype decimal,"col7" oftype integer,"col8" oftype integer,"col9" oftype decimal,"col10" oftype integer,"col11" oftype decimal,"col12" oftype integer];}
block Loader oftype SQLiteLoader {table: "tDbNEGvkAEdn"; file: "test.sqlite";}
}
Original source of the csv: https://sdm.lbl.gov/fastbit/data/samples.html
Description
- Expected: Pipeline should download the file and store the content in a database
- Actual: FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
Debug: [TestPipeline] Overview: Blocks (7 blocks with 1 pipes): -> Extractor (HttpExtractor) -> TextFileInterpreter (TextFileInterpreter) -> RangeSelector (TextRangeSelector) -> CSVInterpreter (CSVInterpreter) -> ColumnDeleter (ColumnDeleter) -> TableInterpreter (TableInterpreter) -> Loader (SQLiteLoader)
[Extractor] Fetching raw data from https://derwebmaster.eu/wp-content/uploads/2024/02/star0000-1.csv
Console output: <--- Last few GCs --->
[99110:0x65186f0] 716409 ms: Mark-Compact (reduce) 2047.2 (2078.4) -> 2047.2 (2079.1) MB, 29.36 / 0.00 ms (+ 602.9 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 999 ms) (average mu = 0.992, current mu = 0.97[99110:0x65186f0] 718079 ms: Mark-Compact (reduce) 2048.2 (2079.1) -> 2048.2 (2080.1) MB, 1666.92 / 0.00 ms (average mu = 0.689, current mu = 0.002) allocation failure; scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory ----- Native stack trace -----
1: 0xcc08f6 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [node]
2: 0x1054130 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
3: 0x1054417 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
4: 0x1273655 [node]
5: 0x128a178 [node]
6: 0x126167e v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
7: 0x1262964 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
8: 0x123f586 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [node]
9: 0x12310e4 v8::internal::FactoryBasev8::internal::Factory::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Taggedv8::internal::Map, v8::internal::AllocationAlignment) [node]
10: 0x12329ae v8::internal::FactoryBasev8::internal::Factory::NewRawOneByteString(int, v8::internal::AllocationType) [node]
11: 0x1232a26 v8::internal::FactoryBasev8::internal::Factory::NewStringFromOneByte(v8::base::Vector
