natergj
JavaScript heap out of memory
Hi There, I am trying to write excel sheet with this awesome module. I have array of JSON object(more than 50k) and trying to add all records in sheet. But unfortunately I am getting Javascript heap out of memory.
I have got similar closed issue here
But I think I am not writing the book on every loop.
here is my code base
async.eachSeries(userData, (item, cb) => {
console.log(item.length);
for (let i = 0; i < item.length; i++) {
cRow++; cCol = 1;
for (let j = 0; j < item[i].length; j++) {
ws.cell(cRow, cCol).string(item[i][j].user);
ws.cell(cRow, questArr[item[i][j].questionId - 1]).string(item[i][j].passed === true ? "P" : "F").style(choicesStyle);
for (let k = 0; k < item[i][j].userSelection.length; k++) {
let uCol = Number(questArr[item[i][j].questionId - 1]) + Number(item[i][j].userSelection[k]);
ws.cell(cRow, uCol).string('X').style(choicesStyle);
}
}
}
cb();
}, () => {
totalRowCnt = cRow;
//ws.cell(1, 1, totalRowCnt, totalColCnt).style(borderStyle);
console.log("User data DONE");
//wb.write('./report/myExcel.xlsx');
wb.write('./report/Report.xlsx', function (err, stats) {
if (err) {
console.error(err);
} else {
console.log(stats); // Prints out an instance of a node.js fs.Stats object
console.log("EXCEL CREATED!");
}
});
});
I am writing after completing all the iteration. For 10k records is working great but for 40K its throwing error
This is the Error
Security context: 0x36565dcfb39 <JS Object>
2: onDataCallback [/opt/lrn-analytics/node_modules/excel4node/distribution/lib/worksheet/builder.js:~435] [pc=0x17a8dec6da6a] (this=0x3b94deeb98e1 <an XMLDocumentCB with map 0x2c7170dcf111>,chunk=0x1bf1448cbb61 <String[3]:
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory 1: node::Abort() [node] 2: 0xff9afc [node] 3: v8::Utils::ReportApiFailure(char const*, char const*) [node] 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node] 5: v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [node] 6: v8::internal::Runtime_AllocateInTargetSpace(int, v8::internal::Object**, v8::internal::Isolate*) [node]
This is my one of the array object
{ lessonId: 'lesson1',
user: '00008d5',
questionId: '21',
title: 'Important Laws and Priorities',
questionText: '\n\t\t\t\t\t<p>Which of the following is true under?</p>\n\t\t\t\t',
passed: true,
choices:
[ { text: 'You should contact Business .',
choiceId: '1',
isCorrect: 'true',
percentage: '83.40',
selected: true,
result: 'Y' },
{ text: 'Bribery is not acceptable.',
choiceId: '2',
isCorrect: 'false',
percentage: '4.04',
selected: true,
result: 'N' },
{ text: 'Reeeee',
choiceId: '3',
isCorrect: 'true',
percentage: '78.73',
selected: true,
result: 'Y' },
{ text: 'A bribe can be....',
choiceId: '4',
isCorrect: 'true',
percentage: '79.85',
selected: true,
result: 'Y' } ] }
You may try to increase the heapsize of V8. https://futurestud.io/tutorials/node-js-increase-the-memory-limit-for-your-process
For super large workbooks that require formatting, increasing the head size is probably the best solution at this point. If the data does not need to be exported in a nicely formatted excel workbook, I would recommend using CSV format, opening a write stream and adding data from your array to that stream line by line. The only limitation with that method would be the size of the original array and if you were streaming that data from an external data source (like SQL/noSQL database), there would be no limitation.
