Profilig heaps with memwatch

[hellboy81]

• Is there possible to profile heap w/o changing source code?
• Is there possible to log in what variables, array indexes, code fragments?
• Is there some alternatives for node-memwatch?

[hellboy81]

• Can node-memwatch be used in production environment?
• Which best practices should be used to detect memory leaks (may be also with v8-profiler)

[jacoscaz]

@hellboy81 I've used memwatch(-next) to trigger heapdump with very good results so far. Saved me a ton of debugging.

[marcominetti]

I'll include heapdump as enhancement to allow automatic dumps on memwatch events (configurable)

[jhiver]

+1

[hellboy81]

+ìnfinity

[joeytwiddle]

When our leak is detected, the app is using about 1GB of RAM. Each heap snapshot takes about 8 seconds. The diff (which we trigger on the next GC) takes 52 seconds!

Diffing is a synchronous process right? That means our server is blocking all requests while the diff is being processed. That's unacceptable for production. So currently we only report "leak detected" in production, and try to reproduce on a dev machine for heap snapshots and diffing.

What would be really great:

1. Instead of starting and ending a HeapDiff, save two HeapSnapshots to a file. (Accept the 8 second hit for each.)
2. Diff the two heaps in a separate process (e.g. a shell command the developer can execute).

Is that possible now? Or is it planned? Thanks!

[jackycute]

Agreed with @joeytwiddle The diffing is blocking and takes long time.

[joeytwiddle]

@jackycute The diffing can still be useful on a development machine, but in staging (and sometimes production) I am now using this approach:

After memwatch reports a possible leak, I use 'heapdump' to take 5 snapshots after each of the next 5 GCs.

const memwatch = require('memwatch-next');
const heapdump = require('heapdump');

const maxHeapDumpsToSave = 5;

let leakDetected = false;
let numHeapDumpsSaved = 0;

memwatch.on('leak', function (info) {
  console.warn(new Date(), "Possible memory leak detected:", info);
  leakDetected = true;
  // We leave it to the 'stats' event to snapshot and diff heaps
  // We don't do it here because it can be a very long time between two 'leak' events,
  // resulting in large and different heaps.
});

memwatch.on('stats', function (d) {
  if (leakDetected && numHeapDumpsSaved < maxHeapDumpsToSave) {
    console.warn(new Date(), "Dumping the heap...");

    const filename = `./heapdump-${process.pid}-${numHeapDumpsSaved}.heapsnapshot`;

    heapdump.writeSnapshot(filename, () => {
      console.warn(new Date(), "Heap dump completed.");
    });

    numHeapDumpsSaved++;
  }
});

When these files appear on disk, I use Chrome devtools to compare them. (The UI for this changed recently, but you can drag files into the devtools pane, or hit Cmd-O / Ctrl-O.)