MASQ-Project
Retaining Unwritten Debts in Memory
When the Node crashes when it has a bunch of unwritten debts in memory, and they disappear without making it into the database.
We build the Node not to crash, so theoretically crashing shouldn't be that big a concern; but remember, before we play the Graceful Shutdown epic, every Node shutdown is indistinguishable from a crash, because it just stops the process dead. Therefore, presently there's no way to move a Node from the running state to the not-running state without crashing it.
This concern might raise the priority of Graceful Shutdown...
Bert — 09/06/2022 I've invented a concept how we can escape from this threat. There is even a card about it and I keep it back in my head hoping it will be implemented. It would solve a trouble in your named case but also in some other. So it has a potential of a general purpose help structure. Solved by combining an impl of Drop with writing important data to database in the program's post-morte moments. When overally crashing, we still can save the data persistently. We just need a smart wrapper. I've also proved the concept in a spike, if you have doubts. Dan Wiebe — 09/06/2022 Drop implementations won't work for what we have now. When you kill a process the way we kill it, the operating system simply doesn't schedule its next timeslice. It's removed from the process queue and thrown away without ever getting another chance to run.
Graceful Shutdown is of course the ultimate solution to this problem. An interim solution (at least for Linux and Mac) would be, instead of terminating the process, to send it a SIGTERM signal. We'd have to sense that signal and make sure all the various threads and actors were made properly aware of it. As a matter of fact, that exact functionality will probably be part of Graceful Shutdown.
