wrye-bash
Memory/cache leaks when the Patch building finishes (or crashes)
In the back burner for now - just reporting it
See also #95 for a cache leak
There should be a finally someplace that closes resources - or even better have the resources play with with
In general we must revisit the patch building process with the resources leak in mind
Before building the (huge test) patch:
after:
after cbash - virtually no change !:
That's on top of my importers branch - TODO: do the same for 304.4
@iaz: to me that's patcher issue number 1
If you are looking for something to tackle here it is.
I am looking at it in the debugger since I turned the eye on the patcher (I have to) and for the life of me I can't see where the memory leaks. For some comments (that may be wrong - XXX commits are experimental to the bone) see https://github.com/wrye-bash/wrye-bash/commit/5da50690382870bb1b76060d091adf1554637b3d
I am dead busy on my wip branch but iff you find something concrete post here. I have not forgotten the imports links merge needless to say (heck it's sitting on my tree which could use some pruning) but will get back to it once the dust settles in my wip branch (depends on stuff there - as pretty much most of Bash).
Will look into it
By memory leaks, you mean it's using an excessive amount of memory, right?
Can you provide some examples/instructions on how to produce this?
The pic above is from 305 - I just built a huge python bashed patch (I have a huge oblivion mod list) - then built a cbash patch with same options
Leak as seen - Bash memory usage goes up during the patch and does not drop after patch dialog is destroyed.
So just download a bunch of oblivion mods and try to bash patch and that should work?
What options, or does it not really matter?
I had it happen while debugging the cell patcher recently - was triggered only with the cell patcher, but in the pic I crammed nearly all (if you cram all you do get a memory error - avoid replace FormIDs and alias mod names).
I actually since I posted have had some more clues but still investigating.
Note: the cell patcher patch I refer to above was in skyrim. So it's not oblivion related - it's in the main code
I dont know where the patcher code is, and it seems to be all over the place too(based on the commit you linked), so can you point me to it?
Oh forget about it for now - will come back to it when I have more data. Yes it's as you see in the commit - I am still moving stuff around, and our dev source guide page needs some love (I just can't afford the energy). It used to be all in the same place - in bosh py which was 30k lines when I started - 30k - but I think that's better :P
have a look at the #293
Re: patchers doc - it's an issue really if you follow my code TODOs comments - I never got round to finishing cause it's a couple weeks and I need to understand all the code there (which I don't). There is wiki page to get you started https://github.com/wrye-bash/wrye-bash/wiki/[dev]-CBash-vs-PBash and that's all I got. If you can document your findings while you learn so much the better
I'm fairly certain by now that this isn't a leak in any traditional sense, but a 'feature' of CPython. I notice the same thing with my rewritten mod checker as well. It temporarily skyrockets memory usage to >1GB, which then gets garbage collected down to ~440MB once the report is generated. Those 440MB don't disappear, but also don't grow. If it were a regular old memory leak, we would expect another run of the mod checker to increase memory usage again, but that doesn't happen. It skyrockets up to 1GB again, but then shrinks down to 440MB again as well.
Instead, I'm pretty sure CPython is simply not releasing memory back to the operating system. Maybe that's an optimization and it reuses the memory for allocations, or maybe it wants to avoid free() overhead. Not sure.
Some relevant SO posts:
- https://stackoverflow.com/a/31089481
- https://stackoverflow.com/a/15492488
- https://stackoverflow.com/a/15457947
