[BUG] PageBuffer free queue in MemoryCache.cs grows unbounded (memory leak)

[pejtersen]

Version LiteDB v5.0.8

Describe the bug The _free queue of PageBuffer in MemoryCache.cs can grow without limit. In our use case we have 6406 entries in the free list. This uses about 55 MB of RAM (which is more than half of the applications entire RAM usage).

Code to Reproduce Unable to write something small and concise at the moment.

Expected behavior The free list should cap somewhere so that we do not keep leaking memory without limit.

Screenshots/Stacktrace

Additional context We insert and update a few hundred thousand documents to four different collections in the same database while at the same time reading from all these collections randomly (but with indices on relevant fields).

[botne]

I see the same issue, but it seems to be limited to operations spanning a lot of documents. Doing a COUNT or trying to delete using a query often ends in OutOfMemoryException.

Doing small transactions it usually ends up using 1662 extendpages before doing snapshot and freeing the cache.

1. How to reduce cache usage on operations that really does not store data (eg. count)
2. Should memorycache be able to reduce itself as well?

[temnava]

I've managed to save some RAM by storing the basic information of an entry in the table, and the payload into the FileStorage. I still get too much memory usage when there are more than 50 messages. Is there any way to force clearing the cache? Will that cause issues?

[gasparecoolshop]

I'm experiencing this issue too, MemoryCache is using more than 100MB most of which for _readable ConcurrentDictionary that contains more than 35k entries.

Is there any way to free up this space? @lbnascimento

[Delog-ru]

Any updates? PageBuffer generated 19MB of data in memory just after about 4000 operations of read and write to db. It's not much in a short term but in a week it stuff up all of my 20GB memory and crashes.

I have two databases and two instances of LiteDB:

1. first instance is in a static field and connection is open until I close an WPF app. In a window closing event I dispose the instance;
2. second instance is running in a separate thread (not UI thread) and opens and closes connection periodically. During opened connection it reads and writes in both instances of LiteDB.

[BaerMitUmlaut]

This is a bit of an issue for me, too. I load around 300k documents, which are only used for a short amount of time, and RAM usage grows uncontrollably.

[zalza13]

Hi everyone! I’ve implemented my own memory cleaner for MemoryCacheV2, and so far, it’s performing well even with large collections and hundreds of operations. 🚀

Check it out here: LiteDb5_Memory

I’d love it if you could give it a try in your own projects and let me know how it performs. Your feedback, suggestions, or any issues you find would mean a lot to me and help improve it further! 😊

cc: @BaerMitUmlaut @Delog-ru @gasparecoolshop @pejtersen @botne @temnava

[zalza13]

Related to #2619