Excessive RAM usage by Microstream

[Sandared]

Environment Details

• MicroStream Version: 06.01.00-MS-GA
• JDK version: openjdk 11.0.7 2020-04-14 OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.7+10) OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.7+10, mixed mode)
• OS: Windows 11 Pro, Version 21H2, Build 22000.556
• Used frameworks: Plain Java

Describe the bug

I have a data model like this

public class Data {
        private Map<String, String> data = new HashMap<>();
    
        public void update(Map<String, String> newData) {
            this.data = newData;
        }
    }

Data is stored as the root element in the StorageManager. When i repeatedly switch the internal data of Data and store it in Microstream, then the RAM usage of Microstream at some point hits the maximum available RAM given to the JVM.

The problem initially arises after some minutes of replacing and storing the HashMap.

When the application is stopped after using a lot of RAM and then restarted, the application takes roughly the same high amount of RAM after a few seconds.

To Reproduce

I have created a small sample application that leads to the above mentioned behavior: https://github.com/Sandared/microstream-ram-usage Just start the application and wait for some minutes. I monitored the application with VisualVM. When the Ram is at about 3-4 GB (my max RAM setting) I stopped the application and then restarted it. The inital run takes the max amount of RAM after some minutes. The second run will take the max amount of RAM after a few seconds.

Expected behavior

I would have expected Microstream to only use the RAM needed to store the currently set Hashmap, i.e., the RAM usage over time should not grow endlessly, but should stay roughly the same, no matter how often I replace the data HashMap.

Screenshots

RAM usage over time of inital run: Sampled Memory of initial run:

RAM usage over time of second run: Sampled memory of second run:

[balajisraghavan]

+1 I'm seeing the exact same issue as well. Microstream causes memory exhaustion when using a thread-local eager storer to save state every minute over 12-16 hours. On disk data size is 99MB