microstream reports one.microstream.storage.exceptions.StorageExceptionConsistency: No entity found for objectId 1000000000001447523

[useryq8]

Environment Details

• MicroStream Version:08.01.00-MS-GA
• JDK version:17.04
• OS:Ubuntu 18.04.6 LTS, centos 7
• Used frameworks: Spring boot 2.6.8

Describe the bug

the store performance become being slow when the storage size is about 1.2G, after cleaning some data , the size does not has any change. so I stop my app, and write a new tools, which read the existing storage the write to new storageManager, after copying , then validate it , but when I open the new storageManager, it reports the following error

one.microstream.storage.exceptions.StorageExceptionConsistency: No entity found for objectId 1000000000001447523

To Reproduce

copy the existing microstream storageManager data to a new one storageManager, then open it and validate it

my core code is like this

1, open the existing and new storageManagers by try
    the DataRoot has some LazyArrayLists

2,
      PListContainer<CellMeta> newCellMetaList = new PListContainer<>(newMeta.getStorage());
      PListContainer<CellMeta> existingCellMetaList = existingTables.get(branch); // existingTables is one LazyArrayList of the existing storageManager

      newCellMetaList.writeOp(metaList -> {
        CellMeta[] metaArray = existingCellMetaList.toArray(new CellMeta[0]);
        metaList.addAll(Arrays.asList(metaArray));
      });

      newTables.put(branch, newCellMetaList);

    newMeta.store(newTables); //  newMeta is new storageManager object
    newTables.store(); // newTables is a a member for DataRoot of new storageManager

Expected behavior

when validating , it works well ,it can read data from new storageManager

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

2023-11-02 13:39:42,322 ERROR a.m.osmdb.cellconn.base.PListContainer.readOp [pool-1-thread-5] Error while reading value, errMsg:Problem in channel #0 one.microstream.storage.exceptions.StorageException: Problem in channel #0 at one.microstream.storage.types.StorageChannelTask$Abstract.checkForProblems(StorageChannelTask.java:114) at one.microstream.storage.types.StorageChannelTask$Abstract.waitOnCompletion(StorageChannelTask.java:176) at one.microstream.storage.types.StorageRequestAcceptor$Default.waitOnTask(StorageRequestAcceptor.java:162) at one.microstream.storage.types.StorageRequestAcceptor$Default.queryByObjectIds(StorageRequestAcceptor.java:191) at one.microstream.storage.embedded.types.EmbeddedStorageBinarySource$Default.readByObjectIds(EmbeddedStorageBinarySource.java:104) at one.microstream.persistence.binary.types.BinaryLoader$Default.readLoadOidData(BinaryLoader.java:790) at one.microstream.persistence.binary.types.BinaryLoader$Default.getObject(BinaryLoader.java:858) at one.microstream.persistence.types.PersistenceManager$Default.getObject(PersistenceManager.java:417) at one.microstream.storage.types.StorageConnection.getObject(StorageConnection.java:455) at one.microstream.storage.types.Database$Default.getObject(Database.java:165) at one.microstream.reference.Lazy$Default.load(Lazy.java:527) at one.microstream.reference.Lazy$Default.get(Lazy.java:508) at one.microstream.collections.lazy.LazyArrayList$Segment.getData(LazyArrayList.java:1223) at one.microstream.collections.lazy.LazyArrayList.get(LazyArrayList.java:260) at java.base/java.util.AbstractList$Itr.next(AbstractList.java:371) at xxx.cellconn.CopyMigrationMetaListValidator.lambda$checkBase$7(CopyMigrationMetaListValidator.java:155) at xxx.cellconn.base.PListContainer.readOp(PListContainer.java:67) at xxx.cellconn.CopyMigrationMetaListValidator.lambda$checkBase$8(CopyMigrationMetaListValidator.java:165) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: one.microstream.storage.exceptions.StorageExceptionConsistency: No entity found for objectId 1000000000001447523 at one.microstream.storage.types.StorageChannel$EntityCollectorByOid.accept(StorageChannel.java:826) at one.microstream.persistence.binary.types.LoadItemsChain$ChannelHashing$ChainItemObjectIdSet.iterate(LoadItemsChain.java:330) at one.microstream.storage.types.StorageChannel$Default.collectLoadByOids(StorageChannel.java:591) at one.microstream.storage.types.StorageRequestTaskLoadByOids$Default.internalProcessBy(StorageRequestTaskLoadByOids.java:58) at one.microstream.storage.types.StorageRequestTaskLoadByOids$Default.internalProcessBy(StorageRequestTaskLoadByOids.java:28) at one.microstream.storage.types.StorageChannelTask$Abstract.processBy(StorageChannelTask.java:252) at one.microstream.storage.types.StorageChannel$Default.work(StorageChannel.java:409) at one.microstream.storage.types.StorageChannel$Default.run(StorageChannel.java:492) ... 1 common frames omitted

[hg-ms]

Hello, The exception is most likely caused by the lazy references in the source storage. Lazy references are tricky to copy from one storage to another as they are bound to a storage. To do so they must be loaded, and you need to unlink (one.microstream.reference.Lazy.Default.$unlink()) the lazy references from the source storage.

To reduce the size of a storage you can enforce cleanup by manually calling issueFullCacheCheck() and issueFullFileCheck() from the storage manager.

In case you want create a “cleaned up” copy of a storage you may use the StorageConverter to copy a non running storage. This will not copy unreferenced (deleted) data.