openmicroscopy icon indicating copy to clipboard operation
openmicroscopy copied to clipboard

Improve performance of graph operations.

Open mtbc opened this issue 5 years ago • 2 comments

Three separate sources of slowness have been identified:

  1. Early in the process when data is being gathered, profiling shows Hibernate eating a lot of memory. Caches may need configuration and later versions of Hibernate may make that easier. GraphTraversal will tend not to be querying the same objects repeatedly.

  2. In reviewing the data gathered via Hibernate, GraphPolicyRule eats much CPU time in rule-matching. The internal implementation of the class could do non-trivial static analysis when initialized, compiling the list of rules to a decision tree of checks, branches, changes, making the subsequent matching much faster.

  3. In actioning the model changes, PostgreSQL can spend much time in managing the _reindexing_required table. Perhaps migration to https://github.com/glencoesoftware/omero-es can somehow help.

mtbc avatar Jul 28 '20 17:07 mtbc

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/omero-storage-reports/41819/2

imagesc-bot avatar Aug 21 '20 09:08 imagesc-bot

non-trivial static analysis when initialized

This could also spot opportunities to reorder rules to reduce number of passes in cases where outcome would remain unchanged. However, there are probably not big wins as the user can already pluck low-hanging fruit, e.g., listing transitions to RELEVANT before transitions from that to IS_NOT_LAST.

mtbc avatar Sep 01 '20 09:09 mtbc