BlenderSourceTools icon indicating copy to clipboard operation
BlenderSourceTools copied to clipboard

Optimizing validate_element in datamodel.py for faster loading of large dmx files

Open Almie opened this issue 1 year ago • 1 comments

I had a scenario where I was trying to load a relatively large dmx file using datamodel.py and it was taking a very long time. I noticed that validate_element is using an O(n) lookup for every element added during load, so I added some sets to convert this to O(1), unless validation fails.

Time comparison

In the case of my dmx file (SFM session, ~57MB), these were the load time before and after the change:

  • before: 1hr+ (I left it running for 1 hour and it's still not finished)
  • after: 10 sec

Let me know if you'd like me to make any adjustments.

Almie avatar Dec 20 '24 00:12 Almie

This is certainly a problem, thank you for identifying it. However these changes cause problems when modifying a datamodel because we never remove items from either set. I'll have to think more about a solution.

Artfunkel avatar Jan 09 '25 21:01 Artfunkel