xarray icon indicating copy to clipboard operation
xarray copied to clipboard

Cache some properties

Open Illviljan opened this issue 4 years ago • 4 comments

Cache some small properties that are rather slow to calculate but doesn't change that often.

Questions that needs to be resolved:

  • [ ] Can these properties change during the lifetime of the class? If so the cache needs to be reset when that happens.
  • [ ] Related to #3514
  • [ ] Tests added
  • [ ] Passes pre-commit run --all-files
  • [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
  • [ ] New functions/methods are listed in api.rst

Notes

  • Mixin classes makes it difficult to cache properties. For example ndim in NdimSizeLenMixin cannot be easily be replaced with cache_readonly.

Illviljan avatar Jun 27 '21 12:06 Illviljan

Unit Test Results

         6 files           6 suites   53m 11s :stopwatch: 16 200 tests 14 465 :heavy_check_mark: 1 735 :zzz: 0 :x: 90 396 runs  82 221 :heavy_check_mark: 8 175 :zzz: 0 :x:

Results for commit 59bd433e.

:recycle: This comment has been updated with latest results.

github-actions[bot] avatar Jun 27 '21 12:06 github-actions[bot]

Thanks for kicking this off @Illviljan .

My concern with this is that adding the _cache nullifies the benefits of slots. @keewis do you have any thoughts?

It also sounds like the speed-ups I was seeing might not be real. Probably we should have an ASV so there's a benchmark we're all looking at.

max-sixty avatar Jun 28 '21 02:06 max-sixty

The case I'm optimizing is dataset interpolation with many variables. Although it's not the bottleneck there I halved the shape time from ~180ms to ~92ms with this change.

Illviljan avatar Jun 28 '21 18:06 Illviljan

Resurrecting this, as discussed on the dev call.

Could we replace the pandas decorator with the one from the standard library? That may require adding __dict__ as a slot.

Then as long as the benchmarks still look good, there was consensus that we should merge.

max-sixty avatar Jul 06 '22 16:07 max-sixty