arkouda icon indicating copy to clipboard operation
arkouda copied to clipboard

Potentially auto-tune comm aggregators to system configuration

Open mhmerrill opened this issue 4 years ago • 1 comments

auto-tune comm aggregators to system configuration

  • since buffers are sized to num-cores x num-locales
  • buffers will spill out of cache potentially at larger node counts
  • possibly tune at server startup
  • this will effect many server operations like sort, int-indexed gather/scatter, boolean-indexed expand/compress, etc.

mhmerrill avatar Mar 30 '21 14:03 mhmerrill

I've been thinking about this off and on and debating if this should/could be done at server startup or if it should be an external process to reduce arkouda start up times. Where I think the tradeoffs are that if this is done in arkouda at startup you're tuning for the current numLocales and system at the cost of startup time, but an offline process (or something that caches values) your results could become stale but won't impact start up time.

ronawho avatar Mar 30 '21 18:03 ronawho