arkouda
arkouda copied to clipboard
Potentially auto-tune comm aggregators to system configuration
auto-tune comm aggregators to system configuration
- since buffers are sized to num-cores x num-locales
- buffers will spill out of cache potentially at larger node counts
- possibly tune at server startup
- this will effect many server operations like sort, int-indexed gather/scatter, boolean-indexed expand/compress, etc.
I've been thinking about this off and on and debating if this should/could be done at server startup or if it should be an external process to reduce arkouda start up times. Where I think the tradeoffs are that if this is done in arkouda at startup you're tuning for the current numLocales and system at the cost of startup time, but an offline process (or something that caches values) your results could become stale but won't impact start up time.