xFasterTransformer icon indicating copy to clipboard operation
xFasterTransformer copied to clipboard

Add env param KV_CACHE_LOCATION to control kv cache memory numanode location

Open a3213105 opened this issue 1 year ago • 2 comments

Usage: before you run instance export KV_CACHE_LOCATION=#memory_numa_node_id_you_want_to_use_for_kv_cache

by defaults, kv_cache location is the same as other parts of instance.

a3213105 avatar Jun 28 '24 04:06 a3213105

@a3213105 we need to put the environment reading to a unified place, please put it to environment.h

pujiang2018 avatar Jul 02 '24 08:07 pujiang2018

@a3213105 we need to put the environment reading to a unified place, please put it to environment.h

DONE

a3213105 avatar Jul 02 '24 13:07 a3213105