libkrun icon indicating copy to clipboard operation
libkrun copied to clipboard

numa support: in 128 cores machine, could only see 16 cores in libkrun vm.

Open ifquant opened this issue 10 months ago • 2 comments

do libkrun support numa? i use libkrun through muvm in a numa machine.

in real hardware:

[root@localhost 4kvm]# numactl -H
available: 0 nodes (0-7)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
node 0 size: 32259 MB
node 0 free: 14635 MB
node 1 cpus: 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
node 1 size: 32725 MB
node 1 free: 7097 MB
node 2 cpus: 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
node 2 size: 32725 MB
node 2 free: 16607 MB
node 3 cpus: 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
node 3 size: 32725 MB
node 3 free: 28808 MB
.....

in libkrun vm

[root@localhost 4kvm]# numactl -H
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
node 0 size: 204309 MB
node 0 free: 200872 MB
node distances:
node 0
0: 10

i also add some code in Vcpu.run

    pub fn run(&mut self) {
        // Start running the machine state in the `Paused` state.
        {
            let mut cpuset: cpu_set_t = unsafe { zeroed() };
            unsafe { CPU_ZERO(&mut cpuset) };
            unsafe { CPU_SET(self.id as usize, &mut cpuset) };
            
            let thread = unsafe { pthread_self() };
            let result = unsafe {
                pthread_setaffinity_np(
                    thread,
                    size_of::<cpu_set_t>(),
                    &cpuset as *const cpu_set_t,
                )
            };
            if result != 0 {
                eprintln!("Failed to set CPU affinity: {}", std::io::Error::last_os_error());
            } else {
                eprintln!("set CPU affinity success");
            }

            // 检查NUMA是否可用
            let numa_available = unsafe { numa_available() };
            if numa_available == -1 {
                    eprintln!("NUMA is not available");
            } else {
                let node = unsafe { numa_node_of_cpu(self.id as libc::c_int) };
                if node == -1 {
                    eprintln!("Cannot determine NUMA node for CPU {}", self.id);
                } else {
                    unsafe { numa_set_preferred(node) };
                    eprintln!("Set NUMA affinity: CPU {} -> NUMA node {}", self.id, node);
                }
             }
        }

        StateMachine::run(self, Self::paused);
    }

i have seen "Set NUMA affinity" has been done from 1~128, but i could only see core 16.

i want to support numa in libkrun. how to do it?.

ifquant avatar Apr 17 '25 07:04 ifquant