zrm

Mappings should be bidirectional

5

Mappings created by miniupnpd add a DNAT rule but no SNAT rule. The SNAT rule is needed so that an application relying on the mapping to avoid need for keepalives...

LLaMA NUMA could be better

16

llama.cpp is memory bound, let's see what has a lot of memory bandwidth: NVIDIA V100 32GB: 900GB/s 2S Epyc 9000 (12xDDR5-4800/S): 922GB/s NVIDIA A100 40GB: 1555GB/s 2S Xeon Max (HBM):...

Numa

22

2S Xeon E5-2690v1 Before: ./main -n 1024 -m models/7B/ggml-model-q4_0.bin --ignore-eos -p "Building a website can be done in 10 simple steps:" -t 32 ... llama_print_timings: sample time = 796.65 ms...

Facilitating NAT traversal

2

One of the key failings of many existing consumer routers is that they make NAT traversal unnecessarily difficult for decentralized applications. I don't presently have your hardware so I can't...