ktransformers
ktransformers copied to clipboard
Published
4 months ago
•
kvcache-ai
kvcache-ai
Error `what(): no weight source` with GLM 4.6-INT8
Reminder
- [x] I have read the above rules and searched the existing issues.
System Info
CPU: 2x Intel Xeon 8592+ ES Q2SP (60c per socket) GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96GB) OS: Arch Linux (kernel 6.17.8-arch1-1)
SGLang: 0.5.5.post3 kt-kernel: 0.1.0 (cloned from github)
Reproduction
python -m sglang.launch_server \
--host 0.0.0.0 \
--port 8000 \
--model /data/models/GLM-4.6-FP8 \
--trust-remote-code \
--mem-fraction-static 0.92 \
--chunked-prefill-size 4096 \
--served-model-name GLM-4.6 \
--enable-mixed-chunk \
--kt-method AMXINT8 \
--kt-weight-path /data/models/GLM-4.6-Channel-INT8 \
--kt-cpuinfer 120 \
--kt-threadpool-count 4 \
--kt-num-gpu-experts 32 \
--kt-max-deferred-experts-per-token 2
Error:
[2025-11-27 06:21:20] WARNING server_args.py:1213: Attention backend not explicitly specified. Use flashinfer backend by default.
[2025-11-27 06:21:21] server_args=ServerArgs(model_path='/data/models/GLM-4.6-FP8', tokenizer_path='/data/models/GLM-4.6-FP8', tokenizer_mode='auto', tokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=None, is_embedding=False, enable_multimodal=None, revision=None, model_impl='auto', host='0.0.0.0', port=8000, grpc_mode=False, skip_server_warmup=False, warmups=None, nccl_port=None, checkpoint_engine_wait_weights_before_ready=False, dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='auto', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, mem_fraction_static=0.92, max_running_requests=None, max_queued_requests=None, max_total_tokens=None, chunked_prefill_size=4096, max_prefill_tokens=16384, schedule_policy='fcfs', enable_priority_scheduling=False, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, schedule_conservativeness=1.0, page_size=1, hybrid_kvcache_ratio=None, swa_full_tokens_ratio=0.8, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', device='cuda', tp_size=1, pp_size=1, pp_max_micro_batch_size=None, stream_interval=1, stream_output=False, random_seed=157567588, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, crash_dump_folder=None, show_time_cost=False, enable_metrics=False, enable_metrics_for_all_schedulers=False, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, collect_tokens_histogram=False, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_trace=False, otlp_traces_endpoint='localhost:4317', export_metrics_to_file=False, export_metrics_to_file_dir=None, api_key=None, served_model_name='GLM-4.6', weight_version='default', chat_template=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, tool_call_parser=None, tool_server=None, sampling_defaults='model', dp_size=1, load_balance_method='round_robin', load_watch_interval=0.1, prefill_round_robin_balance=False, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, enable_lora=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, attention_backend='flashinfer', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, nsa_prefill_backend='flashmla_sparse', nsa_decode_backend='fa3', speculative_algorithm=None, speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, speculative_attention_mode='prefill', speculative_moe_runner_backend=None, speculative_ngram_min_match_window_size=1, speculative_ngram_max_match_window_size=12, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_branch_length=18, speculative_ngram_capacity=10000000, ep_size=1, moe_a2a_backend='none', moe_runner_backend='auto', flashinfer_mxfp4_moe_precision='default', enable_flashinfer_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm='static', init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, mooncake_ib_device=None, max_mamba_cache_size=None, mamba_ssm_dtype='float32', mamba_full_memory_ratio=0.9, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='layer_first', hicache_storage_backend=None, hicache_storage_prefetch_policy='best_effort', hicache_storage_backend_extra_config=None, enable_lmcache=False, kt_weight_path='/data/models/GLM-4.6-Channel-INT8', kt_method='AMXINT8', kt_cpuinfer=120, kt_threadpool_count=4, kt_num_gpu_experts=32, kt_max_deferred_experts_per_token=2, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', multi_item_scoring_delimiter=None, disable_radix_cache=False, cuda_graph_max_bs=256, cuda_graph_bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, enable_layerwise_nvtx_marker=False, enable_nccl_nvls=False, enable_symm_mem=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, disable_overlap_schedule=False, enable_mixed_chunk=True, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, enable_torch_compile=False, enable_piecewise_cuda_graph=False, torch_compile_max_bs=32, piecewise_cuda_graph_max_tokens=4096, piecewise_cuda_graph_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 640, 768, 896, 1024, 1152, 1280, 1408, 1536, 1664, 1792, 1920, 2048, 2176, 2304, 2432, 2560, 2688, 2816, 2944, 3072, 3200, 3328, 3456, 3584, 3712, 3840, 3968, 4096], piecewise_cuda_graph_compiler='eager', torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, enable_weights_cpu_backup=False, enable_draft_weights_cpu_backup=False, allow_auto_truncate=False, enable_custom_logit_processor=False, flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=False, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, keep_mm_feature_on_device=False, enable_return_hidden_states=False, scheduler_recv_interval=1, numa_node=None, enable_deterministic_inference=False, rl_on_policy_target=None, enable_attn_tp_input_scattered=False, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, disaggregation_decode_enable_offload_kvcache=False, num_reserved_decode_tokens=512, disaggregation_decode_polling_interval=1, custom_weight_loader=[], weight_loader_disable_mmap=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, mm_max_concurrent_calls=32, mm_per_request_timeout=10.0, enable_broadcast_mm_inputs_process=False, decrypted_config_file=None, decrypted_draft_config_file=None)
[2025-11-27 06:21:21] Using default HuggingFace chat template with detected content format: openai
[2025-11-27 06:21:25] Init torch distributed begin.
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[2025-11-27 06:21:26] Init torch distributed ends. mem usage=0.00 GB
[2025-11-27 06:21:27] Load weight begin. avail mem=94.31 GB
CPUInfer[0x4ad548e0]: Hello
WorkerPool[0x5db02e10] 4 subpools, [numa:threads][0:30] [1:30] [2:30] [3:30]
===========In NumaPool============
In Numa Worker Pool at NUMA 0, 30 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 1, 30 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 2, 30 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 3, 30 threads
Loading safetensors checkpoint shards: 0% Completed | 0/93 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 1% Completed | 1/93 [00:00<00:14, 6.36it/s]
Loading safetensors checkpoint shards: 2% Completed | 2/93 [00:00<00:14, 6.26it/s]
Loading safetensors checkpoint shards: 3% Completed | 3/93 [00:00<00:15, 5.85it/s]
Loading safetensors checkpoint shards: 4% Completed | 4/93 [00:00<00:14, 6.15it/s]
Loading safetensors checkpoint shards: 5% Completed | 5/93 [00:00<00:13, 6.37it/s]
Loading safetensors checkpoint shards: 6% Completed | 6/93 [00:00<00:14, 6.06it/s]
Loading safetensors checkpoint shards: 8% Completed | 7/93 [00:01<00:14, 5.82it/s]
Loading safetensors checkpoint shards: 9% Completed | 8/93 [00:01<00:14, 6.05it/s]
Loading safetensors checkpoint shards: 10% Completed | 9/93 [00:01<00:13, 6.35it/s]
Loading safetensors checkpoint shards: 11% Completed | 10/93 [00:01<00:13, 6.34it/s]
Loading safetensors checkpoint shards: 12% Completed | 11/93 [00:01<00:13, 6.04it/s]
Loading safetensors checkpoint shards: 13% Completed | 12/93 [00:01<00:13, 5.82it/s]
Loading safetensors checkpoint shards: 14% Completed | 13/93 [00:02<00:13, 5.95it/s]
Loading safetensors checkpoint shards: 15% Completed | 14/93 [00:02<00:16, 4.88it/s]
Loading safetensors checkpoint shards: 16% Completed | 15/93 [00:02<00:14, 5.23it/s]
Loading safetensors checkpoint shards: 17% Completed | 16/93 [00:02<00:14, 5.47it/s]
Loading safetensors checkpoint shards: 18% Completed | 17/93 [00:02<00:13, 5.52it/s]
Loading safetensors checkpoint shards: 19% Completed | 18/93 [00:03<00:13, 5.72it/s]
Loading safetensors checkpoint shards: 20% Completed | 19/93 [00:03<00:12, 6.15it/s]
Loading safetensors checkpoint shards: 22% Completed | 20/93 [00:03<00:11, 6.25it/s]
Loading safetensors checkpoint shards: 23% Completed | 21/93 [00:03<00:12, 6.00it/s]
Loading safetensors checkpoint shards: 24% Completed | 22/93 [00:03<00:11, 6.17it/s]
Loading safetensors checkpoint shards: 25% Completed | 23/93 [00:03<00:10, 6.54it/s]
Loading safetensors checkpoint shards: 26% Completed | 24/93 [00:04<00:11, 6.23it/s]
Loading safetensors checkpoint shards: 27% Completed | 25/93 [00:04<00:10, 6.37it/s]
Loading safetensors checkpoint shards: 28% Completed | 26/93 [00:04<00:09, 6.74it/s]
Loading safetensors checkpoint shards: 29% Completed | 27/93 [00:04<00:09, 7.10it/s]
Loading safetensors checkpoint shards: 30% Completed | 28/93 [00:04<00:08, 7.37it/s]
Loading safetensors checkpoint shards: 31% Completed | 29/93 [00:04<00:08, 7.14it/s]
Loading safetensors checkpoint shards: 32% Completed | 30/93 [00:04<00:09, 6.53it/s]
Loading safetensors checkpoint shards: 33% Completed | 31/93 [00:05<00:10, 6.17it/s]
Loading safetensors checkpoint shards: 34% Completed | 32/93 [00:05<00:10, 5.92it/s]
Loading safetensors checkpoint shards: 35% Completed | 33/93 [00:05<00:09, 6.31it/s]
Loading safetensors checkpoint shards: 37% Completed | 34/93 [00:05<00:09, 6.47it/s]
Loading safetensors checkpoint shards: 38% Completed | 35/93 [00:05<00:09, 6.34it/s]
Loading safetensors checkpoint shards: 39% Completed | 36/93 [00:05<00:09, 6.05it/s]
Loading safetensors checkpoint shards: 40% Completed | 37/93 [00:06<00:09, 5.84it/s]
Loading safetensors checkpoint shards: 41% Completed | 38/93 [00:06<00:09, 5.95it/s]
Loading safetensors checkpoint shards: 42% Completed | 39/93 [00:06<00:08, 6.17it/s]
Loading safetensors checkpoint shards: 43% Completed | 40/93 [00:06<00:11, 4.59it/s]
Loading safetensors checkpoint shards: 44% Completed | 41/93 [00:06<00:10, 4.78it/s]
Loading safetensors checkpoint shards: 45% Completed | 42/93 [00:07<00:09, 5.12it/s]
Loading safetensors checkpoint shards: 46% Completed | 43/93 [00:07<00:09, 5.47it/s]
Loading safetensors checkpoint shards: 47% Completed | 44/93 [00:07<00:08, 5.46it/s]
Loading safetensors checkpoint shards: 48% Completed | 45/93 [00:07<00:08, 6.00it/s]
Loading safetensors checkpoint shards: 49% Completed | 46/93 [00:07<00:07, 6.00it/s]
Loading safetensors checkpoint shards: 51% Completed | 47/93 [00:07<00:07, 6.22it/s]
Loading safetensors checkpoint shards: 53% Completed | 49/93 [00:08<00:06, 7.31it/s]
Loading safetensors checkpoint shards: 54% Completed | 50/93 [00:08<00:05, 7.19it/s]
Loading safetensors checkpoint shards: 55% Completed | 51/93 [00:08<00:06, 6.66it/s]
Loading safetensors checkpoint shards: 56% Completed | 52/93 [00:08<00:06, 6.26it/s]
Loading safetensors checkpoint shards: 57% Completed | 53/93 [00:08<00:06, 6.31it/s]
Loading safetensors checkpoint shards: 58% Completed | 54/93 [00:08<00:05, 6.64it/s]
Loading safetensors checkpoint shards: 59% Completed | 55/93 [00:09<00:05, 6.48it/s]
Loading safetensors checkpoint shards: 60% Completed | 56/93 [00:09<00:05, 6.65it/s]
Loading safetensors checkpoint shards: 61% Completed | 57/93 [00:09<00:05, 6.28it/s]
Loading safetensors checkpoint shards: 62% Completed | 58/93 [00:09<00:05, 6.42it/s]
Loading safetensors checkpoint shards: 63% Completed | 59/93 [00:09<00:05, 6.58it/s]
Loading safetensors checkpoint shards: 65% Completed | 60/93 [00:09<00:04, 6.86it/s]
Loading safetensors checkpoint shards: 66% Completed | 61/93 [00:09<00:04, 6.67it/s]
Loading safetensors checkpoint shards: 67% Completed | 62/93 [00:10<00:04, 6.64it/s]
Loading safetensors checkpoint shards: 68% Completed | 63/93 [00:10<00:04, 6.66it/s]
Loading safetensors checkpoint shards: 69% Completed | 64/93 [00:10<00:04, 6.28it/s]
Loading safetensors checkpoint shards: 70% Completed | 65/93 [00:10<00:04, 5.99it/s]
Loading safetensors checkpoint shards: 71% Completed | 66/93 [00:10<00:04, 5.80it/s]
Loading safetensors checkpoint shards: 72% Completed | 67/93 [00:10<00:04, 5.87it/s]
Loading safetensors checkpoint shards: 73% Completed | 68/93 [00:11<00:04, 5.74it/s]
Loading safetensors checkpoint shards: 74% Completed | 69/93 [00:11<00:04, 5.84it/s]
Loading safetensors checkpoint shards: 75% Completed | 70/93 [00:11<00:03, 6.28it/s]
Loading safetensors checkpoint shards: 76% Completed | 71/93 [00:11<00:03, 6.07it/s]
Loading safetensors checkpoint shards: 77% Completed | 72/93 [00:11<00:03, 6.42it/s]
Loading safetensors checkpoint shards: 80% Completed | 74/93 [00:11<00:02, 7.86it/s]
Loading safetensors checkpoint shards: 81% Completed | 75/93 [00:12<00:02, 7.48it/s]
Loading safetensors checkpoint shards: 82% Completed | 76/93 [00:12<00:02, 7.02it/s]
Loading safetensors checkpoint shards: 83% Completed | 77/93 [00:12<00:02, 7.17it/s]
Loading safetensors checkpoint shards: 84% Completed | 78/93 [00:12<00:02, 6.64it/s]
Loading safetensors checkpoint shards: 85% Completed | 79/93 [00:12<00:02, 6.91it/s]
Loading safetensors checkpoint shards: 86% Completed | 80/93 [00:12<00:01, 6.64it/s]
Loading safetensors checkpoint shards: 87% Completed | 81/93 [00:13<00:01, 6.51it/s]
Loading safetensors checkpoint shards: 88% Completed | 82/93 [00:13<00:01, 6.85it/s]
Loading safetensors checkpoint shards: 90% Completed | 84/93 [00:13<00:01, 7.68it/s]
Loading safetensors checkpoint shards: 91% Completed | 85/93 [00:13<00:01, 7.03it/s]
Loading safetensors checkpoint shards: 92% Completed | 86/93 [00:13<00:00, 7.05it/s]
Loading safetensors checkpoint shards: 94% Completed | 87/93 [00:13<00:00, 6.96it/s]
Loading safetensors checkpoint shards: 95% Completed | 88/93 [00:14<00:00, 6.48it/s]
Loading safetensors checkpoint shards: 96% Completed | 89/93 [00:14<00:00, 6.13it/s]
Loading safetensors checkpoint shards: 97% Completed | 90/93 [00:14<00:00, 6.15it/s]
Loading safetensors checkpoint shards: 98% Completed | 91/93 [00:14<00:00, 6.39it/s]
Loading safetensors checkpoint shards: 99% Completed | 92/93 [00:14<00:00, 6.10it/s]
Loading safetensors checkpoint shards: 100% Completed | 93/93 [00:14<00:00, 6.19it/s]
Loading safetensors checkpoint shards: 100% Completed | 93/93 [00:14<00:00, 6.26it/s]
TP MOE layer 3, pool: 0x5db02e10, expert num: 160, num_experts_per_tok: 8
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 2 at numa 2
terminate called after throwing an instance of 'std::runtime_error'
what(): no weight source
Fatal Python error: Aborted
Thread 0x00007fbd2dfff6c0 (most recent call first):
File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 359 in wait
File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 655 in wait
File "/home/admin/projects/sgl/.venv/lib/python3.12/site-packages/tqdm/_monitor.py", line 60 in run
File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fcd502e56c0 (most recent call first):
File "/home/admin/projects/sgl/.venv/lib/python3.12/site-packages/torch/_inductor/compile_worker/subproc_pool.py", line 61 in _recv_msg
File "/home/admin/projects/sgl/.venv/lib/python3.12/site-packages/torch/_inductor/compile_worker/subproc_pool.py", line 195 in _read_thread
File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012 in run
File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032 in _bootstrap
Thread 0x00007fd38dce1b80 (most recent call first):
File "/home/admin/projects/sgl/.venv/lib/python3.12/site-packages/kt_kernel/utils/amx.py", line 294 in load_weights
File "/home/admin/projects/sgl/sglang/python/sglang/srt/layers/moe/kt_ep_wrapper.py", line 258 in process_weights_after_loading
File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_loader/loader.py", line 618 in load_weights_and_postprocess
File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_loader/loader.py", line 599 in load_model
File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_loader/__init__.py", line 28 in get_model
File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_executor/model_runner.py", line 767 in load_model
File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_executor/model_runner.py", line 410 in initialize
File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_executor/model_runner.py", line 324 in __init__
File "/home/admin/projects/sgl/sglang/python/sglang/srt/managers/tp_worker.py", line 237 in __init__
File "/home/admin/projects/sgl/sglang/python/sglang/srt/managers/scheduler.py", line 312 in __init__
File "/home/admin/projects/sgl/sglang/python/sglang/srt/managers/scheduler.py", line 2712 in run_scheduler_process
File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108 in run
File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314 in _bootstrap
File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/multiprocessing/spawn.py", line 135 in _main
File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/multiprocessing/spawn.py", line 122 in spawn_main
File "<string>", line 1 in <module>
Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, psutil._psutil_linux, pybase64._pybase64, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, zmq.backend.cython._zmq, PIL._imaging, sentencepiece._sentencepiece, yaml._yaml, regex._regex, markupsafe._speedups, PIL._imagingft, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._pcg64, numpy.random._mt19937, numpy.random._generator, numpy.random._philox, numpy.random._sfc64, numpy.random.mtrand, _cffi_backend, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack, _cyutility, scipy._cyutility, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_schur_sqrtm, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._slsqplib, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy._lib._uarray._uarray, scipy.special._ufuncs_cxx, scipy.special._ellip_harm_2, scipy.special._special_ufuncs, scipy.special._gufuncs, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._hausdorff, scipy.spatial._distance_wrap, scipy.spatial.transform._rotation, scipy.spatial.transform._rigid_transform, scipy.optimize._direct, setproctitle._setproctitle, cuda.bindings._bindings.cydriver, cuda.bindings.cydriver, cuda.bindings.driver, tvm_ffi.core, msgspec._core, cuda.bindings._bindings.cynvrtc, cuda.bindings.cynvrtc, cuda.bindings.nvrtc, cuda.bindings._bindings.cyruntime_ptds, cuda.bindings._bindings.cyruntime, cuda.bindings.cyruntime, cuda.bindings.runtime, cuda_utils (total: 101)
Others
No response
It seems that there is a problem with the weight. Is the GLM-4.6-Channel-INT8 you quantified 4numa？
