Unexpected pod delete for the cpu image

Open ChanryLimmy opened this issue 10 months ago • 1 comments

Describe the bug when i create a pod runs on my raspberry pi, it will get an unexpected shutdown. i know this operation is abnormal, but that is what i need. below is a temporary pod that you can see where the process goes. it will be endless restart if i create it with the development.yaml. is this the image problem or the arch or the resources requirement, or something else? do you guys have any idea on that? it runs perfectly on my x86 pc, by the way.

Screenshots or console output

root@r3s:~# kubectl run kokoro-fastapi \
  --image=ghcr.io/remsky/kokoro-fastapi-cpu:latest \
  --port=8880 \
  --restart=Never \
  --rm -it -- /app/entrypoint.sh
If you don't see a command prompt, try pressing enter.
2025-04-05 15:23:07.977 | INFO     | __main__:download_model:60 - Model files already exist and are valid
INFO:     Started server process [11]
INFO:     Waiting for application startup.
03:24:26 PM | INFO     | main:57 | Loading TTS model and voice packs...
03:24:26 PM | INFO     | model_manager:38 | Initializing Kokoro V1 on cpu
03:24:26 PM | DEBUG    | paths:101 | Searching for model in path: /app/api/src/models
03:24:26 PM | INFO     | kokoro_v1:45 | Loading Kokoro model on cpu
03:24:26 PM | INFO     | kokoro_v1:46 | Config path: /app/api/src/models/v1_0/config.json
03:24:26 PM | INFO     | kokoro_v1:47 | Model path: /app/api/src/models/v1_0/kokoro-v1_0.pth
/app/.venv/lib/python3.10/site-packages/torch/nn/modules/rnn.py:123: UserWarning: dropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout=0.2 and num_layers=1
  warnings.warn(
/app/.venv/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:143: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)
03:24:34 PM | DEBUG    | paths:153 | Scanning for voices in path: /app/api/src/voices/v1_0
03:24:34 PM | DEBUG    | paths:131 | Searching for voice in path: /app/api/src/voices/v1_0
03:24:34 PM | DEBUG    | model_manager:77 | Using default voice 'af_heart' for warmup
03:24:35 PM | INFO     | kokoro_v1:73 | Creating new pipeline for language code: a
pod "kokoro-fastapi" deleted
pod default/kokoro-fastapi terminated (Error)

Branch / Deployment used the latest tag for today in 2025-4-5

Operating System Ubuntu 24.10 (GNU/Linux 6.11.0-1010-raspi aarch64) Raspberry Pi Ubuntu 24.10 runs on Raspberry Pi 4b (4c4g)

Additional context without this pod, a top info snapshot is listed below.

%Cpu(s):  3.4 us,  3.3 sy,  0.0 ni, 93.1 id,  0.1 wa,  0.0 hi,  0.1 si,  0.0 st
MiB Mem :   3783.8 total,   1332.3 free,   1031.3 used,   1612.1 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   2752.5 avail Mem

and it is an k3s single node cluster installed with:

curl –sfL \
     https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | \
     INSTALL_K3S_MIRROR=cn sh -s - \
     --system-default-registry "registry.cn-hangzhou.aliyuncs.com"

no cpu and memory limit for that pod, and when it alive, the max cpu get to 950m, the max memory get to 320m. which is provided by kubectl top with watch on it.

Apr 05 '25 16:04 ChanryLimmy

I have the same problem running kokoro in a raspberry pi 4B rev 1. In my case im running it inside docker container but the problem occurs in the same spot.

2025-05-24 19:25:39.726 | INFO | main:download_model:60 - Model files already exist and are valid INFO: Started server process [11] INFO: Waiting for application startup. 07:26:49 PM | INFO | main:57 | Loading TTS model and voice packs... 07:26:49 PM | INFO | model_manager:38 | Initializing Kokoro V1 on cpu 07:26:49 PM | DEBUG | paths:101 | Searching for model in path: /app/api/src/models 07:26:49 PM | INFO | kokoro_v1:45 | Loading Kokoro model on cpu 07:26:49 PM | INFO | kokoro_v1:46 | Config path: /app/api/src/models/v1_0/config.json 07:26:49 PM | INFO | kokoro_v1:47 | Model path: /app/api/src/models/v1_0/kokoro-v1_0.pth /app/.venv/lib/python3.10/site-packages/torch/nn/modules/rnn.py:123: UserWarning: dropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout=0.2 and num_layers=1 warnings.warn( /app/.venv/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:143: FutureWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm. WeightNorm.apply(module, name, dim) 07:27:12 PM | DEBUG | paths:153 | Scanning for voices in path: /app/api/src/voices/v1_0 07:27:12 PM | DEBUG | paths:131 | Searching for voice in path: /app/api/src/voices/v1_0 07:27:12 PM | DEBUG | model_manager:77 | Using default voice 'af_heart' for warmup 07:27:12 PM | INFO | kokoro_v1:73 | Creating new pipeline for language code: a

No limits on CPU or memory and the top information with the container down is this

%Cpu(s): 1.3 us, 0.1 sy, 0.0 ni, 98.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 3797.4 total, 1015.0 free, 894.8 used, 1993.0 buff/cache
MiB Swap: 512.0 total, 35.7 free, 476.2 used. 2902.7 avail Mem

May 24 '25 19:05 IVIGG