CyCoreSystems
Dispatchers container causing my kamailio pod to CrashLoopBackOff in on-premise k8s cluster
Hi,
Thanks very much for your demo - it's been the only thing I've found that may address my SIP routing issues!
Though, I've been stuck on one problem for a week... I'm hoping someone could help me over this hurdle.
I'm running minikube on a Centos 7 VM.
Everything is Running with the exception of the kamailio pod, which is in CrashLoopBackOff.
Kubectl describe says the pod can't find /app:
exec: "/app": stat /app: no such file or directory: unknown
Any debug help would be much appreciated! Thank you, DRE
Commands and editing done:
$ minikube delete
$ minikube start
$ sudo swapoff -a
$ mkdir CyCore; cd CyCore
$ git clone https://github.com/CyCoreSystems/asterisk-k8s-demo.git
$ cd asterisk-k8s-demo/live-demo/k8s
$ vi 02-kamailio.yaml // change CLOUD to "", change nodeSelector to "type: kamailio"
$ vi 03-asterisk.yaml // change CLOUD to ""
$ kubectl label nodes minikube type=kamailio // So kamailio pod will run on minikube node
$ kubectl apply -f 00-namespace.yaml // so secret in voip will apply successfully
$ cd ../..
// edited inbound.conf.tmpl, created config zip.
$ kubectl -n voip create secret generic asterisk-config --from-file=asterisk-config.zip
$ cd live-demo/k8s
$ kubectl apply -f 01-nats.yaml
$ kubectl apply -f 02-kamailio.yaml
$ kubectl apply -f 03-asterisk.yaml
Results:
$ kubectl get all --namespace=voip
NAME READY STATUS RESTARTS AGE
pod/asterisk-758996b56d-t56c9 3/3 Running 4 95s
pod/kamailio-b96hs 2/3 CrashLoopBackOff 5 7m14s
pod/nats-666d56d6b8-q4bb5 1/1 Running 0 20m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/asterisk ClusterIP 10.105.56.62 <none> 5060/UDP 20m
service/kamailio ClusterIP 10.110.250.161 <none> 5060/UDP 20m
service/nats ClusterIP 10.104.131.110 <none> 4222/TCP,6222/TCP,8222/TCP 20m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/kamailio 1 1 0 1 0 type=kamailio 20m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/asterisk 1/1 1 1 20m
deployment.apps/nats 1/1 1 1 20m
NAME DESIRED CURRENT READY AGE
replicaset.apps/asterisk-758996b56d 1 1 1 20m
replicaset.apps/nats-666d56d6b8 1 1 1 20m
$ kubectl describe pod kamailio-b96hs --namespace=voip
Name: kamailio-b96hs
Namespace: voip
Priority: 0
Node: minikube/192.168.49.2
Start Time: Thu, 19 Aug 2021 08:56:05 -0400
Labels: component=kamailio
controller-revision-hash=5d4dd5d7cf
pod-template-generation=3
Annotations: <none>
Status: Running
IP: 192.168.49.2
IPs:
IP: 192.168.49.2
Controlled By: DaemonSet/kamailio
Containers:
kamailio:
Container ID: docker://90752fe442a294a2c37c44303d45dd6114dfe356663a40cfd7f403716331c7cc
Image: cycoresystems/kamailio
Image ID: docker-pullable://cycoresystems/kamailio@sha256:4a0804ab228a8eecd4abcf520a499af6119b4b30ae90a28abeec3f4008dc178b
Port: <none>
Host Port: <none>
State: Running
Started: Thu, 19 Aug 2021 08:57:52 -0400
Ready: True
Restart Count: 0
Environment:
CLOUD:
Mounts:
/data/kamailio from config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-82wpn (ro)
rtpproxy:
Container ID: docker://4a879cbfe529ee7c98efc452148b74caabeebbf6c93b8439839c4aa28cb9f69f
Image: cycoresystems/rtpproxy
Image ID: docker-pullable://cycoresystems/rtpproxy@sha256:e1443ca3b3a2f75cd0f26fb10e439cd6e920883d2ab1784962e56e2c01526beb
Port: <none>
Host Port: <none>
State: Running
Started: Thu, 19 Aug 2021 08:58:08 -0400
Ready: True
Restart Count: 0
Environment:
CLOUD:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-82wpn (ro)
dispatchers:
Container ID: docker://73389818732a94e43e3d8a62398b80f901aa6a1e8189569dc1f38a2e764a8ea4
Image: cycoresystems/dispatchers
Image ID: docker-pullable://cycoresystems/dispatchers@sha256:95f9cefb9d8454a907832f5d02d28a7f76f79e70c9a53c9c9a3aad4c700f27a0
Port: <none>
Host Port: <none>
Command:
/app
-set
voip:asterisk=1:5080
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: ContainerCannotRun
Message: OCI runtime create failed: container_linux.go:370: starting container process caused: exec: "/app": stat /app: no such file or directory: unknown
Exit Code: 127
Started: Thu, 19 Aug 2021 09:01:29 -0400
Finished: Thu, 19 Aug 2021 09:01:29 -0400
Ready: False
Restart Count: 5
Environment: <none>
Mounts:
/data/kamailio from config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-82wpn (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
config:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
default-token-82wpn:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-82wpn
Optional: false
QoS Class: BestEffort
Node-Selectors: type=kamailio
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 7m25s default-scheduler Successfully assigned voip/kamailio-b96hs to minikube
Normal Pulling 7m24s kubelet Pulling image "cycoresystems/kamailio"
Normal Pulled 5m39s kubelet Successfully pulled image "cycoresystems/kamailio" in 1m45.372032898s
Normal Created 5m38s kubelet Created container kamailio
Normal Started 5m38s kubelet Started container kamailio
Normal Pulling 5m38s kubelet Pulling image "cycoresystems/rtpproxy"
Normal Pulled 5m22s kubelet Successfully pulled image "cycoresystems/rtpproxy" in 16.111961198s
Normal Created 5m22s kubelet Created container rtpproxy
Normal Started 5m22s kubelet Started container rtpproxy
Normal Pulled 5m7s kubelet Successfully pulled image "cycoresystems/dispatchers" in 15.121616868s
Normal Pulled 5m2s kubelet Successfully pulled image "cycoresystems/dispatchers" in 3.614521805s
Warning Failed 4m44s (x3 over 5m6s) kubelet Error: failed to start container "dispatchers": Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: exec: "/app": stat /app: no such file or directory: unknown
Normal Pulled 4m44s kubelet Successfully pulled image "cycoresystems/dispatchers" in 4.063114355s
Normal Pulling 4m21s (x4 over 5m22s) kubelet Pulling image "cycoresystems/dispatchers"
Normal Created 4m13s (x4 over 5m6s) kubelet Created container dispatchers
Normal Pulled 4m13s kubelet Successfully pulled image "cycoresystems/dispatchers" in 8.398882695s
Warning BackOff 2m18s (x12 over 4m44s) kubelet Back-off restarting failed container
Just change the command path for 'dispatchers' daemon image in '02-kamailio.yaml' file to '/go/bin/dispatchers':
image: cycoresystems/dispatchers
command:
- /go/bin/dispatchers
- "-set"
- voip:asterisk=1:5080
Thank you Sergei! You rock!! All pods are up and running!
I am seeing a couple errors in the container logs, wondering if you have any more magic bullets you can softly lob my way?
- kamailio/kamailio: "ERROR: rtpproxy [rtpproxy.c:1511]: send_rtpp_command(): proxy <udp:localhost:7722> does not respond, disable it"
- kamailio/dispatchers: "Failed to watch *v1.EndpointSlice"
- asterisk/asterisk: loading errors, but it comes up Ready.
- asterisk/natsgw: "failed to connect to Asterisk"
The only difference from the commands and editing that I enumerated above was your suggested change in 02-kamailio.yaml, and applying 00-rbac.yaml before applying 01-nats.yaml.
Appreciate your time, DRE
$ kubectl get all --namespace=voip
NAME READY STATUS RESTARTS AGE
pod/asterisk-758996b56d-pvbqh 3/3 Running 1 13m
pod/kamailio-5lklk 3/3 Running 3 13m
pod/nats-666d56d6b8-nt6nq 1/1 Running 0 13m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/asterisk ClusterIP 10.109.56.128 <none> 5060/UDP 13m
service/kamailio ClusterIP 10.102.58.75 <none> 5060/UDP 13m
service/nats ClusterIP 10.110.36.234 <none> 4222/TCP,6222/TCP,8222/TCP 13m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/kamailio 1 1 1 1 1 type=kamailio 13m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/asterisk 1/1 1 1 13m
deployment.apps/nats 1/1 1 1 13m
NAME DESIRED CURRENT READY AGE
replicaset.apps/asterisk-758996b56d 1 1 1 13m
replicaset.apps/nats-666d56d6b8 1 1 1 13m
$ kubectl logs kamailio-5lklk --namespace=voip kamailio
0(1) INFO: pv [pv_shv.c:60]: shvar_init_locks(): locks array size 16
0(1) INFO: <core> [core/sctp_core.c:74]: sctp_core_check_support(): SCTP API not enabled - if you want to use it, load sctp module
loading modules under config path: /usr/local/lib64/kamailio/modules:/usr/local/lib64/kamailio/modules_k:/usr/lib/x86_64-linux-gnu/kamailio/modules/
Listening on
udp: 192.168.49.2:5060 advertise 71.51.189.81:5060
udp: 192.168.49.2:5080
Aliases:
udp: minikube:5080
udp: minikube:5060
*: 71--51-189-81.chvl.centurylink.net:*
*: 71.51.189.81:*
0(1) WARNING: <core> [core/daemonize.c:596]: mem_lock_pages(): failed to lock the memory pages (disable swap): Cannot allocate memory [12]
0(1) INFO: rr [../outbound/api.h:52]: ob_load_api(): unable to import bind_ob - maybe module is not loaded
0(1) INFO: rr [rr_mod.c:177]: mod_init(): outbound module not available
0(1) INFO: permissions [permissions.c:598]: mod_init(): default allow file (/etc/kamailio/permissions.allow) not found => empty rule set
0(1) INFO: permissions [permissions.c:612]: mod_init(): default deny file (/etc/kamailio/permissions.deny) not found => empty rule set
0(1) INFO: permissions [trusted.c:180]: init_trusted(): db_url parameter of permissions module not set, disabling allow_trusted
0(1) INFO: permissions [address.c:289]: init_addresses(): db_url parameter of permissions module not set, disabling allow_address
0(1) INFO: utils [utils.c:255]: mod_init(): forward functionality disabled 0(1) INFO: utils [utils.c:177]: pres_db_init(): xcap_auth_status function is disabled
0(1) INFO: <core> [main.c:2669]: main(): processes (at least): 22 - shm size: 67108864 - pkg size: 8388608
0(1) INFO: <core> [core/udp_server.c:153]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992
0(1) INFO: <core> [core/udp_server.c:205]: probe_max_receive_buffer(): SO_RCVBUF is finally 212992
0(1) INFO: <core> [core/udp_server.c:153]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992
0(1) INFO: <core> [core/udp_server.c:205]: probe_max_receive_buffer(): SO_RCVBUF is finally 212992
4(29) ERROR: rtpproxy [rtpproxy.c:1476]: send_rtpp_command(): can't send command to a RTP proxy
4(29) ERROR: rtpproxy [rtpproxy.c:1511]: send_rtpp_command(): proxy <udp:localhost:7722> does not respond, disable it
4(29) WARNING: rtpproxy [rtpproxy.c:1368]: rtpp_test(): can't get version of the RTP proxy
4(29) WARNING: rtpproxy [rtpproxy.c:1404]: rtpp_test(): support for RTP proxy <udp:localhost:7722> has been disabled temporarily
1(26) ERROR: rtpproxy [rtpproxy.c:1476]: send_rtpp_command(): can't send command to a RTP proxy
1(26) ERROR: rtpproxy [rtpproxy.c:1511]: send_rtpp_command(): proxy <udp:localhost:7722> does not respond, disable it
1(26) WARNING: rtpproxy [rtpproxy.c:1368]: rtpp_test(): can't get version of the RTP proxy
1(26) WARNING: rtpproxy [rtpproxy.c:1404]: rtpp_test(): support for RTP proxy <udp:localhost:7722> has been disabled temporarily
3(28) ERROR: rtpproxy [rtpproxy.c:1476]: send_rtpp_command(): can't send command to a RTP proxy
3(28) ERROR: rtpproxy [rtpproxy.c:1511]: send_rtpp_command(): proxy <udp:localhost:7722> does not respond, disable it
3(28) WARNING: rtpproxy [rtpproxy.c:1368]: rtpp_test(): can't get version of the RTP proxy
3(28) WARNING: rtpproxy [rtpproxy.c:1404]: rtpp_test(): support for RTP proxy <udp:localhost:7722> has been disabled temporarily
10(35) ERROR: rtpproxy [rtpproxy.c:1476]: send_rtpp_command(): can't send command to a RTP proxy
10(35) ERROR: rtpproxy [rtpproxy.c:1511]: send_rtpp_command(): proxy <udp:localhost:7722> does not respond, disable it
10(35) WARNING: rtpproxy [rtpproxy.c:1368]: rtpp_test(): can't get version of the RTP proxy
10(35) WARNING: rtpproxy [rtpproxy.c:1404]: rtpp_test(): support for RTP proxy <udp:localhost:7722> has been disabled temporarily
12(37) ERROR: rtpproxy [rtpproxy.c:1476]: send_rtpp_command(): can't send command to a RTP proxy
13(38) ERROR: rtpproxy [rtpproxy.c:1476]: send_rtpp_command(): can't send command to a RTP proxy
5(30) ERROR: rtpproxy [rtpproxy.c:1476]: send_rtpp_command(): can't send command to a RTP proxy
5(30) ERROR: rtpproxy [rtpproxy.c:1511]: send_rtpp_command(): proxy <udp:localhost:7722> does not respond, disable it
5(30) WARNING: rtpproxy [rtpproxy.c:1368]: rtpp_test(): can't get version of the RTP proxy
5(30) WARNING: rtpproxy [rtpproxy.c:1404]: rtpp_test(): support for RTP proxy <udp:localhost:7722> has been disabled temporarily
// repeat
18(43) ERROR: rtpproxy [rtpproxy.c:1476]: send_rtpp_command(): can't send command to a RTP proxy
18(43) ERROR: rtpproxy [rtpproxy.c:1511]: send_rtpp_command(): proxy <udp:localhost:7722> does not respond, disable it
18(43) WARNING: rtpproxy [rtpproxy.c:1368]: rtpp_test(): can't get version of the RTP proxy
18(43) WARNING: rtpproxy [rtpproxy.c:1404]: rtpp_test(): support for RTP proxy <udp:localhost:7722> has been disabled temporarily
17(42) ERROR: rtpproxy [rtpproxy.c:1511]: send_rtpp_command(): proxy <udp:localhost:7722> does not respond, disable it
17(42) WARNING: rtpproxy [rtpproxy.c:1368]: rtpp_test(): can't get version of the RTP proxy
17(42) WARNING: rtpproxy [rtpproxy.c:1404]: rtpp_test(): support for RTP proxy <udp:localhost:7722> has been disabled temporarily
$ kubectl logs kamailio-5lklk --namespace=voip rtpproxy
INFO:main:GLOBAL: rtpproxy started, pid 1
$ kubectl logs kamailio-5lklk --namespace=voip dispatchers
// lots of these "Failed to watch *v1.EndpointSlice"
E0825 08:50:44.664650 1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:serviceaccount:voip:default" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
2021/08/25 08:50:58 current sets:
2021/08/25 08:50:58 set 1: []
$ kubectl logs asterisk-758996b56d-pvbqh --namespace=voip asterisk
Unable to set high priority
[0mXSLT support not found. XML documentation may be incomplete.
XSLT support not found. XML documentation may be incomplete.
PBX UUID: 3c6b7ce2-dc6c-4c38-923a-f53f3c2130cc
Unable to load config file 'stasis.conf'
Could not load Stasis configuration; using defaults
[Aug 25 08:42:12] [1;33mNOTICE[0m[1]: [1;37mloader.c[0m:[1;37m2377[0m [1;37mload_modules[0m: 113 modules will be loaded.
[Aug 25 08:42:12] [1;31mWARNING[0m[1]: [1;37mccss.c[0m:[1;37m4386[0m [1;37minitialize_cc_max_requests[0m: Could not find valid ccss.conf file. Using cc_max_requests default
[Aug 25 08:42:12] [1;31mWARNING[0m[1]: [1;37mccss.c[0m:[1;37m4443[0m [1;37minitialize_cc_devstate_map[0m: Could not find valid ccss.conf file. Using cc_[state]_devstate defaults
[Aug 25 08:42:12] [1;33mNOTICE[0m[1]: [1;37mcdr.c[0m:[1;37m4517[0m [1;37mcdr_toggle_runtime_options[0m: CDR simple logging enabled.
[Aug 25 08:42:12] [1;31mERROR[0m[1]: [1;37mconfig_options.c[0m:[1;37m710[0m [1;37maco_process_config[0m: Unable to load config file 'cel.conf'
[Aug 25 08:42:12] [1;33mNOTICE[0m[1]: [1;37mcel.c[0m:[1;37m1636[0m [1;37mload_module[0m: Failed to process CEL configuration; using defaults
[Aug 25 08:42:12] [1;31mERROR[0m[1]: [1;37mconfig_options.c[0m:[1;37m710[0m [1;37maco_process_config[0m: Unable to load config file 'features.conf'
[Aug 25 08:42:12] [1;33mNOTICE[0m[1]: [1;37mfeatures_config.c[0m:[1;37m1874[0m [1;37mload_config[0m: Could not load features config; using defaults
[Aug 25 08:42:12] [1;31mERROR[0m[1]: [1;37mconfig_options.c[0m:[1;37m710[0m [1;37maco_process_config[0m: Unable to load config file 'acl.conf'
[Aug 25 08:42:12] [1;31mERROR[0m[1]: [1;37mconfig_options.c[0m:[1;37m710[0m [1;37maco_process_config[0m: Unable to load config file 'udptl.conf'
[Aug 25 08:42:12] [1;33mNOTICE[0m[1]: [1;37mudptl.c[0m:[1;37m1338[0m [1;37m__ast_udptl_reload[0m: Could not load udptl config; using defaults
[Aug 25 08:42:12] [1;31mERROR[0m[1]: [1;37mres_sorcery_config.c[0m:[1;37m320[0m [1;37msorcery_config_internal_load[0m: Unable to load config file 'pjproject.conf'
[Aug 25 08:42:12] [1;31mERROR[0m[1]: [1;37mconfig.c[0m:[1;37m1933[0m [1;37mprocess_text_line[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:12] [1;31mERROR[0m[1]: [1;37mres_sorcery_config.c[0m:[1;37m326[0m [1;37msorcery_config_internal_load[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mconfig.c[0m:[1;37m1933[0m [1;37mprocess_text_line[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mres_sorcery_config.c[0m:[1;37m326[0m [1;37msorcery_config_internal_load[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mconfig.c[0m:[1;37m1933[0m [1;37mprocess_text_line[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mres_sorcery_config.c[0m:[1;37m326[0m [1;37msorcery_config_internal_load[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mconfig.c[0m:[1;37m1933[0m [1;37mprocess_text_line[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mres_sorcery_config.c[0m:[1;37m326[0m [1;37msorcery_config_internal_load[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mconfig.c[0m:[1;37m1933[0m [1;37mprocess_text_line[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mres_sorcery_config.c[0m:[1;37m326[0m [1;37msorcery_config_internal_load[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mconfig.c[0m:[1;37m1933[0m [1;37mprocess_text_line[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mres_sorcery_config.c[0m:[1;37m326[0m [1;37msorcery_config_internal_load[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mconfig.c[0m:[1;37m1933[0m [1;37mprocess_text_line[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mres_sorcery_config.c[0m:[1;37m326[0m [1;37msorcery_config_internal_load[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mconfig.c[0m:[1;37m1933[0m [1;37mprocess_text_line[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mres_sorcery_config.c[0m:[1;37m326[0m [1;37msorcery_config_internal_load[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mconfig.c[0m:[1;37m1933[0m [1;37mprocess_text_line[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mres_sorcery_config.c[0m:[1;37m326[0m [1;37msorcery_config_internal_load[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mconfig.c[0m:[1;37m1933[0m [1;37mprocess_text_line[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mres_sorcery_config.c[0m:[1;37m326[0m [1;37msorcery_config_internal_load[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mconfig.c[0m:[1;37m1933[0m [1;37mprocess_text_line[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mres_sorcery_config.c[0m:[1;37m326[0m [1;37msorcery_config_internal_load[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mconfig.c[0m:[1;37m1933[0m [1;37mprocess_text_line[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mres_sorcery_config.c[0m:[1;37m326[0m [1;37msorcery_config_internal_load[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mconfig.c[0m:[1;37m1933[0m [1;37mprocess_text_line[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mres_sorcery_config.c[0m:[1;37m326[0m [1;37msorcery_config_internal_load[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mconfig.c[0m:[1;37m1933[0m [1;37mprocess_text_line[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mres_sorcery_config.c[0m:[1;37m326[0m [1;37msorcery_config_internal_load[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mconfig.c[0m:[1;37m1933[0m [1;37mprocess_text_line[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mres_sorcery_config.c[0m:[1;37m326[0m [1;37msorcery_config_internal_load[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mconfig.c[0m:[1;37m1933[0m [1;37mprocess_text_line[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mres_sorcery_config.c[0m:[1;37m326[0m [1;37msorcery_config_internal_load[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mconfig.c[0m:[1;37m1933[0m [1;37mprocess_text_line[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mres_sorcery_config.c[0m:[1;37m326[0m [1;37msorcery_config_internal_load[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mconfig.c[0m:[1;37m1933[0m [1;37mprocess_text_line[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mres_sorcery_config.c[0m:[1;37m326[0m [1;37msorcery_config_internal_load[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mconfig.c[0m:[1;37m1933[0m [1;37mprocess_text_line[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mres_sorcery_config.c[0m:[1;37m326[0m [1;37msorcery_config_internal_load[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mconfig_options.c[0m:[1;37m710[0m [1;37maco_process_config[0m: Unable to load config file 'pjsip_notify.conf'
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mconfig.c[0m:[1;37m1933[0m [1;37mprocess_text_line[0m: The file 'pjsip_custom.conf' was listed as a #include but it does not exist.
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mres_sorcery_config.c[0m:[1;37m326[0m [1;37msorcery_config_internal_load[0m: Contents of config file 'pjsip.conf' are invalid and cannot be parsed
[Aug 25 08:42:13] [1;33mNOTICE[0m[1]: [1;37mconfbridge/conf_config_parser.c[0m:[1;37m2304[0m [1;37mverify_default_profiles[0m: Adding default_bridge profile to app_confbridge
[Aug 25 08:42:13] [1;33mNOTICE[0m[1]: [1;37mconfbridge/conf_config_parser.c[0m:[1;37m2315[0m [1;37mverify_default_profiles[0m: Adding default_user profile to app_confbridge
[Aug 25 08:42:13] [1;33mNOTICE[0m[1]: [1;37mconfbridge/conf_config_parser.c[0m:[1;37m2326[0m [1;37mverify_default_profiles[0m: Adding default_menu menu to app_confbridge
[Aug 25 08:42:13] [1;31mWARNING[0m[1]: [1;37mapp_voicemail.c[0m:[1;37m14597[0m [1;37mactual_load_config[0m: Failed to load configuration file.
[Aug 25 08:42:13] [1;31mWARNING[0m[1]: [1;37mloader.c[0m:[1;37m2381[0m [1;37mload_modules[0m: Some non-required modules failed to load.
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mloader.c[0m:[1;37m2396[0m [1;37mload_modules[0m: Error loading module 'res_pjsip_registrar_expire.so': /usr/lib/asterisk/modules/res_pjsip_registrar_expire.so: cannot open shared object file: No such file or directory
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mloader.c[0m:[1;37m2396[0m [1;37mload_modules[0m: Failed to resolve dependencies for res_pjsip_transport_websocket
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mloader.c[0m:[1;37m2396[0m [1;37mload_modules[0m: res_pjsip_transport_websocket declined to load.
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mloader.c[0m:[1;37m2396[0m [1;37mload_modules[0m: res_pjsip_notify declined to load.
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mloader.c[0m:[1;37m2396[0m [1;37mload_modules[0m: Failed to resolve dependencies for res_ari_events
[Aug 25 08:42:13] [1;31mERROR[0m[1]: [1;37mloader.c[0m:[1;37m2396[0m [1;37mload_modules[0m: res_ari_events declined to load.
[Aug 25 08:42:13] [1;31mWARNING[0m[1]: [1;37mconfig.c[0m:[1;37m2020[0m [1;37mprocess_text_line[0m: No '=' (equal sign) in line 2 of /etc/asterisk/cli.conf
[1;32mAsterisk Ready.[0m
$ kubectl logs asterisk-758996b56d-pvbqh --namespace=voip config
2021/08/25 08:43:41 running service
2021/08/25 08:43:41 Waiting for Asterisk to be ready...
2021/08/25 08:43:49 change detected
2021/08/25 08:44:08 change detected
$ kubectl logs asterisk-758996b56d-pvbqh --namespace=voip natsgw
t=2021-08-25T08:42:11+0000 lvl=info msg="starting ari-proxy server" version=5.0.1
t=2021-08-25T08:42:11+0000 lvl=eror msg="failed to connect to Asterisk" error="websocket.Dial ws://localhost:8088/ari/events?app=demo: dial tcp 127.0.0.1:8088: connect: connection refused"
t=2021-08-25T08:42:12+0000 lvl=eror msg="failed to connect to Asterisk" error="websocket.Dial ws://localhost:8088/ari/events?app=demo: dial tcp 127.0.0.1:8088: connect: connection refused"
t=2021-08-25T08:42:13+0000 lvl=eror msg="failed to connect to Asterisk" error="websocket.Dial ws://localhost:8088/ari/events?app=demo: bad status"
///these every second
t=2021-08-25T08:52:13+0000 lvl=eror msg="failed to connect to Asterisk" error="websocket.Dial ws://localhost:8088/ari/events?app=demo: bad status"
Hi, I've been stuck on one problem for a month... I hope someone can help me overcome this obstacle. I'm running k8s on a Centos 8 VM. Everything is Running with the exception of the kamailio pod, which is in CrashLoopBackOff.
Results:
``[root@local live-demo]# kubectl get all --namespace=voip NAME READY STATUS RESTARTS AGE pod/asterisk-54d79f8659-gg5lp 3/3 Running 6 (9m7s ago) 155m pod/kamailio-2lqwb 2/3 CrashLoopBackOff 6 (115s ago) 7m59s pod/nats-d98bbdbf-ltng2 1/1 Running 0 22h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/asterisk ClusterIP 10.233.7.7
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/kamailio 1 1 0 1 0 kubernetes.io/hostname=node2 7m59s
NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/asterisk 1/1 1 1 155m deployment.apps/nats 1/1 1 1 22h
NAME DESIRED CURRENT READY AGE replicaset.apps/asterisk-54d79f8659 1 1 1 155m replicaset.apps/nats-d98bbdbf 1 1 1 22h
[root@local live-demo]# kubectl logs kamailio-2lqwb --namespace=voip kamailio
0(1) CRITICAL:
[root@local live-demo]# kubectl describe pod kamailio --namespace=voip
Name: kamailio-2lqwb
Namespace: voip
Priority: 0
Service Account: default
Node: node2/192.168.100.32
Start Time: Thu, 30 Nov 2023 17:37:08 +0500
Labels: component=kamailio
controller-revision-hash=565bc64649
pod-template-generation=1
Annotations:
Normal Scheduled 10m default-scheduler Successfully assigned voip/kamailio-2lqwb to node2 Normal Pulled 10m kubelet Successfully pulled image "cycoresystems/kamailio" in 2.025s (2.025s including waiting) Normal Pulling 10m kubelet Pulling image "cycoresystems/rtpproxy" Normal Created 10m kubelet Created container rtpproxy Normal Pulling 10m kubelet Pulling image "cycoresystems/dispatchers" Normal Started 10m kubelet Started container rtpproxy Normal Pulled 10m kubelet Successfully pulled image "cycoresystems/rtpproxy" in 1.797s (1.797s including waiting) Normal Started 10m kubelet Started container dispatchers Normal Pulled 10m kubelet Successfully pulled image "cycoresystems/dispatchers" in 1.883s (1.883s including waiting) Normal Created 10m kubelet Created container dispatchers Normal Pulled 10m kubelet Successfully pulled image "cycoresystems/kamailio" in 1.807s (1.807s including waiting) Normal Pulled 10m kubelet Successfully pulled image "cycoresystems/kamailio" in 1.828s (1.828s including waiting) Normal Started 10m (x3 over 10m) kubelet Started container kamailio Normal Created 10m (x3 over 10m) kubelet Created container kamailio Normal Pulling 9m45s (x4 over 10m) kubelet Pulling image "cycoresystems/kamailio" Warning BackOff 36s (x47 over 10m) kubelet Back-off restarting failed container kamailio in pod kamailio-2lqwb_voip(9a684224-d8dc-4562-8659-1ebe70e5fd3c)