ceph-helm
ceph-helm copied to clipboard
ceph
ceph-mgr and ceph-osd is not starting
Is this a request for help?: Yes
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT
Version of Helm and Kubernetes:
$ helm version
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:37:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:30:26Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Which chart: ceph
What happened: ceph-mgr and ceph-osd won't start up ouput of ceph-mgr:
+ source variables_entrypoint.sh
++ ALL_SCENARIOS='osd osd_directory osd_directory_single osd_ceph_disk osd_ceph_disk_prepare osd_ceph_disk_activate osd_ceph_activate_journal mgr'
++ : ceph
++ : ceph-config/ceph
++ :
++ :
++ : 0
++ : dockerblade-slot6-oben.example.com
++ : dockerblade-slot6-oben.example.com
++ : /etc/ceph/monmap-ceph
++ : /var/lib/ceph/mon/ceph-dockerblade-slot6-oben.example.com
++ : 0
++ : 0
++ : mds-dockerblade-slot6-oben.example.com
++ : 0
++ : 100
++ : 0
++ : 0
+++ uuidgen
++ : 5700ffd2-02f6-4212-8a76-8a57f3fe2a04
+++ uuidgen
++ : 38e7aef4-c42b-457a-af33-fa8dc3ff1eb7
++ : root=default host=dockerblade-slot6-oben.example.com
++ : 0
++ : cephfs
++ : cephfs_data
++ : 8
++ : cephfs_metadata
++ : 8
++ : dockerblade-slot6-oben.example.com
++ :
++ :
++ : 8080
++ : 0
++ : 9000
++ : 0.0.0.0
++ : cephnfs
++ : dockerblade-slot6-oben.example.com
++ : 0.0.0.0
++ CLI_OPTS='--cluster ceph'
++ DAEMON_OPTS='--cluster ceph --setuser ceph --setgroup ceph -d'
++ MOUNT_OPTS='-t xfs -o noatime,inode64'
++ MDS_KEYRING=/var/lib/ceph/mds/ceph-mds-dockerblade-slot6-oben.example.com/keyring
++ ADMIN_KEYRING=/etc/ceph/ceph.client.admin.keyring
++ MON_KEYRING=/etc/ceph/ceph.mon.keyring
++ RGW_KEYRING=/var/lib/ceph/radosgw/dockerblade-slot6-oben.example.com/keyring
++ MGR_KEYRING=/var/lib/ceph/mgr/ceph-dockerblade-slot6-oben.example.com/keyring
++ MDS_BOOTSTRAP_KEYRING=/var/lib/ceph/bootstrap-mds/ceph.keyring
++ RGW_BOOTSTRAP_KEYRING=/var/lib/ceph/bootstrap-rgw/ceph.keyring
++ OSD_BOOTSTRAP_KEYRING=/var/lib/ceph/bootstrap-osd/ceph.keyring
++ OSD_PATH_BASE=/var/lib/ceph/osd/ceph
+ source common_functions.sh
++ set -ex
+ [[ ! -e /usr/bin/ceph-mgr ]]
+ [[ ! -e /etc/ceph/ceph.conf ]]
+ '[' 0 -eq 1 ']'
+ '[' '!' -e /var/lib/ceph/mgr/ceph-dockerblade-slot6-oben.example.com/keyring ']'
+ timeout 10 ceph --cluster ceph auth get-or-create mgr.dockerblade-slot6-oben.example.com mon 'allow profile mgr' osd 'allow *' mds 'allow *' -o /var/lib/ceph/mgr/ceph-dockerblade-slot6-oben.example.com/keyring
and the output of ceph-osd's osd-prepare-pod:
+ export LC_ALL=C
+ LC_ALL=C
+ source variables_entrypoint.sh
++ ALL_SCENARIOS='osd osd_directory osd_directory_single osd_ceph_disk osd_ceph_disk_prepare osd_ceph_disk_activate osd_ceph_activate_journal mgr'
++ : ceph
++ : ceph-config/ceph
++ :
++ : osd_ceph_disk_prepare
++ : 1
++ : dockerblade-slot5-unten
++ : dockerblade-slot5-unten
++ : /etc/ceph/monmap-ceph
++ : /var/lib/ceph/mon/ceph-dockerblade-slot5-unten
++ : 0
++ : 0
++ : mds-dockerblade-slot5-unten
++ : 0
++ : 100
++ : 0
++ : 0
+++ uuidgen
++ : e101933b-67b3-4267-824f-173d2ef7a47b
+++ uuidgen
++ : 10dd57d2-f3c7-4cab-88ea-8e3771baeaa7
++ : root=default host=dockerblade-slot5-unten
++ : 0
++ : cephfs
++ : cephfs_data
++ : 8
++ : cephfs_metadata
++ : 8
++ : dockerblade-slot5-unten
++ :
++ :
++ : 8080
++ : 0
++ : 9000
++ : 0.0.0.0
++ : cephnfs
++ : dockerblade-slot5-unten
++ : 0.0.0.0
++ CLI_OPTS='--cluster ceph'
++ DAEMON_OPTS='--cluster ceph --setuser ceph --setgroup ceph -d'
++ MOUNT_OPTS='-t xfs -o noatime,inode64'
++ MDS_KEYRING=/var/lib/ceph/mds/ceph-mds-dockerblade-slot5-unten/keyring
++ ADMIN_KEYRING=/etc/ceph/ceph.client.admin.keyring
++ MON_KEYRING=/etc/ceph/ceph.mon.keyring
++ RGW_KEYRING=/var/lib/ceph/radosgw/dockerblade-slot5-unten/keyring
++ MGR_KEYRING=/var/lib/ceph/mgr/ceph-dockerblade-slot5-unten/keyring
++ MDS_BOOTSTRAP_KEYRING=/var/lib/ceph/bootstrap-mds/ceph.keyring
++ RGW_BOOTSTRAP_KEYRING=/var/lib/ceph/bootstrap-rgw/ceph.keyring
++ OSD_BOOTSTRAP_KEYRING=/var/lib/ceph/bootstrap-osd/ceph.keyring
++ OSD_PATH_BASE=/var/lib/ceph/osd/ceph
+ source common_functions.sh
++ set -ex
+ is_available rpm
+ command -v rpm
+ is_available dpkg
+ command -v dpkg
+ OS_VENDOR=ubuntu
+ source /etc/default/ceph
++ TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=134217728
+ case "$CEPH_DAEMON" in
+ OSD_TYPE=prepare
+ start_osd
+ [[ ! -e /etc/ceph/ceph.conf ]]
+ '[' 1 -eq 1 ']'
+ [[ ! -e /etc/ceph/ceph.client.admin.keyring ]]
+ case "$OSD_TYPE" in
+ source osd_disk_prepare.sh
++ set -ex
+ osd_disk_prepare
+ [[ -z /dev/container/block-data ]]
+ [[ ! -e /dev/container/block-data ]]
+ '[' '!' -e /var/lib/ceph/bootstrap-osd/ceph.keyring ']'
+ timeout 10 ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring health
+ exit 1
What you expected to happen: to start up flawlessly
Anything else we need to know: here is my overrides.yml:
network:
public: 10.42.0.0/16
cluster: 10.42.0.0/16
osd_devices:
- name: block-data
device: /dev/container/block-data
zap: "1"
storageclass:
name: ceph-rbd
pool: rbd
user_id: k8s
I am using Rancher 2 / RKE on bare-metal. I am unsure about the network-setup. Maybe i have some issues here:
- All nodes (6) can see and reach each other by IPv4 address only. Although the nodes have names there is no DNS set up outside of the cluster
- Rancher/RKE sets up a flannel-network with CIDR 10.42.0.0/16 which is what i used as
network.publicandnetwork.cluster
Hi :) did you managed to overcome this issue? I'm experiencing the same problem
