postgres-operator-examples
postgres-operator-examples copied to clipboard
CrunchyData
Unable to mount the PVC with nfs storage
I am trying to use the NFS storage for the pod for that we have updated the Values.yaml file with the following code
instances:
- name: instances1
replicas: 1
dataVolumeClaimSpec:
storageClassName: "nfs"
accessModes:
- "ReadWriteMany"
resources:
requests:
storage: 1Gi
But we are getting an error MountVolume.SetUp failed for volum description of the pod as follow:
Name: abc-instances1-qzrp-0
Namespace: default
Priority: 0
Node: ravi-vostro-15-3568/192.168.10.217
Start Time: Wed, 06 Jul 2022 13:08:29 +0530
Labels: controller-revision-hash=abc-instances1-qzrp-6f99d54f99
postgres-operator.crunchydata.com/cluster=abc
postgres-operator.crunchydata.com/data=postgres
postgres-operator.crunchydata.com/instance=abc-instances1-qzrp
postgres-operator.crunchydata.com/instance-set=instances1
postgres-operator.crunchydata.com/patroni=abc-ha
statefulset.kubernetes.io/pod-name=abc-instances1-qzrp-0
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: StatefulSet/abc-instances1-qzrp
Init Containers:
postgres-startup:
Container ID:
Image: registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.3-0
Image ID:
Port: <none>
Host Port: <none>
Command:
bash
-ceu
--
declare -r expected_major_version="$1" pgwal_directory="$2" pgbrLog_directory="$3"
results() { printf '::postgres-operator: %s::%s\n' "$@"; }
safelink() (
local desired="$1" name="$2" current
current=$(realpath "${name}")
if [ "${current}" = "${desired}" ]; then return; fi
set -x; mv --no-target-directory "${current}" "${desired}"
ln --no-dereference --force --symbolic "${desired}" "${name}"
)
echo Initializing ...
results 'uid' "$(id -u)" 'gid' "$(id -G)"
results 'postgres path' "$(command -v postgres)"
results 'postgres version' "${postgres_version:=$(postgres --version)}"
[[ "${postgres_version}" == *") ${expected_major_version}."* ]]
results 'config directory' "${PGDATA:?}"
postgres_data_directory=$([ -d "${PGDATA}" ] && postgres -C data_directory || echo "${PGDATA}")
results 'data directory' "${postgres_data_directory}"
[ "${postgres_data_directory}" = "${PGDATA}" ]
bootstrap_dir="${postgres_data_directory}_bootstrap"
[ -d "${bootstrap_dir}" ] && results 'bootstrap directory' "${bootstrap_dir}"
[ -d "${bootstrap_dir}" ] && postgres_data_directory="${bootstrap_dir}"
install --directory --mode=0700 "${postgres_data_directory}"
results 'pgBackRest log directory' "${pgbrLog_directory}"
install --directory --mode=0775 "${pgbrLog_directory}"
install -D --mode=0600 -t "/tmp/replication" "/pgconf/tls/replication"/{tls.crt,tls.key,ca.crt}
[ -f "${postgres_data_directory}/PG_VERSION" ] || exit 0
results 'data version' "${postgres_data_version:=$(< "${postgres_data_directory}/PG_VERSION")}"
[ "${postgres_data_version}" = "${expected_major_version}" ]
safelink "${pgwal_directory}" "${postgres_data_directory}/pg_wal"
results 'wal directory' "$(realpath "${postgres_data_directory}/pg_wal")"
rm -f "${postgres_data_directory}/recovery.signal"
startup
14
/pgdata/pg14_wal
/pgdata/pgbackrest/log
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
PGDATA: /pgdata/pg14
PGHOST: /tmp/postgres
PGPORT: 5432
KRB5_CONFIG: /etc/postgres/krb5.conf
KRB5RCACHEDIR: /tmp
Mounts:
/pgconf/tls from cert-volume (ro)
/pgdata from postgres-data (rw)
/tmp from tmp (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nwqpk (ro)
nss-wrapper-init:
Container ID:
Image: registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.3-0
Image ID:
Port: <none>
Host Port: <none>
Command:
bash
-c
export NSS_WRAPPER_SUBDIR=postgres CRUNCHY_NSS_USERNAME=postgres CRUNCHY_NSS_USER_DESC="postgres"
# Define nss_wrapper directory and passwd & group files that will be utilized by nss_wrapper. The
# nss_wrapper_env.sh script (which also sets these vars) isn't sourced here since the nss_wrapper
# has not yet been setup, and we therefore don't yet want the nss_wrapper vars in the environment.
mkdir -p /tmp/nss_wrapper
chmod g+rwx /tmp/nss_wrapper
NSS_WRAPPER_DIR="/tmp/nss_wrapper/${NSS_WRAPPER_SUBDIR}"
NSS_WRAPPER_PASSWD="${NSS_WRAPPER_DIR}/passwd"
NSS_WRAPPER_GROUP="${NSS_WRAPPER_DIR}/group"
# create the nss_wrapper directory
mkdir -p "${NSS_WRAPPER_DIR}"
# grab the current user ID and group ID
USER_ID=$(id -u)
export USER_ID
GROUP_ID=$(id -g)
export GROUP_ID
# get copies of the passwd and group files
[[ -f "${NSS_WRAPPER_PASSWD}" ]] || cp "/etc/passwd" "${NSS_WRAPPER_PASSWD}"
[[ -f "${NSS_WRAPPER_GROUP}" ]] || cp "/etc/group" "${NSS_WRAPPER_GROUP}"
# if the username is missing from the passwd file, then add it
if [[ ! $(cat "${NSS_WRAPPER_PASSWD}") =~ ${CRUNCHY_NSS_USERNAME}:x:${USER_ID} ]]; then
echo "nss_wrapper: adding user"
passwd_tmp="${NSS_WRAPPER_DIR}/passwd_tmp"
cp "${NSS_WRAPPER_PASSWD}" "${passwd_tmp}"
sed -i "/${CRUNCHY_NSS_USERNAME}:x:/d" "${passwd_tmp}"
# needed for OCP 4.x because crio updates /etc/passwd with an entry for USER_ID
sed -i "/${USER_ID}:x:/d" "${passwd_tmp}"
printf '${CRUNCHY_NSS_USERNAME}:x:${USER_ID}:${GROUP_ID}:${CRUNCHY_NSS_USER_DESC}:${HOME}:/bin/bash\n' >> "${passwd_tmp}"
envsubst < "${passwd_tmp}" > "${NSS_WRAPPER_PASSWD}"
rm "${passwd_tmp}"
else
echo "nss_wrapper: user exists"
fi
# if the username (which will be the same as the group name) is missing from group file, then add it
if [[ ! $(cat "${NSS_WRAPPER_GROUP}") =~ ${CRUNCHY_NSS_USERNAME}:x:${USER_ID} ]]; then
echo "nss_wrapper: adding group"
group_tmp="${NSS_WRAPPER_DIR}/group_tmp"
cp "${NSS_WRAPPER_GROUP}" "${group_tmp}"
sed -i "/${CRUNCHY_NSS_USERNAME}:x:/d" "${group_tmp}"
printf '${CRUNCHY_NSS_USERNAME}:x:${USER_ID}:${CRUNCHY_NSS_USERNAME}\n' >> "${group_tmp}"
envsubst < "${group_tmp}" > "${NSS_WRAPPER_GROUP}"
rm "${group_tmp}"
else
echo "nss_wrapper: group exists"
fi
# export the nss_wrapper env vars
# define nss_wrapper directory and passwd & group files that will be utilized by nss_wrapper
NSS_WRAPPER_DIR="/tmp/nss_wrapper/${NSS_WRAPPER_SUBDIR}"
NSS_WRAPPER_PASSWD="${NSS_WRAPPER_DIR}/passwd"
NSS_WRAPPER_GROUP="${NSS_WRAPPER_DIR}/group"
export LD_PRELOAD=/usr/lib64/libnss_wrapper.so
export NSS_WRAPPER_PASSWD="${NSS_WRAPPER_PASSWD}"
export NSS_WRAPPER_GROUP="${NSS_WRAPPER_GROUP}"
echo "nss_wrapper: environment configured"
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/tmp from tmp (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nwqpk (ro)
Containers:
database:
Container ID:
Image: registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.3-0
Image ID:
Port: 5432/TCP
Host Port: 0/TCP
Command:
patroni
/etc/patroni
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Liveness: http-get https://:8008/liveness delay=3s timeout=5s period=10s #success=1 #failure=3
Readiness: http-get https://:8008/readiness delay=3s timeout=5s period=10s #success=1 #failure=3
Environment:
PGDATA: /pgdata/pg14
PGHOST: /tmp/postgres
PGPORT: 5432
KRB5_CONFIG: /etc/postgres/krb5.conf
KRB5RCACHEDIR: /tmp
PATRONI_NAME: abc-instances1-qzrp-0 (v1:metadata.name)
PATRONI_KUBERNETES_POD_IP: (v1:status.podIP)
PATRONI_KUBERNETES_PORTS: - name: postgres
port: 5432
protocol: TCP
PATRONI_POSTGRESQL_CONNECT_ADDRESS: $(PATRONI_NAME).abc-pods:5432
PATRONI_POSTGRESQL_LISTEN: *:5432
PATRONI_POSTGRESQL_CONFIG_DIR: /pgdata/pg14
PATRONI_POSTGRESQL_DATA_DIR: /pgdata/pg14
PATRONI_RESTAPI_CONNECT_ADDRESS: $(PATRONI_NAME).abc-pods:8008
PATRONI_RESTAPI_LISTEN: *:8008
PATRONICTL_CONFIG_FILE: /etc/patroni
LD_PRELOAD: /usr/lib64/libnss_wrapper.so
NSS_WRAPPER_PASSWD: /tmp/nss_wrapper/postgres/passwd
NSS_WRAPPER_GROUP: /tmp/nss_wrapper/postgres/group
Mounts:
/dev/shm from dshm (rw)
/etc/database-containerinfo from database-containerinfo (ro)
/etc/patroni from patroni-config (ro)
/etc/pgbackrest/conf.d from pgbackrest-config (ro)
/pgconf/tls from cert-volume (ro)
/pgdata from postgres-data (rw)
/tmp from tmp (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nwqpk (ro)
replication-cert-copy:
Container ID:
Image: registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.3-0
Image ID:
Port: <none>
Host Port: <none>
Command:
bash
-ceu
--
monitor() {
declare -r directory="/pgconf/tls"
exec {fd}<> <(:)
while read -r -t 5 -u "${fd}" || true; do
if [ "${directory}" -nt "/proc/self/fd/${fd}" ] &&
install -D --mode=0600 -t "/tmp/replication" "${directory}"/{replication/tls.crt,replication/tls.key,replication/ca.crt} &&
pkill -HUP --exact --parent=1 postgres
then
exec {fd}>&- && exec {fd}<> <(:)
stat --format='Loaded certificates dated %y' "${directory}"
fi
done
}; export -f monitor; exec -a "$0" bash -ceu monitor
replication-cert-copy
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/pgconf/tls from cert-volume (ro)
/tmp from tmp (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nwqpk (ro)
pgbackrest:
Container ID:
Image: registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest:ubi8-2.38-1
Image ID:
Port: <none>
Host Port: <none>
Command:
pgbackrest
server
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Liveness: exec [pgbackrest server-ping] delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
LD_PRELOAD: /usr/lib64/libnss_wrapper.so
NSS_WRAPPER_PASSWD: /tmp/nss_wrapper/postgres/passwd
NSS_WRAPPER_GROUP: /tmp/nss_wrapper/postgres/group
Mounts:
/etc/pgbackrest/conf.d from pgbackrest-config (ro)
/etc/pgbackrest/server from pgbackrest-server (ro)
/pgdata from postgres-data (rw)
/tmp from tmp (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nwqpk (ro)
pgbackrest-config:
Container ID:
Image: registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest:ubi8-2.38-1
Image ID:
Port: <none>
Host Port: <none>
Command:
bash
-ceu
--
monitor() {
exec {fd}<> <(:)
until read -r -t 5 -u "${fd}"; do
if
[ "${filename}" -nt "/proc/self/fd/${fd}" ] &&
pkill -HUP --exact --parent=0 pgbackrest
then
exec {fd}>&- && exec {fd}<> <(:)
stat --dereference --format='Loaded configuration dated %y' "${filename}"
elif
{ [ "${directory}" -nt "/proc/self/fd/${fd}" ] ||
[ "${authority}" -nt "/proc/self/fd/${fd}" ]
} &&
pkill -HUP --exact --parent=0 pgbackrest
then
exec {fd}>&- && exec {fd}<> <(:)
stat --format='Loaded certificates dated %y' "${directory}"
fi
done
}; export directory="$1" authority="$2" filename="$3"; export -f monitor; exec -a "$0" bash -ceu monitor
pgbackrest-config
/etc/pgbackrest/server
/etc/pgbackrest/conf.d/~postgres-operator/tls-ca.crt
/etc/pgbackrest/conf.d/~postgres-operator_server.conf
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/etc/pgbackrest/conf.d from pgbackrest-config (ro)
/etc/pgbackrest/server from pgbackrest-server (ro)
/tmp from tmp (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nwqpk (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
cert-volume:
Type: Projected (a volume that contains injected data from multiple sources)
SecretName: abc-cluster-cert
SecretOptionalName: <nil>
SecretName: abc-replication-cert
SecretOptionalName: <nil>
postgres-data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: abc-instances1-qzrp-pgdata
ReadOnly: false
database-containerinfo:
Type: DownwardAPI (a volume populated by information about the pod)
Items:
limits.cpu -> cpu_limit
requests.cpu -> cpu_request
limits.memory -> mem_limit
requests.memory -> mem_request
metadata.labels -> labels
metadata.annotations -> annotations
pgbackrest-server:
Type: Projected (a volume that contains injected data from multiple sources)
SecretName: abc-instances1-qzrp-certs
SecretOptionalName: <nil>
pgbackrest-config:
Type: Projected (a volume that contains injected data from multiple sources)
ConfigMapName: abc-pgbackrest-config
ConfigMapOptional: <nil>
SecretName: abc-pgbackrest
SecretOptionalName: 0xc0015e69f3
patroni-config:
Type: Projected (a volume that contains injected data from multiple sources)
ConfigMapName: abc-config
ConfigMapOptional: <nil>
ConfigMapName: abc-instances1-qzrp-config
ConfigMapOptional: <nil>
SecretName: abc-instances1-qzrp-certs
SecretOptionalName: <nil>
tmp:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: 16Mi
dshm:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
kube-api-access-nwqpk:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 59s default-scheduler 0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
Normal Scheduled 57s default-scheduler Successfully assigned default/abc-instances1-qzrp-0 to ravi-vostro-15-3568
Warning FailedMount 24s (x7 over 56s) kubelet MountVolume.SetUp failed for volume "pvc-2a38d4f9-4a44-49a3-8a34-fb68ea6fed79" : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs -o vers=3 10.43.122.174:/export/pvc-2a38d4f9-4a44-49a3-8a34-fb68ea6fed79 /var/lib/kubelet/pods/c3ab470f-e2e7-48a6-8101-dbf9d5068273/volumes/kubernetes.io~nfs/pvc-2a38d4f9-4a44-49a3-8a34-fb68ea6fed79
Output: mount: /var/lib/kubelet/pods/c3ab470f-e2e7-48a6-8101-dbf9d5068273/volumes/kubernetes.io~nfs/pvc-2a38d4f9-4a44-49a3-8a34-fb68ea6fed79: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program.
(a) Are you still experiencing this problem? If so, what do the pvc and pv look like for a pod created by the postgres-operator?
(b) Also are you able to create a pod with NFS storage in general in this environment? (I'm trying to make sure that this isn't a problem with NFS in general in your env.)
HI @rg2609,
We are closing this issue and hope you have been able to mount the PVC with NFS but If you continue to have issues or have not resolved your issue feel free to create a new issue or re-open this one.
