zalando
[UI] Backups tab: Error loading stored clusters. Please try again or start over.
When opening the "Backup tab" in the operator ui, I get a message "Error loading stored clusters. Please try again or start over." Retrying doesn't help.
The UI makes a request to http://localhost:8081/stored_clusters, which returns 500 Internal Server Error.
Output from kubectl logs deployment/postgres-operator-ui:
--- Logging error ---
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
response = self.full_dispatch_request()
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/operator_ui/main.py", line 190, in wrapper
return f(*args, **kwargs)
File "/operator_ui/main.py", line 741, in get_stored_clusters
prefix=SPILO_S3_BACKUP_PREFIX,
File "/operator_ui/spiloutils.py", line 252, in read_stored_clusters
Prefix=prefix,
File "/usr/lib/python3.6/site-packages/botocore/client.py", line 276, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/lib/python3.6/site-packages/botocore/client.py", line 559, in _make_api_call
api_params, operation_model, context=request_context)
File "/usr/lib/python3.6/site-packages/botocore/client.py", line 605, in _convert_to_request_dict
api_params, operation_model, context)
File "/usr/lib/python3.6/site-packages/botocore/client.py", line 637, in _emit_api_params
params=api_params, model=operation_model, context=context)
File "/usr/lib/python3.6/site-packages/botocore/hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/usr/lib/python3.6/site-packages/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/usr/lib/python3.6/site-packages/botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "/usr/lib/python3.6/site-packages/botocore/handlers.py", line 223, in validate_bucket_name
if not VALID_BUCKET.search(bucket) and not VALID_S3_ARN.search(bucket):
TypeError: expected string or bytes-like object
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/logging/handlers.py", line 936, in emit
self.socket.send(msg)
File "/usr/lib/python3.6/site-packages/gevent/_socket3.py", line 390, in send
return _socket.socket.send(self._sock, data, flags)
OSError: [Errno 9] Bad file descriptor
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/logging/handlers.py", line 857, in _connect_unixsocket
self.socket.connect(address)
File "/usr/lib/python3.6/site-packages/gevent/_socket3.py", line 307, in connect
raise error(result, strerror(result))
FileNotFoundError: [Errno 2] No such file or directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/logging/handlers.py", line 939, in emit
self._connect_unixsocket(self.address)
File "/usr/lib/python3.6/logging/handlers.py", line 868, in _connect_unixsocket
self.socket.connect(address)
File "/usr/lib/python3.6/site-packages/gevent/_socket3.py", line 307, in connect
raise error(result, strerror(result))
FileNotFoundError: [Errno 2] No such file or directory
Call stack:
File "/usr/lib/python3.6/site-packages/gevent/greenlet.py", line 536, in run
result = self._run(*self.args, **self.kwargs)
File "/usr/lib/python3.6/site-packages/gevent/baseserver.py", line 26, in _handle_and_close_when_done
return handle(*args_tuple)
File "/usr/lib/python3.6/site-packages/gevent/pywsgi.py", line 1482, in handle
handler.handle()
File "/usr/lib/python3.6/site-packages/gevent/pywsgi.py", line 454, in handle
result = self.handle_one_request()
File "/usr/lib/python3.6/site-packages/gevent/pywsgi.py", line 671, in handle_one_request
self.handle_one_response()
File "/usr/lib/python3.6/site-packages/gevent/pywsgi.py", line 935, in handle_one_response
self.run_application()
File "/usr/lib/python3.6/site-packages/gevent/pywsgi.py", line 908, in run_application
self.result = self.application(self.environ, self.start_response)
File "/usr/lib/python3.6/site-packages/flask/app.py", line 2463, in __call__
return self.wsgi_app(environ, start_response)
File "/operator_ui/main.py", line 160, in __call__
return self.app(environ, start_response)
File "/usr/lib/python3.6/site-packages/flask/app.py", line 2449, in wsgi_app
response = self.handle_exception(e)
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1870, in handle_exception
self.log_exception((exc_type, exc_value, tb))
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1891, in log_exception
"Exception on %s [%s]" % (request.path, request.method), exc_info=exc_info
Message: 'Exception on /stored_clusters [GET]'
Arguments: ()
127.0.0.1 - - [2020-04-27 16:55:29] "GET /stored_clusters HTTP/1.1" 500 411 2.097737
10.1.6.1 - - [2020-04-27 16:55:32] "GET /health HTTP/1.1" 200 117 0.001161
10.1.6.1 - - [2020-04-27 16:55:42] "GET /health HTTP/1.1" 200 117 0.002137
10.1.6.1 - - [2020-04-27 16:55:52] "GET /health HTTP/1.1" 200 117 0.001463
10.1.6.1 - - [2020-04-27 16:56:02] "GET /health HTTP/1.1" 200 117 0.000974
10.1.6.1 - - [2020-04-27 16:56:12] "GET /health HTTP/1.1" 200 117 0.001322
10.1.6.1 - - [2020-04-27 16:56:22] "GET /health HTTP/1.1" 200 117 0.001323
10.1.6.1 - - [2020-04-27 16:56:32] "GET /health HTTP/1.1" 200 117 0.001397
10.1.6.1 - - [2020-04-27 16:56:42] "GET /health HTTP/1.1" 200 117 0.001307
10.1.6.1 - - [2020-04-27 16:56:52] "GET /health HTTP/1.1" 200 117 0.001155
10.1.6.1 - - [2020-04-27 16:57:02] "GET /health HTTP/1.1" 200 117 0.001214
10.1.6.1 - - [2020-04-27 16:57:12] "GET /health HTTP/1.1" 200 117 0.002490
10.1.6.1 - - [2020-04-27 16:57:22] "GET /health HTTP/1.1" 200 117 0.001992
10.1.6.1 - - [2020-04-27 16:57:32] "GET /health HTTP/1.1" 200 117 0.001392
10.1.6.1 - - [2020-04-27 16:57:42] "GET /health HTTP/1.1" 200 117 0.002429
10.1.6.1 - - [2020-04-27 16:57:52] "GET /health HTTP/1.1" 200 117 0.001851
10.1.6.1 - - [2020-04-27 16:58:02] "GET /health HTTP/1.1" 200 117 0.003601
10.1.6.1 - - [2020-04-27 16:58:12] "GET /health HTTP/1.1" 200 117 0.001118
10.1.6.1 - - [2020-04-27 16:58:22] "GET /health HTTP/1.1" 200 117 0.001185
10.1.6.1 - - [2020-04-27 16:58:32] "GET /health HTTP/1.1" 200 117 0.001606
10.1.6.1 - - [2020-04-27 16:58:42] "GET /health HTTP/1.1" 200 117 0.001739
operator_ui.main ERROR Exception on /stored_clusters [GET]
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
response = self.full_dispatch_request()
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/operator_ui/main.py", line 190, in wrapper
return f(*args, **kwargs)
File "/operator_ui/main.py", line 741, in get_stored_clusters
prefix=SPILO_S3_BACKUP_PREFIX,
File "/operator_ui/spiloutils.py", line 252, in read_stored_clusters
Prefix=prefix,
File "/usr/lib/python3.6/site-packages/botocore/client.py", line 276, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/lib/python3.6/site-packages/botocore/client.py", line 559, in _make_api_call
api_params, operation_model, context=request_context)
File "/usr/lib/python3.6/site-packages/botocore/client.py", line 605, in _convert_to_request_dict
api_params, operation_model, context)
File "/usr/lib/python3.6/site-packages/botocore/client.py", line 637, in _emit_api_params
params=api_params, model=operation_model, context=context)
File "/usr/lib/python3.6/site-packages/botocore/hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/usr/lib/python3.6/site-packages/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/usr/lib/python3.6/site-packages/botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "/usr/lib/python3.6/site-packages/botocore/handlers.py", line 223, in validate_bucket_name
if not VALID_BUCKET.search(bucket) and not VALID_S3_ARN.search(bucket):
TypeError: expected string or bytes-like object
My environment:
# How I created the postgres-operator
kubectl apply -f manifests/configman.yaml
kubectl apply -f manifests/api-service.yaml
kubectl apply -f manifests/operator-service-account-rbac.yaml
kubectl apply -f manifests/postgres-operator.yaml
# How I created the UI:
kubectl apply -f ui/manifests
Everything except backups seem to work.
Do you have backups set up and configured the path in the UI? If not, it's expected to see these errors. I admit that documentation is missing for this part, and unset variables could be handled nicer.
Nope, I didn't change any of the defaults. Some documentation on how to do this would be very nice
I'm also facing the same issue and would really appreciate if someone can help with a workaround. thanks :)
I understood what should be changed. UI tries to load backups from S3 storage and there is no way in helm chart to change backet-name or s3 endpoint / API key I am working on a new version of UI with these additional parameters
@ifelsedeveloper one month ago, I downloaded the chart and I am changing it in my own.
I made it work by :
kubectl -n <ns-operator> edit deployment postgres-operator-ui -o yaml
Then, I went to env:, and i add :
- name: APP_URL
- name: AWS_ACCESS_KEY_ID
value: AKIJKLA4YYYXXXXXXXXX
- name: AWS_SECRET_ACCESS_KEY
value: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
- name: AWS_DEFAULT_REGION
value: ap-southeast-1
- name: SPILO_S3_BACKUP_BUCKET
value: my-bucket-name-where-db-backup
Hopefully, I didn't forget one day to override it by the helm upgrade.
In the meantime , i will open PR to support these env vars with the Helm Chart
However it is not totally work. Another issue appear #1045
Hi all... kubectl logs -f pod/postgres-operator-ui ....Here I am using postgres-operator-ui tag: v1.5.0-dirty helm-chart.
I am getting this error -->
10.150.15.195 - - [2020-08-06 11:04:25] "GET /health HTTP/1.1" 200 117 0.001019
operator_ui.main ERROR Exception on /stored_clusters/acid-cluster-ck/98ed1226-7488-4acd-95a4-bb36997b06 [GET]
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functionsrule.endpoint
File "/operator_ui/main.py", line 193, in wrapper
return f(*args, **kwargs)
File "/operator_ui/main.py", line 802, in get_basebackups
use_aws_instance_profile=USE_AWS_INSTANCE_PROFILE,
File "/operator_ui/spiloutils.py", line 325, in read_basebackups
)._backup_list(detail=True)._backup_list(prefix=f"{prefix}{pg_cluster}{suffix}/wal/")
File "/operator_ui/spiloutils.py", line 316, in
<Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message><Endpoint>s3.amazonaws.com</Endpoint><Bucket>signalprofi</Bucket><RequestId>884C7752816E96CB</RequestId><HostId>kvKh4jvugX6pr3EFL67gZv5UK1+gVyjC5VZ4sylJM8wI14PoTIIYmt91OoIY=</HostId></Error>
any one help me for PostgreSQL operator-ui restoring from aws s3 bucket----->Error loading snapshots. Please try again or start over.
Thank you...
I've bumped into the same issue today, backup page doesn't list anything and there are lots of errors in the UI pod log. I could also try to edit the deployment as in https://github.com/zalando/postgres-operator/issues/937#issuecomment-653781015 but this manual edit is against reproducibility. Also, https://github.com/zalando/postgres-operator/issues/1045#issuecomment-666945550 mentions a different env var to be edited, should it be added next to the other missing env vars? Any update on a possible Helm chart fix to configure the custom S3 parameters?
I'm using self hosted s3 storage (Minio), where can I enter my custom endpoint? The env var WAL_S3_ENDPOINT doesn't work.
I've fixed this by adding the missing fields to the deployment.yaml chart template file.
@FxKu Are you already working on this? And could a solution instead be a shared secret between the operator and the UI for all S3 details?
I'm using self hosted s3 storage (Minio), where can I enter my custom endpoint? The env var
WAL_S3_ENDPOINTdoesn't work.
Got it working with minio as well with AWS_ENDPOINT
- https://github.com/zalando/postgres-operator/blob/master/ui/operator_ui/main.py#L99-L108
--
EDIT: Well, too happy too soon. It is now able to find clusters/versions after defining AWS_ENDPOINT but not the basebackups directly, when expanding into them. I've also had to set WALE_S3_ENDPOINT to get it to stop error-ing but it just shows "No snapshot" after that
Tracing it up to here atm -> https://github.com/zalando/postgres-operator/blob/master/ui/operator_ui/spiloutils.py#L307-L335
@FxKu just wondering, is there a reason for the prefix to be removed from the _backup_list call?
https://github.com/zalando/postgres-operator/commit/636ba9b84624a3552182e5156b1c6d752b213c97#diff-f2093f81d8ffea1536acf97edae2b1522e0729edbba2a370a0ee6bbb4ecd4702
EDIT: Looking into wal.e -- seems like the backup_ctx is supposed to memoize that hmm.
@FxKu I just tested it with the extra _backup_list with the prefix and it brings the snapshots "back" https://github.com/zalando/postgres-operator/compare/master...aranair:aranair.test-prefix
Without it, it would always just show "No snapshots found". ~Do you want me to make a PR of this or are there other concerns with that removal?~
Yet another edit: It seems like after that change, wal.e "sort-of" finds the files but I would see multiple instances of each snapshot shown above^
- I also rolled it back to the previous implementation in v1.5 and it would still display duplicates, so I suspect the issue may have been around(?)
Separately, the frontend is also having issues querying for DOM elements with ID with / in them.
Does anyone have this all set-up, outside of the main team, who knows what's the correct format they should be in?
The error in the UI could be more user friendly.
It'd be nice if UI would check if corresponding env vars are empty & to allow to set them in the UI.
I suppose for GCP/Azure/Oracle/others where WAL-G is being used, take a look at https://github.com/zalando/postgres-operator/issues/1513
I have same issue with Operator UI running on Openshift and Minio S3 storage.
The suggested fix partially works:
- Snapshots are visible
- Data is duplicated
- "Clone at" and "Clone at latest state" buttons don't work.
The UI piece isn't getting a lot of attention, but even if it was great I don't know how useful it'd be. The rest of the project seems very alive to me (and works great).
Hello :)
Any progress on this ? On our side it did not see any snapshots unfortunately.
Best Regards,
dead project apparently. no question everything is configured properly. I can run the backup script from the operator and see all the files in minio, but the UI doesn't show the snapshots. pretty clear there's a bug somewhere that was pointed out by aranair 4 years ago. but no response.
Hello, I was trying to configure the backups and had the same errors, Nevertheless, it was a misconfiguration, and finally, I made it work.
I was testing it in a local environment:
k8s: minikube Backups bucket endpoint: minio version 15.0.6 postgres-operator : 1.14.0 postgres-operator-ui: 1.14.0
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
minio minio 1 2025-03-05 12:31:27.512058257 +0100 CET deployed minio-15.0.6 2025.2.28
postgres-operator postgres-operator 2 2025-03-07 10:10:32.211690634 +0100 CET deployed postgres-operator-1.14.0 1.14.0
postgres-operator-ui postgres-operator 7 2025-03-07 12:36:04.369436022 +0100 CET deployed postgres-operator-ui-1.14.0 1.14.0
I added this configuration to the postgres-operator-ui:
envs:
targetNamespace: "*"
teams:
- "acid"
- "demo"
extraEnvs:
- name: WALE_S3_ENDPOINT
value: http+path://minio.minio.svc:9000
- name: AWS_ENDPOINT
value: http://minio.minio.svc:9000
- name: SPILO_S3_BACKUP_PREFIX
value: spilo/
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: backups-creds-secret
key: AWS_ACCESS_KEY_ID
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: backups-creds-secret
key: AWS_SECRET_ACCESS_KEY
- name: SPILO_S3_BACKUP_BUCKET
value: test
Now the UI Backups tap is working, and I can see the snapshots:
Whether it is useful or not, it is working.
~@kadaffy could you please share pod_environment_configmap and pod_environment_secret?~
UPD:
It seems that backups are only displayed when created using WAL-E. I was able to run it with this configuration (in addition to configuration from https://github.com/zalando/postgres-operator/issues/937#issuecomment-2706303970):
-
pod-envs configmap and secret
apiVersion: v1 kind: ConfigMap metadata: name: pod-envs data: AWS_ENDPOINT: "http://minio.minio:9000" AWS_S3_FORCE_PATH_STYLE: "true" WAL_S3_BUCKET: test WAL_BUCKET_SCOPE_PREFIX: "" # every 5 min for test purposes BACKUP_SCHEDULE: '*/5 * * * *' BACKUP_NUM_TO_RETAIN: "5" --- apiVersion: v1 kind: Secret metadata: name: pod-envs type: Opaque data: AWS_ACCESS_KEY_ID: YWRtaW4K AWS_SECRET_ACCESS_KEY: YWRtaW4K -
postgres-operator configuration:
configKubernetes: pod_environment_configmap: "pod-envs" pod_environment_secret: "pod-envs"
Backups are not displayed if created with that option:
USE_WALG_BACKUP: "true"
