att-comdev
Unable to execute 'get_status' API using Armada Client
Is this a bug report or feature request? (choose one): This is a bug
Python Version (output of python --version):
Python 3.5.2
Development or Deployment Environment?: Deployment Environment (CI/CD lab)
Release Tag or Master: Master
Expected Behavior:
Armada should be able check that Tiller is up and running properly with get_status API call
What Actually Happened: Problem seems to be related to armada config. We see the following errors in the logs of the Armada API container
*** Operational MODE: single process ***
Loading paste environment: config:/etc/armada/api-paste.ini
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
return loadobj(APP, uri, name=name, **kw)
File "/usr/local/lib/python3.5/dist-packages/paste/deploy/loadwsgi.py", line 271, in loadobj
global_conf=global_conf)
File "/usr/local/lib/python3.5/dist-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
global_conf=global_conf)
File "/usr/local/lib/python3.5/dist-packages/paste/deploy/loadwsgi.py", line 320, in _loadconfig
return loader.get_context(object_type, name, global_conf)
File "/usr/local/lib/python3.5/dist-packages/paste/deploy/loadwsgi.py", line 450, in get_context
global_additions=global_additions)
File "/usr/local/lib/python3.5/dist-packages/paste/deploy/loadwsgi.py", line 559, in _pipeline_app_context
APP, pipeline[-1], global_conf)
File "/usr/local/lib/python3.5/dist-packages/paste/deploy/loadwsgi.py", line 458, in get_context
section)
File "/usr/local/lib/python3.5/dist-packages/paste/deploy/loadwsgi.py", line 517, in _context_from_explicit
value = import_string(found_expr)
File "/usr/local/lib/python3.5/dist-packages/paste/deploy/loadwsgi.py", line 22, in import_string
return pkg_resources.EntryPoint.parse("x=" + s).load(False)
File "/usr/local/lib/python3.5/dist-packages/pkg_resources/init.py", line 2405, in load
return self.resolve()
File "/usr/local/lib/python3.5/dist-packages/pkg_resources/init.py", line 2411, in resolve
module = import(self.module_name, fromlist=['name'], level=0)
File "./armada/api/server.py", line 31, in
How to Reproduce the Issue (as minimally as possible): Re-run the genesis-full pipeline in CI/CD lab to start deployment from scratch
Any Additional Comments: N.A.
We tested with the patch set in CI/CD lab but were unable to deploy the UCP components with the new test image (http://docker.io/gerrive/armada:git-4470f4e). We are seeing the below errors.
Seems like the new changes caused some side effects with deployment of UCP components.
ubuntu@genesis:~$ sudo kubectl get pods -n ucp -o wide
NAME READY STATUS RESTARTS AGE IP NODE
etcd-5bcbbd679c-mf4cr 1/1 Running 0 9m 10.97.244.208 genesis
mariadb-0 1/1 Running 0 9m 10.97.244.210 genesis
mariadb-1 1/1 Running 0 6m 10.97.244.212 genesis
mariadb-2 1/1 Running 0 5m 10.97.244.213 genesis
memcached-5bf49657db-rfxsk 1/1 Running 0 9m 10.97.244.207 genesis
postgresql-0 1/1 Running 0 9m 10.97.244.211 genesis
rabbitmq-f68649644-7w49s 1/1 Running 0 9m 10.97.244.209 genesis
ubuntu@genesis:~$
2017-12-02 02:21:25.091 1 INFO armada.handlers.armada [-] Deploying: UCP Services
2017-12-02 02:21:25.221 1 INFO armada.handlers.chartbuilder [-] Building dependency chart maas for release maas
2017-12-02 02:21:25.259 1 DEBUG armada.handlers.armada [-] RELEASE: maas sync /usr/local/lib/python3.5/dist-packages/armada/handlers/armada.py:253
2017-12-02 02:21:25.259 1 INFO armada.handlers.armada [-] Installing release maas
2017-12-02 02:21:25.265 1 INFO armada.handlers.tiller [-] Wait: False, Timeout: 3600
2017-12-02 02:21:25.319 1 CRITICAL armada [-] Unhandled error: armada.exceptions.tiller_exceptions.GetReleaseStatusException: Failed to get ucp-maas status 0 version
2017-12-02 02:21:25.319 1 ERROR armada Traceback (most recent call last):
2017-12-02 02:21:25.319 1 ERROR armada File "/usr/local/lib/python3.5/dist-packages/armada/handlers/tiller.py", line 325, in install_release
2017-12-02 02:21:25.319 1 ERROR armada release_request, self.timeout, metadata=self.metadata)
2017-12-02 02:21:25.319 1 ERROR armada File "/usr/local/lib/python3.5/dist-packages/grpc/_channel.py", line 492, in __call__
2017-12-02 02:21:25.319 1 ERROR armada return _end_unary_response_blocking(state, call, False, deadline)
2017-12-02 02:21:25.319 1 ERROR armada File "/usr/local/lib/python3.5/dist-packages/grpc/_channel.py", line 440, in _end_unary_response_blocking
2017-12-02 02:21:25.319 1 ERROR armada raise _Rendezvous(state, None, None, deadline)
2017-12-02 02:21:25.319 1 ERROR armada grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, render error in "maas/deployment-rack.yaml": template: maas/deployment-rack.yaml:35:62: executing "maas/deployment-rack.yaml" at <include "helm-toolki...>: error calling include: template: maas/charts/helm-toolkit/utils/_hash.tpl:22:4: executing "helm-toolkit.utils.hash" at <include $wtf $contex...>: error calling include: template: maas/configmap-bin.yaml:39:52: executing "maas/configmap-bin.yaml" at <include "helm-toolki...>: error calling include: template: maas/charts/helm-toolkit/utils/_template.tpl:22:3: executing "helm-toolkit.utils.template" at <include $wtf $contex...>: error calling include: template: no template "maas/bin/_register-rack-controller.sh.tpl" associated with template "gotpl")>
2017-12-02 02:21:25.319 1 ERROR armada
2017-12-02 02:21:25.319 1 ERROR armada During handling of the above exception, another exception occurred:
2017-12-02 02:21:25.319 1 ERROR armada
2017-12-02 02:21:25.319 1 ERROR armada Traceback (most recent call last):
2017-12-02 02:21:25.319 1 ERROR armada File "/usr/local/lib/python3.5/dist-packages/armada/handlers/tiller.py", line 382, in get_release_status
2017-12-02 02:21:25.319 1 ERROR armada status_request, self.timeout, metadata=self.metadata)
2017-12-02 02:21:25.319 1 ERROR armada File "/usr/local/lib/python3.5/dist-packages/grpc/_channel.py", line 492, in __call__
2017-12-02 02:21:25.319 1 ERROR armada return _end_unary_response_blocking(state, call, False, deadline)
2017-12-02 02:21:25.319 1 ERROR armada File "/usr/local/lib/python3.5/dist-packages/grpc/_channel.py", line 440, in _end_unary_response_blocking
2017-12-02 02:21:25.319 1 ERROR armada raise _Rendezvous(state, None, None, deadline)
2017-12-02 02:21:25.319 1 ERROR armada grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, getting deployed release "ucp-maas": release: "ucp-maas" not found)>
2017-12-02 02:21:25.319 1 ERROR armada
2017-12-02 02:21:25.319 1 ERROR armada During handling of the above exception, another exception occurred:
2017-12-02 02:21:25.319 1 ERROR armada
2017-12-02 02:21:25.319 1 ERROR armada Traceback (most recent call last):
2017-12-02 02:21:25.319 1 ERROR armada File "/usr/local/bin/armada", line 10, in <module>
2017-12-02 02:21:25.319 1 ERROR armada sys.exit(main())
2017-12-02 02:21:25.319 1 ERROR armada File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 722, in __call__
2017-12-02 02:21:25.319 1 ERROR armada return self.main(*args, **kwargs)
2017-12-02 02:21:25.319 1 ERROR armada File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 697, in main
2017-12-02 02:21:25.319 1 ERROR armada rv = self.invoke(ctx)
2017-12-02 02:21:25.319 1 ERROR armada File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 1066, in invoke
2017-12-02 02:21:25.319 1 ERROR armada return _process_result(sub_ctx.command.invoke(sub_ctx))
2017-12-02 02:21:25.319 1 ERROR armada File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 895, in invoke
2017-12-02 02:21:25.319 1 ERROR armada return ctx.invoke(self.callback, **ctx.params)
2017-12-02 02:21:25.319 1 ERROR armada File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 535, in invoke
2017-12-02 02:21:25.319 1 ERROR armada return callback(*args, **kwargs)
2017-12-02 02:21:25.319 1 ERROR armada File "/usr/local/lib/python3.5/dist-packages/click/decorators.py", line 17, in new_func
2017-12-02 02:21:25.319 1 ERROR armada return f(get_current_context(), *args, **kwargs)
2017-12-02 02:21:25.319 1 ERROR armada File "/usr/local/lib/python3.5/dist-packages/armada/cli/apply.py", line 120, in apply_create
2017-12-02 02:21:25.319 1 ERROR armada wait).invoke()
2017-12-02 02:21:25.319 1 ERROR armada File "/usr/local/lib/python3.5/dist-packages/armada/cli/apply.py", line 186, in invoke
2017-12-02 02:21:25.319 1 ERROR armada resp = armada.sync()
2017-12-02 02:21:25.319 1 ERROR armada File "/usr/local/lib/python3.5/dist-packages/armada/handlers/armada.py", line 330, in sync
2017-12-02 02:21:25.319 1 ERROR armada timeout=chart_timeout)
2017-12-02 02:21:25.319 1 ERROR armada File "/usr/local/lib/python3.5/dist-packages/armada/handlers/tiller.py", line 328, in install_release
2017-12-02 02:21:25.319 1 ERROR armada status = self.get_release_status(release)
2017-12-02 02:21:25.319 1 ERROR armada File "/usr/local/lib/python3.5/dist-packages/armada/handlers/tiller.py", line 385, in get_release_status
2017-12-02 02:21:25.319 1 ERROR armada raise ex.GetReleaseStatusException(release, version)
2017-12-02 02:21:25.319 1 ERROR armada armada.exceptions.tiller_exceptions.GetReleaseStatusException: Failed to get ucp-maas status 0 version
2017-12-02 02:21:25.319 1 ERROR armada