KMS error when bringing up tarmak cluster

[dippynark]

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened: Tarmak fails to unseal vault

Jul 01 15:03:09 vault-2.tarmak.local vault-unsealer[1103]: time="2018-07-01T15:03:09Z" level=error msg="error unsealing vault: unable to get key 'vault-unseal-0': AccessDeniedException: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.
Jul 01 15:03:09 vault-2.tarmak.local vault-unsealer[1103]: status code: 400, request id: ddd11958-7d3f-11e8-bc4f-b5662b89072a"

What you expected to happen: Tarmak unseals vault

How to reproduce it (as minimally and precisely as possible): Unsure, will try to repoduce

[ptla]

I've experienced the same recently and the following observations were made: -The ssm secret was in place in AWS Systems Manager > Parameter Store but outdated. -Doing a tarmak cluster apply does not recreate it, while from tests from a healthy test hub cluster if you destroy it and recreate it will create a new secret. Hypothesis: The issue could be related with stale secret in ssm. Reproduction is still not possible at the moment.