vmware
Release shedlocks on shutdown
What is the feature request? What problem does it solve? Some of the logic in the Control Service (e.g. watching Kubernetes jobs for changes) is currently executed by a single instance of the service at any one time, regardless of how many instances there are. This is achieved by using locking (shedlock) around the said logic. When the service is restarted (most often during helm upgrades) if there is an active lock during the shutdown phase, this lock is not released. This in turn causes the subsequently restarted instance of the service to fail to acquire the lock until the previous lock has expired.
Suggested solution Handle the @PreDestroy spring hook and manually release all active locks. Since shedlock does not provide API for releasing locks, try to spy into it by subclassing the LockProvider class. A solution can be found in this branch: https://github.com/vmware/versatile-data-kit/tree/person/tpalashki/control-service
Acceptance criteria When the service is restarted, locks are acquired successfully.
