DependencyTrack
ERROR [VulnerabilityIndexer] An error occurred while removing a vulnerability from the index
Current Behavior
Running DT as a AWS ECS Fargate service, using the latest version of the application. I'm using a aws Aurora PostgreSQL serverless datbase as the back end. this had all been working fine on V4.9, I then upgraded to V4.10 This is happening every time the application tries to update the vulnerability index.. Has this been observed by anyone else
org.apache.lucene.store.LockObtainFailedException: Lock held by this virtual machine: /data/.dependency-track/index/vulnerability/write.lock
at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:139)
at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41)
at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:923)
at org.dependencytrack.search.IndexManager.openIndex(IndexManager.java:189)
at org.dependencytrack.search.IndexManager.getIndexWriter(IndexManager.java:200)
at org.dependencytrack.search.VulnerabilityIndexer.remove(VulnerabilityIndexer.java:102)
at org.dependencytrack.search.VulnerabilityIndexer.remove(VulnerabilityIndexer.java:46)
at org.dependencytrack.tasks.IndexTask.inform(IndexTask.java:58)
at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:110)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Steps to Reproduce
Run DT v4.10 in a ecs fargate container
Expected Behavior
to not crash
Dependency-Track Version
4.10.1
Dependency-Track Distribution
Container Image
Database Server
PostgreSQL
Database Server Version
No response
Browser
Google Chrome
Checklist
- [X] I have read and understand the contributing guidelines
- [X] I have checked the existing issues for whether this defect was already reported
Can you try the following:
- Shutting down the application
- Manually removing /data/.dependency-track/index/vulnerability/write.lock
- Starting the application again
Maybe the application failed to clean up after itself at some point.
Sorry missed a bit of information on the tickets. This is running as a ECS fargate services/task as such i cant access the file system to remove the '/data/.dependency-track/index/vulnerability/write.lock' I did redeploy the task and that seems to have fixed it. Its odd that there was a write lock as by the nature of the ecs task they dont share any common filesystem.
Its odd that there was a write lock as by the nature of the ecs task they dont share any common filesystem.
Normally DT takes care to remove any locks when it shuts down. But that won't happen when the container is killed (e.g. due to OutOfMemoryException), instead of gracefully terminated.
@valentijnscholten I remember you also running into this? Is that still happening? Are you also on Fargate?
I'm running on ECS, but not fargate (Self managed EC2s). De change that we did back then helped and will also help non-ECS deployments I believe. But sometimes I still see locking issues when restarting. Even when the old containers is shutdown before the new one starts. But these errors are on startup and recovery is automatic.
Current Behavior
Running DT as a AWS ECS Fargate service, using the latest version of the application. I'm using a aws Aurora PostgreSQL serverless datbase as the back end. this had all been working fine on V4.9, I then upgraded to V4.10 This is happening every time the application tries to update the vulnerability index.. Has this been observed by anyone else
org.apache.lucene.store.LockObtainFailedException: Lock held by this virtual machine: /data/.dependency-track/index/vulnerability/write.lock at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:139) at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:923) at org.dependencytrack.search.IndexManager.openIndex(IndexManager.java:189) at org.dependencytrack.search.IndexManager.getIndexWriter(IndexManager.java:200) at org.dependencytrack.search.VulnerabilityIndexer.remove(VulnerabilityIndexer.java:102) at org.dependencytrack.search.VulnerabilityIndexer.remove(VulnerabilityIndexer.java:46) at org.dependencytrack.tasks.IndexTask.inform(IndexTask.java:58) at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:110) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source)Steps to Reproduce
Run DT v4.10 in a ecs fargate container
Expected Behavior
to not crash
Dependency-Track Version
4.10.1
Dependency-Track Distribution
Container Image
Database Server
PostgreSQL
Database Server Version
No response
Browser
Google Chrome
Checklist
- [x] I have read and understand the contributing guidelines
- [x] I have checked the existing issues for whether this defect was already reported
Please can you share your task definition json, because I tried multiple ways but I was able to make it run on AWS ECS Fargate. It always gave me the same write.lock error. Not even v4.9 or below worked for me.
I have the same problem with AWS Apprunner. It suspends the app to save on the CPU usage.
Shutting down application
| 2024-03-14T13:12:19.392+01:00 | 2024-03-14 12:12:19,392 INFO [AlpineServlet] Stopping Dependency-Track
| 2024-03-14T13:12:19.405+01:00 | 2024-03-14 12:12:19,404 INFO [IndexSubsystemInitializer] Closing search indexes
| 2024-03-14T13:12:19.414+01:00 | 2024-03-14 12:12:19,414 ERROR [VulnerabilityIndexer] An error occurred while adding a vulnerability to the index
| 2024-03-14T13:12:19.414+01:00 | org.apache.lucene.store.LockObtainFailedException: Lock held by this virtual machine: /data/.dependency-track/index/vulnerability/write.lock
