seqeralabs
Timeout on /containers when creating a container with cellpose
Here is the build log: https://wave.seqera.io/view/builds/dded52ad814da816_1
I'm trying to build something with this package: https://github.com/MouseLand/cellpose
It appears that this container times out. I have a feeling that this takes a long time because of the CUDA dependencies that this package has. I'd imagine that many ML focused packages right now are running into a similar issue.
I would love to be able to address this issue for the ML community.
Currently all build have an hard limits to 15 minutes. It could looked into how to have different build profiles having a longer expiration time.
What are the implications of doubling this build time? Is the 15 arbitrary or is there a reason for the hard limit?
One side the longer the build the easier for a malicious user to saturate our build cluster.
Also a too long build would may cause the invoking client (e.g. a nextflow pipeline job to hold for all build time doing anything and burning money)
How about having extended build times only for authenticated users? That might mitigate the risk of malicious users a little.
Another option is to make it an opt-in flag, so that longer build times are only used if specifically requested. This is what GitHub Actions does for example.
Suggestion from @pditommaso in weekly meeting: Quick patch to double timeout to 30 mins for when user is authed + freeze is set. This will apply to all builds in Seqera Containers @odagayev.
Can investigate more complex solutions at a later date if we continue to see problems.
I'm afraid this won't work. This is for seqera.io/containers which is mostly unauthed users. Let me know if this is possible.
This has been released. Build submitted by auth users and having freeze enable will have 25 minutes (instead of 15 min) build time.