Netflix
Fix code packaging when the flow file is in the root directory
From Darshan Patel on Metaflow Slack -
hey team, I'm getting an error while running a flow on AWS Batch. Metaflow is able to create a batch job definition and kicks a run on Batch but on the batch agent I'm getting following error. do you know what can cause this error?
Setting up task environment.
Downloading code package...
Code package downloaded.
Task is starting.
python: can't open file 'test.py': [Errno 2] No such file or directory
@savingoyal my team encountered this error when running metaflow within a poetry environment on Ubuntu. We tracked down the problem to this line. On Ubuntu, poetry managed virtual environments are stored at: /home/ubuntu/.cache/pypoetry/virtualenvs/. The flow we are calling is part of a shared generic training library we use to easily configure and launch training jobs either locally or on batch. However since metaflow is ignoring "hidden" paths when constructing the tarball and the flow we are calling is stored within a package inside a hidden poetry path, the flow isn't getting added to the tarball. Do you have any advice on how to handle this? There is the exclude_hidden flag in the walk function, but that isn't exposed to the user. Else I'm thinking it could be a good idea to add some extra logic to make sure the flow is added to the tarball no matter what. Thanks!
We ended up solving this issue by having our training library copy the flow to a non-hidden tempdir before calling it. Kinda hacky but works
Apparently the walk function was ignoring my experimentation folder .scrap where my flow was. 6 hours of debugging.
git commit -m "sui**de"
