google
Verify gcs target
I just re-read #209 -- a general thought: Should the fix even be in the config parser? It seems like the source of the error has to do with the pipeline args / environment.
Furthermore, another possible fix is to change what kinds of errors we choose to retry and others we let bring down the pipeline. These are generally set here: https://github.com/google/weather-tools/blob/65b484ebdbc6af3c9311dbd097bf14712d96df4d/weather_dl/download_pipeline/util.py#L20
I just re-read #209 -- a general thought: Should the fix even be in the config parser? It seems like the source of the error has to do with the pipeline args / environment.
Furthermore, another possible fix is to change what kinds of errors we choose to retry and others we let bring down the pipeline. These are generally set here:
https://github.com/google/weather-tools/blob/65b484ebdbc6af3c9311dbd097bf14712d96df4d/weather_dl/download_pipeline/util.py#L20
I see point that logically it might fit better in the error handler, but IMO it makes more sense for the check to be done in the arg parser since it's fail-fast. Even if we fix the error handler to tear down the job in case target location is unreachable, it does needlessly incur costs for the user to run the job up to the point of failure, and it takes somewhere between 5-20 minutes for it to reach there.
Ok, the validation is very reasonable to me. I like the principle of failing fast.
My one nit is: this should at least occur in the run function and not in the parser, since it's testing resource choices rather than parsing (e.g. it's validating, not parsing).