Sean Harkins

I had posted an issue around some of this discussion on an internal Development Seed repo but I realized that it might be worthwhile to share. I copied it over...

@martindurant Is the expanded use of `_run_coros_in_chunks` across all `gather` operations which you mention in ref https://github.com/fsspec/s3fs/issues/537#issuecomment-944335863 still under consideration?

The gridMET recipe is also currently blocked by this ref https://github.com/pangeo-forge/staged-recipes/pull/79

@TomAugspurger I've been locked up on other things today but I'll try to run some testing with this tomorrow 👍

@cisaacstern @rabernat I got a message on Slack from @kvnkho that he is hoping to have a PR addressing prepped later this week.

@TomAugspurger Thus far on the bakery side we are taking a 2 pronged logging approach. Individual worker logging is redirected to cloud provider log capture. In the case of AWS...

@TomAugspurger I'll continue tracking this here. Recipe module logging level is configurable at flow registration time with this wrapper https://github.com/pangeo-forge/pangeo-forge-prefect/blob/master/pangeo_forge_prefect/flow_manager.py#L58. This works correctly within each cloud provider's log solution (Cloudwatch...

Running this [recipe](https://github.com/pangeo-forge/staged-recipes/pull/20) the failure is occurring on [open_dataset](https://github.com/pangeo-forge/pangeo-forge/blob/master/pangeo_forge/recipe.py#L421). Currently running on a cluster with a single worker node using the following versions (this PR branch of `xarray` is necessary...

@rabernat The is being run using the `PrefectPipelineExecutor` against our AWS bakery with a Prefect agent and dynamically created Dask cluster running on our Fargate infrastructure and S3 storage. For...

@rabernat Let me kick off a new run with the log level set for `debug` and collect the worker logs for you.