Cody Peterson
Cody Peterson
Hi @alexszym, This is a very core scenario for AzureML. We are introducing new features with v2 in the coming months to make this very easy. We'll build end-to-end MLOps...
What's your concern? With using MPI?
hi @seanmcclure22! in addition to the `dask-mpi` suggestion, we do have a public example of how a Dask cluster can be setup on Azure ML compute here: https://github.com/Azure/azureml-examples/tree/main/cli/jobs/single-step/dask/nyctaxi It uses...
https://pypi.org/project/dask-cloudprovider/#history version 0.4.0 was released a few hours ago and contains the updated code
can you check the `portforward_out_log.txt` (or named something similar-ish)? that usually has a clue on connection timeouts Is this through WSL?
so clearly a bug in getting/parsing the hostname on Windows machine - as a workaround I *think* WSL might work. but the longer term solution with more stability will be...
@drabastomek @FredFLi would either of you know exactly what the issue is here?
yep, using the `azureml.core.Environment` object you can pass in via `environment_definition`: https://azure.github.io/azureml-examples/docs/cheatsheet/environment#advanced-environment-variables so: ```python from azureml.core import Workspace from dask_cloudprovider import AzureMLCluster ws = Workspace.from_config() env = ws.environments['AzureML-Dask-CPU'].clone('myenv') env.environment_variables['FOO'] =...
to address a related issue - translating between Spark and Pandas/Dask - any objection to adding an alias `abfss` in addition to `az`?