AzureML needs read-write permissions on its current working directory

[JesusChuz]

Azure functions running on consumption, Elastic Premium and App Service Plans would leave the /home/site/wwwroot as ready only, creating Out Of Memory errors. But in reality the error is created due to a Read-Only filesystem exception.

• A workaround to the problem is to use a custom container, since that would leave the wwwroot writable.

Result: Failure Exception: MemoryError: Engine process terminated. This is most likely due to system running out of memory. Please retry with increased memory. |session_id=l_c23cf85a-4ea0-424d-826c-a5105e38dfe4 Stack: File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/dispatcher.py", line 402, in _handle__invocation_request call_result = await self._loop.run_in_executor( File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/dispatcher.py", line 611, in _run_sync_func return ExtensionManager.get_sync_invocation_wrapper(context, File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/extension.py", line 215, in _raw_invocation_wrapper result = function(**args) File "/home/site/wwwroot/HttpTrigger/__init__.py", line 59, in main dataset = Dataset.Tabular.from_delimited_files(path = [(datastore, '/home/train-dataset/tabular/iris.csv')]) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/data/_loggerfactory.py", line 132, in wrapper return func(*args, **kwargs) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/data/dataset_factory.py", line 357, in from_delimited_files dataflow = dataprep().read_csv(path, File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/readers.py", line 100, in read_csv df = Dataflow._path_to_get_files_block(path, archive_options) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/dataflow.py", line 2376, in _path_to_get_files_block return datastore_to_dataflow(path) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/_datastore_helper.py", line 40, in datastore_to_dataflow datastore, datastore_value = get_datastore_value(source) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/_datastore_helper.py", line 92, in get_datastore_value _set_auth_type(workspace) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/_datastore_helper.py", line 184, in _set_auth_type get_engine_api().set_aml_auth(SetAmlAuthMessageArgument(auth_type, json.dumps(auth_value))) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/api.py", line 19, in get_engine_api _engine_api = EngineAPI() File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/api.py", line 103, in __init__ connect_to_requests_channel() File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/api.py", line 99, in connect_to_requests_channel self._engine_server_secret = self.sync_host_secret(self.requests_channel.host_secret) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/_aml_helper.py", line 38, in wrapper return send_message_func(op_code, message, cancellation_token) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/api.py", line 304, in sync_host_secret response = self._message_channel.send_message('Engine.SyncHostSecret', message_args, cancellation_token) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/engine.py", line 275, in send_message raise message['error'] File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/engine.py", line 223, in process_responses response = self._read_response(caller='MultiThreadMessageChannel.process_responses') File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/engine.py", line 148, in _read_response raise error

• Is there a way to point the SDK to write somewhere else in the container for example on the /home folder that is not set as read-only?

Function to repro the behavior: https://github.com/RoweKevin/AzureML-OOM-Repo

[anliakho2]

Thanks for bringing it to our attention, this should not be the case. I will open a bug and this should be fixed in the next release.

[eri820503]

I have similar issue as this case. I push my azure ml running script to azure function which is a read-only file system so no file can be saved in current working directory. But azure function offers a read-write directory /tmp and every temporary file can be saved in here.

When azure ml is running, it will create two log files as follow:

1. automl.log: which can be set to save in another directory by AutoMLConfig's argument debug_log in azureml.train.automl
2. azureml_automl.log: is there any way as above to save to another directory?
   • Because RemoteExperimentLauncher in azureml/train/automl/_experiment_drivers/remote_experiment_launcher.py will write azureml_automl.log in current working directory, it cause error in azure function.

Another thing may need to be mentioned. When I upload my pandas dataframe to azure ml dataset via register_pandas_dataframe in azureml.core.Dataset.Tabular, it seems that this method will save a temporary file in the current working directory? And cause the same error (Result: Failure Exception: MemoryError: Engine process terminated.) as @JesusChuz mentioned.