log_model fails after upgrading to Databricks 10.0ML

[JMCun]

We have recently upgraded Databricks runtime to 10.0 ML and we are experiencing issues when trying to log a model which uses the LightGBMRegressor class. The same exact code runs in 9.1ML without issues. Also we have tried logging models which use different libraries and it seems to work without issues.

Model: ml_model = LightGBMRegressor(objective='regression', featuresCol="features", labelCol=df_store_w.columns[6], alpha=0.2, learningRate=0.3, numLeaves=31)

Line of code with fails to run:

mlflow.spark.log_model(model, artifact_path = 'Model', conda_env = conda_env)

Error description:

com.databricks.backend.daemon.data.common.InvalidMountException: Error while using path /databricks/mlflow-tracking/4457284236158073/323c4fbf1ff94e03a31261ffe8164c79/artifacts/Model/sparkml for resolving path '/4457284236158073/323c4fbf1ff94e03a31261ffe8164c79/artifacts/Model/sparkml' within mount at '/databricks/mlflow-tracking'.

Thanks

AB#1984512

[imatiach-msft]

@JMCun interesting, I think this is the first time someone has mentioned trying mlflow.spark.log_model with synapseml, so I'm not surprised that it is failing. It sounds like we need to look into adding support for this scenario. Thank you for bringing this up.

[serena-ruan]

Hey @JMCun, actually I think mlflow.spark.log_model should work with models in our library that extends pyspark.ml.Model, but LightGBMRegressor doesn't follow this rule so it doesn't work (pls try LightGBMRegressionModel it should work). Besides, even for LightGBMRegressor the error shouldn't be the error message you get. For your errors, I found this: (https://docs.microsoft.com/en-us/azure/databricks/kb/machine-learning/mlflow-artifacts-no-client-error#invalid-mount-exception) Hope it helps!

[JMCun]

Thank you for the replies.

We can log them without issues on Databaricks 9.1ML. Maybe Databricks changed something on the new release. I'll investigate a little bit more, try LightGBMRegressionModel and let you know.