[BUG]Lightgbm classifier cannot get training results after setting verbosity=1 & isProvideTrainingMetric=True

[stupidoge]

SynapseML version

1.4.0

System information

• Language version (e.g. python 3.8, scala 2.12): scala 2.1.2, python3.10.2
• Spark Version (e.g. 3.4.1): 3.4.1
• Spark Platform (e.g. Synapse, Databricks): Databricks

Describe the problem

cannot see the training results or loss curve after setting verbosity and isProvideTrainingMetric

Code to reproduce issue

lgbm_assembler = VectorAssembler(inputCols=feature_cols, outputCol="features")
lgbm = LightGBMClassifier(
    objective="binary",
    numIterations=100,
    learningRate=0.1,
    isUnbalance=True,
    featuresCol="features",
    labelCol="label",
    isProvideTrainingMetric=True, 
    metric="auc",
    verbosity=1
)

pipeline = Pipeline(stages=[lgbm_assembler, lgbm])
model = pipeline.fit(train_data)
predictions = model.transform(val_data)

Other info / logs

No response

What component(s) does this bug affect?

• [ ] area/cognitive: Cognitive project
• [ ] area/core: Core project
• [ ] area/deep-learning: DeepLearning project
• [X] area/lightgbm: Lightgbm project
• [ ] area/opencv: Opencv project
• [ ] area/vw: VW project
• [ ] area/website: Website
• [ ] area/build: Project build system
• [ ] area/notebooks: Samples under notebooks folder
• [ ] area/docker: Docker usage
• [X] area/models: models related issue

What language(s) does this bug affect?

• [ ] language/scala: Scala source code
• [X] language/python: Pyspark APIs
• [ ] language/r: R APIs
• [ ] language/csharp: .NET APIs
• [ ] language/new: Proposals for new client languages

What integration(s) does this bug affect?

• [ ] integrations/synapse: Azure Synapse integrations
• [ ] integrations/azureml: Azure ML integrations
• [X] integrations/databricks: Databricks integrations

[stupidoge]

@svotaw @imatiach-msft Hi all, would you mind solving this issues? I thought you mentioned we could use some log function, but now, I cannot see any related functions.

[sohaibsyed21]

@stupidoge did you ever find a fix? I ran a model with verbosity=1 and after a long run saw no output, version 1.0.4. @svotaw @imatiach-msft any updates please?

[stupidoge]

@sohaibsyed21 not fixed.. they don't add this function to this api

[stupidoge]

@mhamilton723 @JessicaXYWang Hi maintainers, could you please give any suggestions?