linfa icon indicating copy to clipboard operation
linfa copied to clipboard

Support streaming/incremental fitting

Open joelchen opened this issue 1 year ago • 0 comments

Memory is limited and using swap is painfully slow to fit massive amounts of data. There is discussion of incremental fitting in https://github.com/rust-ml/linfa/issues/8 years ago but I cannot find incremental_fit in the code. Spark has https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.mllib.regression.StreamingLinearRegressionWithSGD.html which also helps with limited memory with streaming data.

joelchen avatar May 07 '24 18:05 joelchen