smartcore icon indicating copy to clipboard operation
smartcore copied to clipboard

Support streaming/incremental fitting

Open joelchen opened this issue 1 year ago • 0 comments

Memory is limited and using swap is painfully slow to fit massive amounts of data. There is discussion of incremental learning in https://github.com/smartcorelib/smartcore/issues/46 years ago but I cannot find partial_fit in the code. Spark has https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.mllib.regression.StreamingLinearRegressionWithSGD.html which also tackles limited memory with streaming data.

joelchen avatar May 07 '24 18:05 joelchen