Migrate AddonRecommender job from Scala to PySpark and from longitudinal to clients_daily

Open crankycoder opened this issue 7 years ago • 3 comments

The longitudinal dataset is being retired by the end of 2019, we'll need to migrate to whatever the datapipeline team has setup as a replacement.

Jul 30 '18 13:07 crankycoder

The active bug to track on the data platform side of things is: https://bugzilla.mozilla.org/show_bug.cgi?id=1485152

The plan in a nutshell is to migrate everything to clients_daily

Aug 31 '18 18:08 crankycoder

Similarity job has been updated in https://github.com/mozilla/python_mozetl/pull/265

The Collaborative job needs to be updated from longitudinal and from Scala to PySpark still

Scala code is over at: https://github.com/mozilla/telemetry-batch-view/blob/master/src/main/scala/com/mozilla/telemetry/ml/AddonRecommender.scala

Nov 07 '18 19:11 crankycoder

It was decided to not migrate it to Pyspark because of performance concerns. We can think about this again if we'll want to implement new features in the collaborative job and Scala code maintenance will be too expensive.

Jan 19 '21 01:01 evgenyrp