Koert Kuipers
Koert Kuipers
this pullreq is WIP because i am waiting to hear if there is any good reason this should not be exposed...
ok i will create new pullreq to just expose isolation level and only allow Serializable. i will let this one remain WIP just in case it turns out current code...
@allisonport-db i dont think that disabling that check will solve the issue? the codepath to set isolation level is simply not present in oss delta. see https://github.com/delta-io/delta/issues/1261
@allisonport-db oh i got it know... you are saying let it be set despite fact that it doesn't do anything.
https://issues.apache.org/jira/browse/SPARK-26133 in spark 3 OneHotEncoderEstimator was renamed to OneHotEncoder
Jackson has to be upgraded to 2.9.8. EitherModule was moved. See also https://github.com/salesforce/TransmogrifAI/issues/162 and https://github.com/salesforce/TransmogrifAI/issues/173
https://issues.apache.org/jira/browse/SPARK-26127 Spark removed a bunch of setters on regression and classification models. Since in TransmogrifAI these are overridden this no longer compiles.
in-house we are now using/testing transmogrifai with spark 3.0.0 snapshots. compileTestScala works, but not all tests (as in ./gradlew test) pass.
it seems OpDefaultScalaModule is broken with jackson 2.9.9 (which is what spark 3 uses) this took me a bit of time to track down because the error was hidden in...
ok we have a fully working branch against spark 3.0.0-SNAPSHOT that passes all tests it was mostly a pain because they changed the random number generator in spark 3 so...