py-why
Replace all occurrences of get Pandas' get_dummies() with skLearn OneHotEncoder
An earlier issue #1111 observed inconsistent behaviour from RegressionEstimator subclasses when new data for do() method had different rows than the originally fitted data, which caused categorical variables to be encoded inconsistently. This is because the do() operator allows unseen data to be processed with an existing Estimator.
This issue occurs because categorical encoding was using Pandas' get_dummies(), which does not allow additional data to be encoded using an existing encoder. An alternative, skLearn OneHotEncoder, returns an Encoder object which can be used to encode additional data consistently. skLearn is already a DoWhy dependency. For this reason skLearn is preferred over get_dummies.
This additional change goes further to replace all occurrences of get_dummies with OneHotEncoder, so that if functionality to process additional data is added to other classes in future (e.g. via do operator), the consistency bug won't happen again.
- Features added to RegressionEstimator which remember a set of Encoders are pushed down to the base class CausalEstimator.
- All CausalEstimator subclasses call reset_encoders() on each fit(), implementing the lifecycle assumption that fit() implies entirely new data and to forget existing data.
- get_dummies was also used by the UnobservedCommonCause Refuter, but this usage has no side-effects and references to the encoded data are not retained. It was replaced simply for consistency of using skLearn.
- get_dummies was also used by the do-sampler's propensity score utility function binarize_discrete. Elsewhere in these utility functions skLearn LabelEncoder is used. So, for consistency, this occurrence is also replaced by skLearn OneHotEncoder.
After the swap, all these changes are heavily covered by existing tests.
@amit-sharma I added some tests which aim to verify that encoding is consistent despite permuting data row order. It was a bity tricky working within the interfaces of the Estimator classes - I focused on estimate_effect() and do(x). With Regression estimators the effects of common causes are additive, so the ATE is almost unchanged despite changes in these variables! To check for consistency of these variables' encoding using I used the do() operator, the result of which is affected by common causes.
In the process I discovered that the RegressionEstimator implementation of do() has a seemingly long-standing bug where the order of the arguments is reversed:
CausalEstimator base class (treatment_value, dataframe):
def _do(self, x, data_df=None):
RegressionEstimator (dataframe, treatment_value):
def _do(self, data_df: pd.DataFrame, treatment_val):
I've fixed RegressionEstimator to match the base class interface. I searched for all instances of _do( and only needed to fix the implementation of estimate_effect in Regression.
Changed from:
effect_estimate = self._do(data, treatment_value) - self._do(data, control_value)
to:
effect_estimate = self._do(treatment_value, data) - self._do(control_value, data)
I'm sorry this has turned into a big PR but hopefully it's worth it!
Build docs appears to be failing due to lack of disk space on the worker environment