linearmodels icon indicating copy to clipboard operation
linearmodels copied to clipboard

How to get the absorbed coefficients?

Open Alalalalaki opened this issue 4 years ago • 2 comments

I do a y = Xβ + Dθ + e by using the AbsorbingLS, where D is an absorbed category variable.

I want to get the value of θ. I fail to find it in the AbsorbingLSResults. I find the absorbed_effects, which has the dimension of the obs. I think this might be , however I find it have the number of unique values different from the number of the categories in D and thus I am confused about what this absorbed_effects is.

In addition, I want to know how can I get the value of when X contains other category variables. I have tried using dmatrix to create X and doing Xβ = np.asarray(X) @ res.params.values.reshape(-1,1). But the sum of and absorbed_effects is different from res.fitted_values.

Thanks.

Alalalalaki avatar Oct 16 '21 15:10 Alalalalaki

The absorbed coefficients are often difficult to interpret because in many models the D variable array is not full rank. The only claim for the absorbed effects they span the space of D. The coefficient are regularized by finding a set that san subject to minimizing the inner product theta'theta. ULtimately this is done by the LSMR algorithm which is in SciPy. You should be able to recover D'theta I think since you have y, beta, X and e.

I will need to check what fitted values is. I think it is the fitted value from regression D\Y on D\X where A\B means B with the span of A annihilated.

bashtage avatar Oct 18 '21 17:10 bashtage

@bashtage Yes I can recover from the results. Actually I did a test to compare an OLS with several fixed effects and an AbsorbingLS with one category variable to be absorbed. I find that the resids fitted_values and the params except for the const are exactly the same, the coefficient of the const is different in two estimations and I think this is due to the absorbed dummy matrix is not full rank. So the fitted_values is actually correct. Then the only question left is that what is this absorbed_effects?

Alalalalaki avatar Oct 19 '21 21:10 Alalalalaki