How to get the absorbed coefficients?
I do a y = Xβ + Dθ + e by using the AbsorbingLS, where D is an absorbed category variable.
I want to get the value of θ. I fail to find it in the AbsorbingLSResults. I find the absorbed_effects, which has the dimension of the obs. I think this might be Dθ, however I find it have the number of unique values different from the number of the categories in D and thus I am confused about what this absorbed_effects is.
In addition, I want to know how can I get the value of Xβ when X contains other category variables. I have tried using dmatrix to create X and doing Xβ = np.asarray(X) @ res.params.values.reshape(-1,1). But the sum of Xβ and absorbed_effects is different from res.fitted_values.
Thanks.
The absorbed coefficients are often difficult to interpret because in many models the D variable array is not full rank. The only claim for the absorbed effects they span the space of D. The coefficient are regularized by finding a set that san subject to minimizing the inner product theta'theta. ULtimately this is done by the LSMR algorithm which is in SciPy. You should be able to recover D'theta I think since you have y, beta, X and e.
I will need to check what fitted values is. I think it is the fitted value from regression D\Y on D\X where A\B means B with the span of A annihilated.
@bashtage Yes I can recover Dθ from the results. Actually I did a test to compare an OLS with several fixed effects and an AbsorbingLS with one category variable to be absorbed. I find that the resids fitted_values and the params except for the const are exactly the same, the coefficient of the const is different in two estimations and I think this is due to the absorbed dummy matrix is not full rank. So the fitted_values is actually correct. Then the only question left is that what is this absorbed_effects?