StructuralEquationModels.jl
StructuralEquationModels.jl copied to clipboard
StructuralEquationModels
MethodError: no method matching SemObservedData
I used Onyx to automatically generate a model specification for a factor-of-curves model that models linear and quadratic change over four dimensions including second-order-factors across dimensions. I load some simulated data (see attached file gca_simulated.csv ), specify the model but I get an error from the model constructor. Please help. Here is the error:
)
ERROR: MethodError: no method matching SemObservedData(::Matrix{…}, ::Vector{…}, ::Matrix{…}, ::Base.ReshapedArray{…}, ::Int64)
The type `SemObservedData` exists, but no method is defined for this combination of argument types when trying to construct it.
Closest candidates are:
SemObservedData(::D, ::Vector{Symbol}, ::Matrix{S}, ::Vector{S}, ::Int64) where {D<:Union{Nothing, AbstractMatrix}, S<:Number}
@ StructuralEquationModels C:\Users\XXX\.julia\packages\StructuralEquationModels\JhO26\src\observed\data.jl:27
SemObservedData(; data, observed_vars, specification, observed_var_prefix, kwargs...)
@ StructuralEquationModels C:\Users\XXX\.julia\packages\StructuralEquationModels\JhO26\src\observed\data.jl:34
Stacktrace:
[1] SemObservedData(; data::DataFrame, observed_vars::Nothing, specification::ParameterTable{…}, observed_var_prefix::Symbol, kwargs::@Kwargs{…})
@ StructuralEquationModels C:\Users\XXX\.julia\packages\StructuralEquationModels\JhO26\src\observed\data.jl:45
[2] get_fields!(kwargs::Dict{…}, specification::ParameterTable{…}, observed::Type{…}, implied::Type{…}, loss::Type)
@ StructuralEquationModels C:\Users\XXX\.julia\packages\StructuralEquationModels\JhO26\src\frontend\specification\Sem.jl:106
[3] Sem(; specification::ParameterTable{…}, observed::Type{…}, implied::Type{…}, loss::Type{…}, kwargs::@Kwargs{…})
@ StructuralEquationModels C:\Users\XXX\.julia\packages\StructuralEquationModels\JhO26\src\frontend\specification\Sem.jl:16
[4] top-level scope
@ REPL[22]:1
Here is my code:
using CSV, DataFrames
using StructuralEquationModels
data = CSV.read("gca_simulated.csv", DataFrames.DataFrame;
normalizenames = true,
delim = ',',
missingstring = ["", "NA", "NaN"],
ignorerepeated = true
)
graph = @StenoGraph begin
# regressions
IS → fixed(1.0)*S_4
IS → fixed(1.0)*S_5
IS → fixed(1.0)*S_6
LS → fixed(1.0)*S_2
LS → fixed(3.0)*S_4
LS → fixed(4.0)*S_5
LS → fixed(5.0)*S_6
QS → fixed(4.0)*S_3
QS → fixed(9.0)*S_4
QS → fixed(16.0)*S_5
QS → fixed(25.0)*S_6
QS → fixed(1.0)*S_2
IS → fixed(1.0)*S_1
IS → fixed(1.0)*S_3
RS → fixed(1.0)*S_2
RS → fixed(1.0)*S_3
RS → fixed(1.0)*S_4
RS → fixed(1.0)*S_5
RS → fixed(1.0)*S_6
LS → fixed(2.0)*S_3
IS → fixed(1.0)*S_2
LS → fixed(1.0)*S_1
QS → fixed(1.0)*S_1
IC → fixed(1.0)*C_4
IC → fixed(1.0)*C_5
IC → fixed(1.0)*C_6
LC → fixed(1.0)*C_2
LC → fixed(3.0)*C_4
LC → fixed(4.0)*C_5
LC → fixed(5.0)*C_6
QC → fixed(4.0)*C_3
QC → fixed(9.0)*C_4
QC → fixed(16.0)*C_5
QC → fixed(25.0)*C_6
QC → fixed(1.0)*C_2
IC → fixed(1.0)*C_1
IC → fixed(1.0)*C_3
RC → fixed(1.0)*C_2
RC → fixed(1.0)*C_3
RC → fixed(1.0)*C_4
RC → fixed(1.0)*C_5
RC → fixed(1.0)*C_6
LC → fixed(2.0)*C_3
IC → fixed(1.0)*C_2
LC → fixed(1.0)*C_1
QC → fixed(1.0)*C_1
IF → fixed(1.0)*F_4
IF → fixed(1.0)*F_5
IF → fixed(1.0)*F_6
LF → fixed(1.0)*F_2
LF → fixed(3.0)*F_4
LF → fixed(4.0)*F_5
LF → fixed(5.0)*F_6
QF → fixed(4.0)*F_3
QF → fixed(9.0)*F_4
QF → fixed(16.0)*F_5
QF → fixed(25.0)*F_6
QF → fixed(1.0)*F_2
IF → fixed(1.0)*F_1
IF → fixed(1.0)*F_3
RF → fixed(1.0)*F_2
RF → fixed(1.0)*F_3
RF → fixed(1.0)*F_4
RF → fixed(1.0)*F_5
RF → fixed(1.0)*F_6
LF → fixed(2.0)*F_3
IF → fixed(1.0)*F_2
LF → fixed(1.0)*F_1
QF → fixed(1.0)*F_1
IM → fixed(1.0)*M_4
IM → fixed(1.0)*M_5
IM → fixed(1.0)*M_6
LM → fixed(1.0)*M_2
LM → fixed(3.0)*M_4
LM → fixed(4.0)*M_5
LM → fixed(5.0)*M_6
QM → fixed(4.0)*M_3
QM → fixed(9.0)*M_4
QM → fixed(16.0)*M_5
QM → fixed(25.0)*M_6
QM → fixed(1.0)*M_2
IM → fixed(1.0)*M_1
IM → fixed(1.0)*M_3
RM → fixed(1.0)*M_2
RM → fixed(1.0)*M_3
RM → fixed(1.0)*M_4
RM → fixed(1.0)*M_5
RM → fixed(1.0)*M_6
LM → fixed(2.0)*M_3
IM → fixed(1.0)*M_2
LM → fixed(1.0)*M_1
QM → fixed(1.0)*M_1
IS → label(:GI__IS)*GI
IC → label(:GI__IC)*GI
IF → label(:GI__IF)*GI
IM → label(:GI__IM)*GI
LS → label(:GL__LS)*GL
LC → label(:GL__LC)*GL
LF → label(:GL__LF)*GL
LM → label(:GL__LM)*GL
# residuals, variances and covariances
RS ↔ label(:VAR_RS)*RS
IS ↔ label(:VAR_IS)*IS
IS ↔ label(:IS_LS)*LS
LS ↔ label(:VAR_LS)*LS
S_2 ↔ label(:es)*S_2
S_3 ↔ label(:es)*S_3
S_4 ↔ label(:es)*S_4
S_5 ↔ label(:es)*S_5
S_6 ↔ label(:es)*S_6
S_1 ↔ label(:es)*S_1
RC ↔ label(:VAR_RC)*RC
IC ↔ label(:VAR_IC)*IC
IC ↔ label(:IC_LC)*LC
LC ↔ label(:VAR_LC)*LC
C_2 ↔ label(:ec)*C_2
C_3 ↔ label(:ec)*C_3
C_4 ↔ label(:ec)*C_4
C_5 ↔ label(:ec)*C_5
C_6 ↔ label(:ec)*C_6
C_1 ↔ label(:ec)*C_1
RF ↔ label(:VAR_RF)*RF
IF ↔ label(:VAR_IF)*IF
IF ↔ label(:IS_LF)*LF
LF ↔ label(:VAR_LF)*LF
F_2 ↔ label(:ef)*F_2
F_3 ↔ label(:ef)*F_3
F_4 ↔ label(:ef)*F_4
F_5 ↔ label(:ef)*F_5
F_6 ↔ label(:ef)*F_6
F_1 ↔ label(:ef)*F_1
RM ↔ label(:VAR_RM)*RM
IM ↔ label(:VAR_IM)*IM
IM ↔ label(:IM_LM)*LM
LM ↔ label(:VAR_LM)*LM
M_2 ↔ label(:em)*M_2
M_3 ↔ label(:em)*M_3
M_4 ↔ label(:em)*M_4
M_5 ↔ label(:em)*M_5
M_6 ↔ label(:em)*M_6
M_1 ↔ label(:em)*M_1
GL ↔ fixed(1.0)*GL
GI ↔ fixed(1.0)*GI
GI ↔ label(:COV_GI_GL)*GL
QS ↔ label(:VAR_QS)*QS
QC ↔ label(:VAR_QC)*QC
QF ↔ label(:VAR_QF)*QF
QM ↔ label(:VAR_QM)*QM
IS ↔ fixed(0.0)*QS
IS ↔ fixed(0.0)*RS
IS ↔ fixed(0.0)*IC
IS ↔ fixed(0.0)*LC
IS ↔ fixed(0.0)*QC
IS ↔ fixed(0.0)*RC
IS ↔ fixed(0.0)*IF
IS ↔ fixed(0.0)*LF
IS ↔ fixed(0.0)*QF
IS ↔ fixed(0.0)*RF
IS ↔ fixed(0.0)*IM
IS ↔ fixed(0.0)*LM
IS ↔ fixed(0.0)*QM
IS ↔ fixed(0.0)*RM
IS ↔ fixed(0.0)*GL
LS ↔ fixed(0.0)*QS
LS ↔ fixed(0.0)*RS
LS ↔ fixed(0.0)*IC
LS ↔ fixed(0.0)*LC
LS ↔ fixed(0.0)*QC
LS ↔ fixed(0.0)*RC
LS ↔ fixed(0.0)*IF
LS ↔ fixed(0.0)*LF
LS ↔ fixed(0.0)*QF
LS ↔ fixed(0.0)*RF
LS ↔ fixed(0.0)*IM
LS ↔ fixed(0.0)*LM
LS ↔ fixed(0.0)*QM
LS ↔ fixed(0.0)*RM
LS ↔ fixed(0.0)*GI
QS ↔ fixed(0.0)*RS
QS ↔ fixed(0.0)*IC
QS ↔ fixed(0.0)*LC
QS ↔ fixed(0.0)*QC
QS ↔ fixed(0.0)*RC
QS ↔ fixed(0.0)*IF
QS ↔ fixed(0.0)*LF
QS ↔ fixed(0.0)*QF
QS ↔ fixed(0.0)*RF
QS ↔ fixed(0.0)*IM
QS ↔ fixed(0.0)*LM
QS ↔ fixed(0.0)*QM
QS ↔ fixed(0.0)*RM
QS ↔ fixed(0.0)*GL
QS ↔ fixed(0.0)*GI
RS ↔ fixed(0.0)*IC
RS ↔ fixed(0.0)*LC
RS ↔ fixed(0.0)*QC
RS ↔ fixed(0.0)*RC
RS ↔ fixed(0.0)*IF
RS ↔ fixed(0.0)*LF
RS ↔ fixed(0.0)*QF
RS ↔ fixed(0.0)*RF
RS ↔ fixed(0.0)*IM
RS ↔ fixed(0.0)*LM
RS ↔ fixed(0.0)*QM
RS ↔ fixed(0.0)*RM
RS ↔ fixed(0.0)*GL
RS ↔ fixed(0.0)*GI
IC ↔ fixed(0.0)*QC
IC ↔ fixed(0.0)*RC
IC ↔ fixed(0.0)*IF
IC ↔ fixed(0.0)*LF
IC ↔ fixed(0.0)*QF
IC ↔ fixed(0.0)*RF
IC ↔ fixed(0.0)*IM
IC ↔ fixed(0.0)*LM
IC ↔ fixed(0.0)*QM
IC ↔ fixed(0.0)*RM
IC ↔ fixed(0.0)*GL
LC ↔ fixed(0.0)*QC
LC ↔ fixed(0.0)*RC
LC ↔ fixed(0.0)*IF
LC ↔ fixed(0.0)*LF
LC ↔ fixed(0.0)*QF
LC ↔ fixed(0.0)*RF
LC ↔ fixed(0.0)*IM
LC ↔ fixed(0.0)*LM
LC ↔ fixed(0.0)*QM
LC ↔ fixed(0.0)*RM
LC ↔ fixed(0.0)*GI
QC ↔ fixed(0.0)*RC
QC ↔ fixed(0.0)*IF
QC ↔ fixed(0.0)*LF
QC ↔ fixed(0.0)*QF
QC ↔ fixed(0.0)*RF
QC ↔ fixed(0.0)*IM
QC ↔ fixed(0.0)*LM
QC ↔ fixed(0.0)*QM
QC ↔ fixed(0.0)*RM
QC ↔ fixed(0.0)*GL
QC ↔ fixed(0.0)*GI
RC ↔ fixed(0.0)*IF
RC ↔ fixed(0.0)*LF
RC ↔ fixed(0.0)*QF
RC ↔ fixed(0.0)*RF
RC ↔ fixed(0.0)*IM
RC ↔ fixed(0.0)*LM
RC ↔ fixed(0.0)*QM
RC ↔ fixed(0.0)*RM
RC ↔ fixed(0.0)*GL
RC ↔ fixed(0.0)*GI
IF ↔ fixed(0.0)*QF
IF ↔ fixed(0.0)*RF
IF ↔ fixed(0.0)*IM
IF ↔ fixed(0.0)*LM
IF ↔ fixed(0.0)*QM
IF ↔ fixed(0.0)*RM
IF ↔ fixed(0.0)*GL
LF ↔ fixed(0.0)*QF
LF ↔ fixed(0.0)*RF
LF ↔ fixed(0.0)*IM
LF ↔ fixed(0.0)*LM
LF ↔ fixed(0.0)*QM
LF ↔ fixed(0.0)*RM
LF ↔ fixed(0.0)*GI
QF ↔ fixed(0.0)*RF
QF ↔ fixed(0.0)*IM
QF ↔ fixed(0.0)*LM
QF ↔ fixed(0.0)*QM
QF ↔ fixed(0.0)*RM
QF ↔ fixed(0.0)*GL
QF ↔ fixed(0.0)*GI
RF ↔ fixed(0.0)*IM
RF ↔ fixed(0.0)*LM
RF ↔ fixed(0.0)*QM
RF ↔ fixed(0.0)*RM
RF ↔ fixed(0.0)*GL
RF ↔ fixed(0.0)*GI
IM ↔ fixed(0.0)*QM
IM ↔ fixed(0.0)*RM
IM ↔ fixed(0.0)*GL
LM ↔ fixed(0.0)*QM
LM ↔ fixed(0.0)*RM
LM ↔ fixed(0.0)*GI
QM ↔ fixed(0.0)*RM
QM ↔ fixed(0.0)*GL
QM ↔ fixed(0.0)*GI
RM ↔ fixed(0.0)*GL
RM ↔ fixed(0.0)*GI
# means
Symbol(1) → label(:const__RS)*RS
Symbol(1) → label(:const__QS)*QS
Symbol(1) → label(:const__LS)*LS
Symbol(1) → label(:const__IS)*IS
Symbol(1) → label(:const__RC)*RC
Symbol(1) → label(:const__QC)*QC
Symbol(1) → label(:const__LC)*LC
Symbol(1) → label(:const__IC)*IC
Symbol(1) → label(:const__RF)*RF
Symbol(1) → label(:const__QF)*QF
Symbol(1) → label(:const__LF)*LF
Symbol(1) → label(:const__IF)*IF
Symbol(1) → label(:const__RM)*RM
Symbol(1) → label(:const__QM)*QM
Symbol(1) → label(:const__LM)*LM
Symbol(1) → label(:const__IM)*IM
Symbol(1) → fixed(0.0)*S_1
Symbol(1) → fixed(0.0)*S_2
Symbol(1) → fixed(0.0)*S_3
Symbol(1) → fixed(0.0)*S_4
Symbol(1) → fixed(0.0)*S_5
Symbol(1) → fixed(0.0)*S_6
Symbol(1) → fixed(0.0)*C_1
Symbol(1) → fixed(0.0)*C_2
Symbol(1) → fixed(0.0)*C_3
Symbol(1) → fixed(0.0)*C_4
Symbol(1) → fixed(0.0)*C_5
Symbol(1) → fixed(0.0)*C_6
Symbol(1) → fixed(0.0)*F_1
Symbol(1) → fixed(0.0)*F_2
Symbol(1) → fixed(0.0)*F_3
Symbol(1) → fixed(0.0)*F_4
Symbol(1) → fixed(0.0)*F_5
Symbol(1) → fixed(0.0)*F_6
Symbol(1) → fixed(0.0)*M_1
Symbol(1) → fixed(0.0)*M_2
Symbol(1) → fixed(0.0)*M_3
Symbol(1) → fixed(0.0)*M_4
Symbol(1) → fixed(0.0)*M_5
Symbol(1) → fixed(0.0)*M_6
Symbol(1) → fixed(0.0)*GL
Symbol(1) → fixed(0.0)*GI
end
lat = [:IS,:LS,:QS,:RS,:IC,:LC,:QC,:RC,:IF,:LF,:QF,:RF,:IM,:LM,:QM,:RM,:GL,:GI]
obs=[:S_1,:S_2,:S_3,:S_4,:S_5,:S_6,:C_1,:C_2,:C_3,:C_4,:C_5,:C_6,:F_1,:F_2,:F_3,:F_4,:F_5,:F_6,:M_1,:M_2,:M_3,:M_4,:M_5,:M_6]
partable = ParameterTable(
graph,
latent_vars = lat,
observed_vars = obs
)
model = Sem(
specification = partable,
data = data,
meanstructure = true
)
I just understood that the problem is that the data file has missing values. Can you help me out how I can adjust the above code that my Stenograph model can be fitted using SemObservedMissing internally? This is also relevant for the tutorial, I think.
On a more general note, I think that this is a potential issue that many novice users will struggle with. I see two ways out. Either, there is a generic constructor that chooses the appropriate SemObserved object for the users or, probably preferred, there should be some proper error handling if data and observed do not match. The current error message has no information for users why the error happened.
I came a little bit farther:
# observed ---------------------------------------------------------------------------------
observed = SemObservedMissing(specification = partable, data = data)
# imply ------------------------------------------------------------------------------------
imply_ram = RAM(specification = partable)
# loss -------------------------------------------------------------------------------------
ml = SemFIML(observed = observed, specification = partable)
loss_ml = SemLoss(ml)
# optimizer -------------------------------------------------------------------------------------
optimizer = SemOptimizerOptim()
model_ml = Sem(observed, imply_ram, loss_ml)
fit(model_ml)
┌ Warning: EM Algorithm for MVN missing data did not converge. Likelihood for FIML is not interpretable.
│ Maybe try passing different starting values via 'start_em = ...'
└ @ StructuralEquationModels ~/.julia/packages/StructuralEquationModels/JhO26/src/observed/EM.jl:78
ERROR: MethodError: no method matching getindex(::Nothing, ::BitVector)
The function `getindex` exists, but no method is defined for this combination of argument types.
Stacktrace:
[1] maybeview
@ ./views.jl:148 [inlined]
[2] copy_per_pattern!(fiml::SemFIML{…}, model::Sem{…})
@ StructuralEquationModels ~/.julia/packages/StructuralEquationModels/JhO26/src/loss/ML/FIML.jl:218
[3] prepare_SemFIML!(semfiml::SemFIML{…}, model::Sem{…})
@ StructuralEquationModels ~/.julia/packages/StructuralEquationModels/JhO26/src/loss/ML/FIML.jl:203
[4] evaluate!(objective::Float64, gradient::Vector{…}, hessian::Nothing, semfiml::SemFIML{…}, implied::RAM{…}, model::Sem{…}, param_labels::Vector{…})
@ StructuralEquationModels ~/.julia/packages/StructuralEquationModels/JhO26/src/loss/ML/FIML.jl:104
[5] evaluate!
@ ~/.julia/packages/StructuralEquationModels/JhO26/src/objective_gradient_hessian.jl:27 [inlined]
[6] evaluate!(objective::Float64, gradient::Vector{…}, hessian::Nothing, loss::SemLoss{…}, model::Sem{…}, params::Vector{…})
@ StructuralEquationModels ~/.julia/packages/StructuralEquationModels/JhO26/src/objective_gradient_hessian.jl:134
[7] evaluate!
@ ~/.julia/packages/StructuralEquationModels/JhO26/src/objective_gradient_hessian.jl:87 [inlined]
[8] #288
@ ~/.julia/packages/StructuralEquationModels/JhO26/src/optimizer/optim.jl:144 [inlined]
[9] (::NLSolversBase.var"#make_fdf##4#make_fdf##5"{…})(G::Vector{…}, x::Vector{…})
@ NLSolversBase ~/.julia/packages/NLSolversBase/n7XXO/src/objective_types/incomplete.jl:55
[10] value_gradient!!(obj::NLSolversBase.OnceDifferentiable{Float64, Vector{Float64}, Vector{Float64}}, x::Vector{Float64})
@ NLSolversBase ~/.julia/packages/NLSolversBase/n7XXO/src/interface.jl:82
[11] initial_state(method::Optim.LBFGS{…}, options::Optim.Options{…}, d::NLSolversBase.OnceDifferentiable{…}, initial_x::Vector{…})
@ Optim ~/.julia/packages/Optim/7krni/src/multivariate/solvers/first_order/l_bfgs.jl:168
[12] optimize
@ ~/.julia/packages/Optim/7krni/src/multivariate/optimize/optimize.jl:43 [inlined]
[13] optimize(f::NLSolversBase.InplaceObjective{…}, initial_x::Vector{…}, method::Optim.LBFGS{…}, options::Optim.Options{…}; inplace::Bool, autodiff::Symbol)
@ Optim ~/.julia/packages/Optim/7krni/src/multivariate/optimize/interface.jl:274
[14] optimize
@ ~/.julia/packages/Optim/7krni/src/multivariate/optimize/interface.jl:264 [inlined]
[15] #fit#283
@ ~/.julia/packages/StructuralEquationModels/JhO26/src/optimizer/optim.jl:143 [inlined]
[16] fit
@ ~/.julia/packages/StructuralEquationModels/JhO26/src/optimizer/optim.jl:105 [inlined]
[17] #fit#259
@ ~/.julia/packages/StructuralEquationModels/JhO26/src/optimizer/abstract.jl:40 [inlined]
[18] fit(model::Sem{SemObservedMissing{…}, RAM{…}, SemLoss{…}}; engine::Symbol, start_val::Nothing, kwargs::@Kwargs{})
@ StructuralEquationModels ~/.julia/packages/StructuralEquationModels/JhO26/src/optimizer/abstract.jl:43
[19] top-level scope
@ ~/Downloads/sem-issue/julia.jl:371
Some type information was truncated. Use `show(err)` to see complete types.
Ah forgot the meanstructure:
imply_ram = RAM(specification = partable, meanstructure = true)
Still no convergence:
Fitted Structural Equation Model
===============================================
--------------------- Model -------------------
Structural Equation Model
- Loss Functions
SemFIML
- Fields
observed: SemObservedMissing
implied: RAM
------------- Optimization result -------------
* Status: failure (reached maximum number of iterations)
* Candidate solution
Final objective value: 1.033510e+02
* Found with
Algorithm: L-BFGS
* Convergence measures
|x - x'| = 1.30e+00 ≰ 1.5e-08
|x - x'|/|x'| = 1.25e-04 ≰ 0.0e+00
|f(x) - f(x')| = 1.07e-02 ≰ 0.0e+00
|f(x) - f(x')|/|f(x')| = 1.03e-04 ≰ 1.0e-10
|g(x)| = 1.01e-01 ≰ 1.0e-08
* Work counters
Seconds run: 24 (vs limit Inf)
Iterations: 1000
f(x) calls: 3123
∇f(x) calls: 3123
This seems unnecessarily complicated. I think there should be a shortcut to set all this and it should be similar to
model = Sem( specification = partable, data = data, meanstructure = true, fiml=true )
What do you think, @Maximilian-Stefan-Ernst ?
I would honestly go down the dispatch route, i.e., if we get data with missingness but no loss and no obs use fiml. Having an argument called fiml does more harm than good in all other cases (i.e. where a loss function is specified or fiml is incompatible with other things) and we can detect this need of users without having them tell us.
I would honestly go down the dispatch route, i.e., if we get data with missingness but no loss and no obs use fiml.
Yes, I would love that! @Maximilian-Stefan-Ernst, this should actually be explained in detail in the tutorial paper that is currently under review.