Elixir crash quitely when loading ONNX model

[GES233]

Background and Description

I wanna build elixir's DiffSinger wrapper with ONNX via Ortex. DiffSinger's ONNX version is supported and can be executed on OpenUTAU currently.

But elixir is crashed whitout crash dump or message when I try to load acoustic model(size: 312 MB) that extraced from OptnUTAU's singer lib. The following is a reproduction of that issue.

Livebook Session

# Run as: iex --dot-iex path/to/notebook.exs

# Title: DiffSinger in Elixir via Ortex

Mix.install([{:ortex, "~> 0.1.10"}, {:nx, "~> 0.8"}])

# ── Model preparation ──

# You should modify path within your model's location.
model_root_path = "<bla bla>/Code/QyEditor/priv/Qixuan_v2.5.0_DiffSinger_OpenUtau"
model_path = fn sub -> Path.join(model_root_path, sub) end
# Variance Model
# pitch_predict_path = model_path.("")
linguisitic_path = model_path.("dsvariance/0816_qixuan_multilingual_multivar.qixuan.linguistic.onnx")
variance_path = model_path.("dsvariance/0816_qixuan_multilingual_multivar.qixuan.variance.onnx")
# Acostic Model
acostic_model_path = model_path.("0816_qixuan_multilingual_acoustic.qixuan.onnx")
# Vocoder
vocoder_path = model_path.("dsvocoder/nsf_hifigan_qixuan_004.onnx")


# ── Identify the inputs and outputs of model ──

# Simplest one is vocoder, it receive mel spectrum and pitch, and puts waveform.

# 装载前，进程的内存占用在 21.8M 左右，代码在 25.6M 左右
vocoder = Ortex.load(vocoder_path)
# #Ortex.Model<
#   inputs: [
#     {"mel",
#      "Tensor {
#        ty: Float32,
#          dimensions: [ 1, -1, 128 ],
#        }",
#      [1, -1, 128]},
#     {"f0", "Tensor {\n    ty: Float32,\n    dimensions: [\n        1,\n        -1,\n    ],\n}",
#      [1, -1]}
#   ]
#   outputs: [
#     {"waveform",
#       "Tensor {\n    ty: Float32,\n    dimensions: [\n        1,\n        -1,\n    ],\n}", [1, -1]}
#   ]>
# 内存占用没有显著变化。


acostic_model = Ortex.load(acostic_model_path)
# raise
# Runtime terminated unexpectedly - no connection

# ── Preparing inputs ──

before loading the larger model, I loaded a tiny version(less than 100 MB) and it success.

Enviornment

Currently, I use my laptop to coding.

Dependencies

• System: Windows 11
• Processor: i5-8250U
• RAM: 8GB
• Elixir version: 1.16.0 and 1.17.0 (via Livebook) with OTP/26
• Rust version: 1.18.2
• Build chain: Visual Studio 2022

Expectations

• figure out why it crashed?
• Is it fixable?
• [PURELY PERSONAL ISSUE, for the sake of my desktop only has Intel Arc] How to add another execution provider like OpenVINO or OneDNN from source?

Screenshot

[mortont]

could you run the same commands outside of livebook to try capturing the crash? Also, while the model is loading, can you monitor the system memory? Typically there are more verbose logs that may help diagnose this.

[GES233]

I hadn't been on Github because I was perparing for an exam finished today, and now I've replicated the problem on my new desktop on Ubuntu 22.04 (WSL):

~ iex -S mix
Erlang/OTP 27 [erts-15.1] [source] [64-bit] [smp:3:3] [ds:3:3:10] [async-threads:1] [jit:ns]

==> diff_singer
Compiling 1 file (.ex)
==> qy_core
Compiling 1 file (.ex)
==> scripts
Generated scripts app
Interactive Elixir (1.17.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> model_path = "/home/ges233/Projects/TinyMIDIEditor/pico_editor/priv/Qixuan_v2.5.0_DiffSinger_OpenUtau"
"/home/ges233/Projects/TinyMIDIEditor/pico_editor/priv/Qixuan_v2.5.0_DiffSinger_OpenUtau"
iex(2)> add = fn sub -> Path.join(model_path, sub) end
#Function<42.39164016/1 in :erl_eval.expr/6>
iex(3)> acostic_model_path = add.("0816_qixuan_multilingual_acoustic.qixuan.onnx")
"/home/ges233/Projects/TinyMIDIEditor/pico_editor/priv/Qixuan_v2.5.0_DiffSinger_OpenUtau/0816_qixuan_multilingual_acoustic.qixuan.onnx"
iex(4)> model = Ortex.load(acostic_model_path)
Segmentation fault (core dumped)

The session is from a repo that I want to build an elixir's diffsinger wrapper.

And there're a not-very-obvious memory changes founded through the task manager.

1. when I type the command in iex to load the model
2. Elixir crashed with Segmentation fault (core dumped)

I will continue to try to locate whether the problem is due to Ort or Ortex and will append some stuff if I find anything new.