evalutils Add models to the algorithm templates

Request from Erdi:

it would also be super cool if I could upload a tensorflow/pytorch model directly?

Maybe we can do something with ONNX?

I don't have experience with ONNX but I don't have great experiences with MMDNN. I think supporting tensorflow and pytorch would be somewhat easy.

I think the first step would be to add support in evalutils by having a place to drop your model in the templated repo, then we can see how it would work, then integrate if it's good

Sep 24 '20 10:09 jmsmkn

@jmsmkn That's a good suggestion. As you say it should be easy to just support PyTorch and TensorFlow models for this for now. I could create a quick mockup for PyTorch when I find some time. I guess we have to carefully think about the interface for the template.

Sep 24 '20 12:09 silvandeleemput

Would be really good to know if ONNX could be used, so that we do not have to maintain support for all of the frameworks: https://onnx.ai/supported-tools.html#buildModel

Sep 24 '20 13:09 jmsmkn

Ok, let me do some research on ONNX first.

Sep 24 '20 19:09 silvandeleemput

We now use ONNX Runtime (CPU only) for bodyct-multiview-nodule-detection and this works fine, also model/weigths conversion from PyTorch to ONNX format is very easy to do (probably very similar for TensorFlow). Haven't tested it for the GPUs yet.

The only caveat I found for using ONNX Runtime (CPU mode) on grand-challenge is that you must explicitly specify the CPU affinities, since it has no permissions for automatic affinity resolution there. See the following code for creating an onnxruntime.InferenceSession from a ONNX model file, which accounts for this: https://github.com/DIAGNijmegen/bodyct-multiview-nodule-detection/blob/7a6fd7e0590eeeeecf4cee2afa032e0bdeeaeeff/packages/onnxruntime_utils.py

May 13 '21 07:05 silvandeleemput

Templating will be removed from this repo as that has moved to https://github.com/DIAGNijmegen/rse-grand-challenge-forge.

Aug 05 '24 19:08 jmsmkn