onnx
Inference-TensorFlow-Bert-Model-for-High-Performance-in-ONNX-Runtime Requires Update
Inference-TensorFlow-Bert-Model-for-High-Performance-in-ONNX-Runtime (https://github.com/onnx/tutorials/blob/master/tutorials/Inference-TensorFlow-Bert-Model-for-High-Performance-in-ONNX-Runtime.ipynb) requires an update. As-is, the following steps fail:
Step 2 - Optimize Model: Does not include the step to wget BertOnnxModelKeras.py is not included, which leads to failure in the next part of the step iin bert_model_optimization.py when importing it on line 36.
The second part of Step 2 also fails when running bert_model_optimization.py, as the example currently reads python bert_op_scripts/bert_model_optimization.py --input <bert.onnx> --output <bert_cpu.onnx> --framework tensorflow, which returns an error that "framework" is not an attribute of the script any longer. The correct attribute name for this argument is now --model_type, which is found in the function parse_arguments in the script.
Step 3 - Inference the Optimized Model with ONNX Runtime: Running the step after installing onnxruntime-gpu appears to fail with any version of CUDA greater than 9.1, giving the error: "UserWarning: Cannot load onnxruntime.capi. Error: 'libcublas.so.9.1: cannot open shared object file: No such file or directory'." This is fixed by installing onnxruntime-gpu version 1.1.0.
This suggests that only specific versions of CUDA are compatible with the libcublas.so files the script attempts to access - having clarity on which versions of CUDA are supported with onnxruntime would be ideal, however, perhaps once the example as a whole is updated, this will no longer be an issue.
The final part of Step 3 to "get the perf numbers from TensorFlow model" fails as well, returning, in my attempts at reproducing the example, [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Non-zero status code returned while running SkipLayerNormalization node. though I've seen similar errors with varying Layer names. Attempting to modify the shapes of the values in the "inputs" dict to be 3-dimensional yields [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Invalid rank for input: segment_ids:0 Got: 3 Expected: 2 Please fix either the inputs or the model. I have personally yet to find a solution to this final step, rendering completion of the example unachievable.
@agregory96, thanks for the comment.
I have a notebook for end to end run: Inference TensorFlow Bert Model with ONNX Runtime on CPU.
You can follow this notebook for OnnxRuntime GPU inference. Note that --model_type bert_keras shall be added for bert_model_optimization.py when you copy the commands.
