[React Native Inference Session Initialization With Buffer Error] "Error: Uint8Array is not supported"

[jackylu0124]

Describe the bug

I tried to initialize an inference session with a Uint8Array representation of a very simple .ort model (see screenshot of the .ort file viewed in Netron below), but it gave me the [Error: Uint8Array is not supported] error. The Uint8Array from my original code is from my database, but for the purpose of easier reproduction and isolation of the issue, I have included a .txt file that contains the content of the .ort file in hex code (see https://github.com/jackylu0124/ort-react-native-buffer-issue/blob/main/onnx_hex_files/LinearBlend_v001.txt), and the string of the hex code content (see line 46 in App.tsx) is parsed into a Uint8Array buffer before it's being passed into ort.InferenceSession.create(). I have also tried passing in a raw ArrayBuffer rather than a Uint8Array buffer, but it didn't work either. According to the documentation (https://onnxruntime.ai/docs/api/js/interfaces/InferenceSessionFactory.html), both should work.

I am currently running the app on Android Studio's emulator on Windows.

P.S. I have verified that the hex code content in the .txt file is correct by parsing it and converting it to bytes in Python and was able to use it to initialize an inference session successfully using ONNX Runtime's Python API (but the same doesn't work for ONNX Runtime's React Native API, and hence leading to this GitHub issue).

Urgency

High urgency, this is a blocking issue in my project.

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
• ONNX Runtime version: "onnxruntime-react-native": "^1.12.1" (in package.json)

To Reproduce

Link to repo with minimal code for error reproduction: https://github.com/jackylu0124/ort-react-native-buffer-issue Please follow instructions on https://reactnative.dev/docs/environment-setup in order to set up and run the project with Android Studio's emulator.

Once the project is set up and running, you can click/press the "CREATE INFERENCE SESSION WITH BUFFER" button on the top of the screen, which will try to initialize an inference session with the Uint8Array buffer representing the ONNX model, and the error will be logged into the console. (see Gif below)

Expected behavior

The inference session should be initialized successfully with no error messages emitted since initializing inference session with Uint8Array buffer or raw ArrayBuffer is supported as stated in the documentation. (https://onnxruntime.ai/docs/api/js/interfaces/InferenceSessionFactory.html).

Screenshots

The .ort Model that I am using (it's a very simple model for linear-blending two inputs e.g. blend = (1 - alpha) * img1 + alpha * img2) Error Message Gif

[fs-eire]

The function is not implemented yet because the current react native data transfer bridge is not efficient to pass a buffer - they have to be converted into a string ( using BASE64 or whatever encoding ). The recommended way is to put the model into the resource folder and load it as static resource.

Here is an example of how to do this by using expo-asset.

[jackylu0124]

Hi @fs-eire, thank you very much for the insight! I tried using Buffer.from(linear_blend_model_uint8_buffer).toString("base64") to get the BASE64 string representation of my Uint8Array buffer and passed the BASE64 string into the ort.InferenceSession.create(linear_blend_model_base64) function, but the inference session creation still failed (with the error mesage [Error: Can't load a model: No content provider: THE_BASE64_STRING_THAT_I_PASSED_IN_TO_CREATE]). I have also updated the code in my linked repo above to include this change.

The reason I want to initialize the inference session with an array buffer is because directly storing the ONNX models introduces security risk because people would then be able to access the ONNX model files on the device. One strategy I have come across is to encrypt the array buffer representation of the ONNX model file and then descrypt it before passing the array buffer into the inference session's create() function (see this link for more detailed discussion (https://github.com/microsoft/onnxruntime/issues/3556)[https://github.com/microsoft/onnxruntime/issues/3556]). Do you by chance know if there are any similar ways to encrypt the models or any solutions to this particular issue that I am describing?

Additionally, from my testing, using the Buffer module for the data transfer bridge is fairly fast, could you guys see if that could be implemented also?

Thanks a lot for the help again!

[fs-eire]

Hi @fs-eire, thank you very much for the insight! I tried using Buffer.from(linear_blend_model_uint8_buffer).toString("base64") to get the BASE64 string representation of my Uint8Array buffer and passed the BASE64 string into the ort.InferenceSession.create(linear_blend_model_base64) function, but the inference session creation still failed (with the error mesage [Error: Can't load a model: No content provider: THE_BASE64_STRING_THAT_I_PASSED_IN_TO_CREATE]). I have also updated the code in my linked repo above to include this change.

The reason I want to initialize the inference session with an array buffer is because directly storing the ONNX models introduces security risk because people would then be able to access the ONNX model files on the device. One strategy I have come across is to encrypt the array buffer representation of the ONNX model file and then descrypt it before passing the array buffer into the inference session's create() function (see this link for more detailed discussion (https://github.com/microsoft/onnxruntime/issues/3556)[https://github.com/microsoft/onnxruntime/issues/3556]%5Bhttps://github.com/microsoft/onnxruntime/issues/3556%5D)). Do you by chance know if there are any similar ways to encrypt the models or any solutions to this particular issue that I am describing?

Additionally, from my testing, using the Buffer module for the data transfer bridge is fairly fast, could you guys see if that could be implemented also?

Thanks a lot for the help again!

could you share some pointers to the Buffer for data transfer bridge? From React Native doc the supported types are really limited.

[jackylu0124]

Yeah sure, I don't know if these are the ones you are referring to, but I find these resources on ArrayBuffer and the various kinds of TypedArray helpful:

1. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/ArrayBuffer
2. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/TypedArray
3. https://javascript.info/arraybuffer-binary-arrays

Thanks for the help again!

[jackylu0124]

Hi @fs-eire,

I have added my sample implementations for initializing inference session with Uint8Array and ArrayBuffer for React Native on Android platform to the error reproduction repo (same as the one linked above: https://github.com/jackylu0124/ort-react-native-buffer-issue).

I have added two different implementations for passing ONNX model buffer data between React Native and Native Java code, and have verified that they load and run inference correctly. One of them uses base64 string encoding and decoding for data passing (please see the BufferInitializationTestApp_string_fix/ folder) while the other one uses JavaScript's Array to React Native Java's ReadableArray conversion for data passing (please see the BufferInitializationTestApp_array_fix/ folder). In addition to the main app entry file App.tsx, the most important files that I have made changes to are node_modules/onnxruntime-react-native/android/src/main/java/ai/onnxruntime/reactnative/OnnxruntimeModule.java, node_modules/onnxruntime-react-native/lib/backend.ts, and node_modules/onnxruntime-react-native/lib/binding.ts. You can find my changes by searching for comments that start with // JL. Note that I only modified the TypeScript files and not the JavaScript files in the other folders because I am working with a TypeScript project, and I also didn't rebuild the entire React Native package. I only modified the aforementioned files for the sake of demonstration.

Notes on passing ONNX model data buffer with base64 string encoding and decoding

1. This approach is in fact how the current package handles passing tensor inputs from React Native JavaScript to Native Java code. Since the package is already doing that for passing tensor data, I don't see a reason why the same can't be done for passing the ONNX model data buffer.
2. For the sake of simplicity, I used the base64 string encoding itself as the session key, but there might be better ways for creating the session key.

Notes on passing ONNX model data buffer with JavaScript's Array to React Native Java's ReadableArray conversion

1. For the sake of simplicity, I used the joined "stringified" content of the passed array separated by commas for the session key, but there might be better alternatives for creating the session key.
2. Performance/efficiency of the data transfer brige should not be a concern here because I have tested iterating over an array of 1 billion elements (corresponding to the scenario where the ONNX model is roughly 1 GB), and it still finishes very quickly.
3. In the case of implementation for the native code (Objective C) for iOS (which I didn't add code for), we would use NSArray, and the same implementation logic still applies because the node_modules/onnxruntime-react-native/ios/OnnxruntimeModule.mm file includes the C++ runtime header file <onnxruntime/onnxruntime_cxx_api.h>, which would then allow us to initialize an inference session with the ONNX model data buffer using the C++ API.
4. For passing array from React Native JavaScript to both the Android native Java code and the iOS native Objective C code, I found this StackOverflow post quite helpful: https://stackoverflow.com/questions/55514888/how-to-convert-react-js-array-to-nsarray-in-ios-and-string-in-android.

Finally, is there any chance you guys could please add official support to the onnxruntime-react-native package for initializing inference session with Uint8Array and ArrayBuffer on both iOS and Android platforms? I would really appreciate your help and the addition of this feature because as mentioned earlier, directly storing and loading the .ort model on the mobile device presents great security risks, and buffer storing and loading will greatly mitigate these issues. Thank you very much for your help again and please don't hesitate to let me know if you have any questions.

[jackylu0124]

Hi @fs-eire,

I saw your pull request, thank you very much for the implementation! Would you please be able to make the fix for the iOS platform as well? The same logic should apply, and an iOS implementation would be truely appreciated! Thank you so much for your work again!!

[jackylu0124]

Hi @fs-eire,

Is there any update on this? Do you by chance have a timeline on when the model loading via buffer feature will be supported in the NPM onnxruntime-react-native release for both Android and iOS? Thanks a lot again!

[fs-eire]

@skottmckay @YUNQIUGUO

[jackylu0124]

Hi @fs-eire, @skottmckay, @YUNQIUGUO,

A gentle ping, is there any update on this by any chance? Thanks a lot in advance!

[YUNQIUGUO]

Hi @jackylu0124 , we've been working on supporting to load from ArrayBuffer on the iOS side. will send out updates when finished.

[jackylu0124]

Hi @YUNQIUGUO,

Got it, thank you very much for the update! To confirm, the Android side is also still in the works and hasn't been integrated into the dev release right? Thanks for the help again!

[jackylu0124]

Hi @fs-eire , @skottmckay , @YUNQIUGUO ,

Is there any update on this on both the Anroid side and also the iOS side by any chance? I would also really apprecaite any estimated timeline on this fix. Thanks a lot again!

[jackylu0124]

Hi @fs-eire, @skottmckay, @YUNQIUGUO,

Sorry about pinging again, is there any update on this by any chance? Thanks a lot in advance!

[YUNQIUGUO]

Hi @fs-eire, @skottmckay, @YUNQIUGUO,

Sorry about pinging again, is there any update on this by any chance? Thanks a lot in advance!

Sorry for the late reply!

For Android side pr, it's open as there's some pending implementation detail (regarding passing in base64 encoded buffer/UInt8Array, etc. ). I will help do the changes and help check in the pr hopefully this week.

For iOS side, so sorry lately haven't had enough time focusing on that. still be in progress. hopefully will send out a pr next week.

For all changes to be supported in official ort react native package, it will be included in 1.14 release (planned till next year). but once it's checked in the main repo, you should be able to build from source and try at least. Thanks!

[jackylu0124]

Hi @YUNQIUGUO,

No worries and thank you very much for the update! Do you by chance know approximately when will the changes be checked into the main repo and approximately when will 1.14 be officially released next year? Also once the changes are checked into the main repo, is there any chance we could please have a dev build (like the ones circled in red in the screenshot below) on the NPM page? I would really really appreciate that! Thank you very much again!

[YUNQIUGUO]

Hi @jackylu0124!

Sorry for the late reply.

Also once the changes are checked into the main repo, is there any chance we could please have a dev build (like the ones circled in red in the screenshot below) on the NPM page?

AFAIK our CI produces dev version npm package on a regular basis already, so once the changes are checked in, should have the npm dev package ready to try.

when will the changes be checked into the main repo and approximately when will 1.14 be officially released next year?

Regarding 1.14 release, it is planned early February next year. Just sent out the pr for iOS support.

[jackylu0124]

Hi @YUNQIUGUO,

No worries and sorry for the late reply from my side as well. And thank you so much for the update and the work on making this feature possible! I really appreciate it.

For the Android support, has it been officially merged into the latest 1.14.0-dev.20221124-4ca62b9ee8 release (https://www.npmjs.com/package/onnxruntime-react-native/v/1.14.0-dev.20221124-4ca62b9ee8)? I see a red cross mark (see screenshot below) for the Android pull request and am wondering if the Android pull request is officially in now or is it still waiting to pass some tests.

Thank you so much for all your help again!