Teachable machine pose model work with Tfjs React Native

[danielchung928]

May I know is it possible to use the Teachable machine exported pose model to integrate with the Tensorflow React Native module?

From the example code of Teachable machine pose model predict method:

async function predict() { // Prediction 1: run input through posenet // estimatePose can take in an image, video or canvas html element const { pose, posenetOutput } = await model.estimatePose(webcam.canvas); // Prediction 2: run input through teachable machine classification model const prediction = await model.predict(posenetOutput);

... }

In the React Native world, any idea how to get the "posenetOutput" from the TensorCamera and fit into the Teachable machine pose model predict method?

[irealva]

Hi,

Thanks so much for writing in.Are you trying to replicate our training code but using tfjs-react-native? If so, the pose itself is not the input to the training model, but another activation we're using which is here:

https://github.com/googlecreativelab/teachablemachine-community/blob/master/libraries/pose/src/custom-posenet.ts#L231

You would need to replicate the function we have there to generate the posenetOutput which is a concatenated tensor with the heatmap and offset vectors of posenet (they are the intermediate outputs of posenet before decoding the pose keypoints).

[irealva]

Duplicate of #145

[jdnichollsc]

Hello folks, can we use the camera from React Native with these libraries?

[evanjmg]

Any update on this? Using tf.loadLayersModel(bundleResourceIO(modelJson, modelWeights)); is the best I've seen but can't use teachable machine automatic metadata etc. Btw, it isn't accurate with this technique