mozilla
Make model convertible by CoreML
It would be wonderful if DeepSpeech models could be converted to CoreML, for offline use in apps. Here is documentation to do just that. https://developer.apple.com/documentation/coreml/converting_trained_models_to_core_ml Thanks!
Hmmm, so I would probably need to write a custom conversion tool, like it says at the bottom of the page, I guess.
@MatthewWaller I'd guess so. Which seems like a large outlay in time and knowledge.
Just as an update, I'm examining this Keras model converter script, which comes from Apple's own Python CoreML tools. Could be a good precedent for defining the needed layers. You're right though @kdavis-mozilla, looks like a large project.
I think it might be possible to convert with a third party tool. I haven’t written the python conversion scripts myself, but this could be useful (https://github.com/Microsoft/MMdnn/blob/master/README.md). But that’s only half the battle. Then I need to find out how to preprocess the audio, so I’m trying to find out how to get MFCC in Swift. One developer used a C library to do this in iOS, so that might be the way to go.
@MatthewWaller I lack context here, but we have MFCC computation in C already, can't you leverage that?
If I want to use any C libraries I have to port them over to objective c or swift to use them in iOS or macOS. And that’s something I haven’t done yet, and I would prefer to do the calculations all in swift, for longevity sake
@MatthewWaller I came accross https://github.com/tf-coreml/tf-coreml while looking at some tensorflow lite stuff, isn't it already addressing what you want to do?
It does! I hadn’t seen that one. Well, hopefully we just need to get the MFCC one way or another. I’ve got a couple of projects in the hopper before I get back o this one, but that’s exciting!
@lissyx I managed to feed mfcc into a core data model, but I'm not sure where to go to implement the link you sent to convert to coreml, specifically, I'm not sure where to find a list of output tensor names present in the TF graph, (the README.md gives an example of output_feature_names = ['softmax:0'])
Any ideas? Would welcome your help as well @kdavis-mozilla !
Hm I remember documenting that to someone else needing to access some intermediate tensor, on discourse. You should have a look there, I cannot search for it for the moment, I'll try and find it tomorrow if you don't find :-)
@MatthewWaller Any news on that ? The upcoming #1463 might benefit from such support
@lissyx unfortunately I haven't been able to convert to CoreML. The https://github.com/tf-coreml/tf-coreml, which Apple also recommends officially, cannot handle cycles. I tried and got the error, and as a limitation is states: "TF graph must be cycle free (cycles are generally created due to control flow ops like if, while, map, etc.)"
Not sure how to get around this at present. You can see my issue here: https://github.com/tf-coreml/tf-coreml/issues/124
The author states: "I think the simplest way to deal with such graphs for now is to abstract the weight matrices and bias vectors from pre-trained TF. And then use them to build a CoreML model directly using the neural network builder API provided by coremltools." But I'm not sure how to practically go about that.
@MatthewWaller Did you try CoreML on the PR or on master?
I think some, maybe all, cycles should be removed in the PR.
I tried an earlier version on master. Is there a pre-trained model I could use? I see an alpha in the release from 3 days ago. Would that work?
@reuben Can you give @MatthewWaller a preliminary model for the PR to test CoreML?
@MatthewWaller The alpha release is only for the inference binaries, so far, it does not bundle any model change.
@MatthewWaller a preliminary model can be found here: https://github.com/reuben/DeepSpeech/releases/tag/v0.0.1-alpha
@wshamp I was. I found them to be 'logits:0'. As an update overall, I got the model, but I'm stumped at FailedPreconditionError. Here is the issue I filed with tf-coreml. The full stack trace and my full code for converting is there so far. I haven't heard back yet, but anyone else can troubleshoot as well :)
Hmm my quick google that error seems to indicate an issue with the graph initializing variables not the converter. I hit the same error.
@MatthewWaller The branch stores the decoder state in the graph in the variables previous_state_c and previous_state_h. It's a convenient place to store this state info.
As far as I understand, @reuben correct me if I'm mistaken, in exporting[1] the graph the previous_state_c and previous_state_h should be removed[2] or at least not included.
Maybe the model @reuben provided mistakenly included previous_state_c and previous_state_h?
That blacklist doesn't remove previous_state_{c,h}, but rather makes the freezing process ignore them, since I want them to be variables (not constants) in the final exported graph.
The idea is that before you start feeding audio features and fetching the logits tensor, you have to run the initialize_state op (see the create_inference_graph function in DeepSpeech.py[1]).
In our C++ code we do it inside DS_SetupStream (deepspeech.cc[2]).
[1] https://github.com/mozilla/DeepSpeech/blob/7b873365f8bfffe2ea84dcd34058b537e9095765/DeepSpeech.py#L1718-L1756 [2] https://github.com/mozilla/DeepSpeech/blob/7b873365f8bfffe2ea84dcd34058b537e9095765/native_client/deepspeech.cc#L567
Some other notes: the graph in that URL uses LSTMBlockFusedCell, which is probably not supported by tf-coreml, but the weights are compatible with a normal LSTMCell, so with a bit of massaging on the saver when importing, you can use a static_rnn + LSTMCell.
If you can't workaround the previous_state_{c,h} thing, an alternative is fetching the state and feeding it back every time, eliminating the need for the variable.
static_rnn uses tf.cond OPs when you specify the sequence lengths. If tf.cond OPs are not supported by CoreML, you could try not passing sequence lengths to the RNN. It'll degrade the accuracy, but maybe only by a bit.
Let me know if you run into any other issues.
For sure @lissyx ! Here is the official Google page about being able to convert Tensorflow Lite to CoreML.
Oops, Yep. Here it is @lissyx and @kdavis-mozilla https://developers.googleblog.com/2017/12/announcing-core-ml-support.html