Lukasz Kaiser
I think the best idea is to report on the TF and google colab lists as this does not look like an error specific to T2T.
The problem with TF 2.0 at least for now is that when you want speed (use @tf.function or functional Keras mode) you're back in TF 1.0 graph-mode land. With shape...
It took us a while to get familiar with readthedocs, but we do have the API docs there at last! https://trax-ml.readthedocs.io/en/latest/ We're now working on getting the colabs (quick intro...
I created #897 for starters, please take it if you wish :). We also updated the main page README and the trax intro colab -- it has a direct Transformer...
We're working on providing more examples, will update here when there's one for NER.
There are some (enwiki, imagenet) in the Reformer colabs, but we hope to add more, will update here.
I believe the input signature must be a triple of `(sig(inputs), sig(targets), sig(loss_weights))`. Looks like you only added the first part, can you try with all 3?
We are quite busy debugging the current backends for now, but if someone made a PR with a nicely working one, sure!
I believe they should be easy to implement. We don't have them by default yet - a PR adding them would be welcome!
That sounds very good! Another possible place would be here, in the main attention file: https://github.com/google/trax/blob/master/trax/layers/attention.py#L205 Could you first quickly explain your method? I'm not sure I understand 100% if...