kuza55
kuza55
@bzamecnik Thanks for looking into this; I thought this should be resolved since I patched Keras to passed in colocate_gradients_with_ops=True to TensorFlow. I wonder if I missed a place in...
It's merged into keras in the tensorflow backend when it calls tf.gradient On Dec 5, 2017 10:25 PM, "J" wrote: > Hi @kuza55 , where did you pass > colocate_gradients_with_ops=True...
This seems like a bug in Keras, there's really no reason this shouldn't work.
Don't really have a setup to replicate this atm. I would try to create a reduced test case and file a bug against Keras; copying models the way I do...
Sounds like you're hitting a bottleneck somewhere; could be CPU, could be memory bandwidth, really hard to say from over here.
Did you divide your batch size by the number of GPUs you're using? Not really sure what else could be causing problems.
Sorry, I misspoke, what I meant to say was: If you followed the instructions, you probably multiplied your batch size by the number of GPUs you're using. This increases perf,...
This looks like an issue with how Keras serializes/deserializes models; unless you really need to de/serialize the multi-gpu version, I would recommend keeping a copy of the original single GPU...
So, the problem seems to be the hard-coded batch size. The util function assumes that if it takes a slice of the input and passes it into the model, the...
I'm running into some flakiness with mypy too. Blowing away the cache is unnecessary, merely re-running pants check is sufficient. I am running into this both in pre-commit as well...