tf.sqrt in WGAN-GP gradient penalty formula throws NaNs

[Imperssonator]

I've found in training on my own image dataset that I get an InvalidArgumentError: Nan in summary histogram. This was caused by the following line:

slopes = tf.sqrt(tf.reduce_sum(tf.square(grad_D_X_hat), reduction_indices=red_idx))

You'd think that the sum of squared values would never give anything less than 0, but apparently it does, because adding 1e-8 inside the sqrt() fixed the problem:

slopes = tf.sqrt( 1e-8 + tf.reduce_sum(tf.square(grad_D_X_hat), reduction_indices=red_idx))

[chunhanl]

I faced the same problem while training mnist with WGANGP, this solutions works fine for me. Thanks~

[hujinsen]

@Imperssonator faced the same problem too, that's a werid problem, thanks for saved my life!