implimenting DP-GAN in Keras

[Mohammad-Bakhtiari]

Hi. I'm using TFP and Keras to implement DP-GAN. I define the optimizer as follows:

optimizer = DPRMSPropGaussianOptimizer(l2_norm_clip=args.l2_norm_clip,
                                               noise_multiplier=args.noise_multiplier,
                                               num_microbatches=args.microbatches,
                                               learning_rate=args.discriminator_lr,
                                               decay=args.discriminator_decay)
loss = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.losses.Reduction.NONE)
model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])

I also use model.train_on_batch in Keras to train both generator and discriminator. There is no need to apply noise on the gradients of discriminator when fake images are processed to the best of my knowledge. Therefore, I train the discriminator two times one for real and another for fake images. Accordingly, I change l2_clip_norm and noise_multiplier two times, while to make DP ineffective in applying discriminator gradients for fake images, I set optimizer's parameters using keras.backend as follows:

tf.keras.backend.set_value(model.optimizer.l2_norm_clip, 20.0)
tf.keras.backend.set_value(model.optimizer.noise_multiplier, 0.0)
tf.keras.backend.set_value(model.optimizer.num_microbatches, 1)

I have no problem running the code, and I know TFP has issues in Keras. However, to be sure, I want a second opinion. So, In case you notice any problems or have any suggestions, I'll be grateful to share it with me.

[galenmandrew]

A few points. First the generator never sees user data, so it shouldn't be trained with noise. (Not sure if you were doing that.) The discriminator sees both, fake data created by the generator and real data. What matters is to clip and noise the gradients computed on real user data. Also as described in Algorithm 1 of this paper: https://openaccess.thecvf.com/content_CVPRW_2019/papers/CV-COPS/Torkzadehmahani_DP-CGAN_Differentially_Private_Synthetic_Data_and_Label_Generation_CVPRW_2019_paper.pdf. You should probably also clip (but not noise) the gradients computed on fake data (on line 15) to make sure that the discriminator update isn't dominated by the gradients on fake data.

[Mohammad-Bakhtiari]

Thanks. First of all, I don't inflict any noise on the generator; the only concern is about the discriminator. I think it would be beneficial to clip the gradients computed on fake data, as you mentioned. Separately training discriminator in two steps on artificial and real data can have two issues: 1- the quality of training: Even though two-step training brought acceptable results in my work, I think the simultaneous update of gradients based on both real and fake data will better off the learning process. 2- computation time: The training of discriminator on real data will take over 20 times than training the fake data(I described the procedure before). With 1050Ti, the first call of model.train_on_batch takes 3.8 seconds while the second one takes 0.16 seconds.