pytorch
Add support for Normalized SGD (NSGD) with Perturbation
🚀 Feature
Add Normalized SGD (NSGD) with Perturbation support to Opacus with NormalizedSGDPOptimizer that uses gradient normalization instead of hard clipping, providing better convergence properties for non-convex optimization problems.
Motivation
Standard DP-SGD uses hard clipping which can lead to gradient bias and suboptimal convergence, especially in non-convex settings. Normalization-based methods address this by normalizing gradients rather than clipping them, which can provide better gradient approximation and improved convergence properties for non-convex optimization problems.
Pitch
Implement NSGD based on "Differentially Private Non-Convex Optimization": https://arxiv.org/pdf/2206.13033
- Per-sample gradient normalization (not hard clipping)
- Normalization formula:
g_normalized = g / (r + ||g||)wherer > 0is a regularization parameter - Better convergence properties for non-convex optimization compared to clipping-based methods
- Reduces gradient bias introduced by hard clipping thresholds
Usage
privacy_engine = PrivacyEngine()
model, optimizer, dataloader = privacy_engine.make_private(
module=model,
optimizer=optimizer,
data_loader=dataloader,
noise_multiplier=1.0,
max_grad_norm=0.1, # Used as regularization_param if regularization_param not provided
clipping="normalized", # New option
regularization_param=0.1, # Optional: explicit regularization parameter
)
Alternatives
- Existing
DPOptimizer(hard clipping): Uses hard clipping threshold which can introduce bias and suboptimal convergence in non-convex settings - Existing
AdaClipDPOptimizer: Adapts a single global clipping norm over time, but still uses hard clipping which can bias gradients
Additional context
I'm happy to submit a PR for this feature if it aligns with the project's goals. I'm implementing Normalized-SGD (NSGD) with Perturbation as part of my thesis work, and I believe making it available in the main library could benefit the broader Opacus community.
Hi @ParthS007, thank you for suggesting this feature. We recommend implementing it as part of the research folder. The goal of the research folder is to provide implementation of new techniques that may not be established as SOTA yet, but have such potential.
