Frozen parameters in GaussianFourierProjection

[vvvm23]

Hi, just a beginner with diffusion models and have been using your implementations as reference. I have a question about this class

Why is requires_grad set to false in the weight parameter? Won't this mean, during training, the noise level embeddings won't be updated?

Thanks!

[patil-suraj]

cc @patrickvonplaten

[patrickvonplaten]

Hey @vvvm23,

It's set to False because we don't want to train those parameters. I followed the implementaton of the original model here: https://github.com/yang-song/score_sde_pytorch/blob/1618ddea340f3e4a2ed7852a0694a809775cf8d0/models/layerspp.py#L37

Does this make sense?

[vvvm23]

Hi @patrickvonplaten

I somewhat misphrased my original question, I'm aware setting requires_grad to False prevents that particular parameter from accumulating gradients, essentially stopping the training of those parameters.

But why would we not want to train the noise level embeddings? Or is this just a simple, fixed (albeit randomly initialised) projection from a per-batch noise value to a different space, which would later have some learned transformation applied to it?

Thanks!

[patrickvonplaten]

Hey @vvvm23,

sinusoidal position features like GaussianFourierProjection don't need training because every embedding already has a distinctly different vector that the model can use a "cue" to know what time position has been passed to it.

If one wants to train position embedding vectors (or time embedding vectors here), one can just randomly initialize such a vector and let the model learn it. If however we use sinusoidal embeddings, there is no need to learn it

[vvvm23]

Okay thank you @patrickvonplaten ! That explanation makes a lot of sense~