New optimizers

[GiovanniCanali]

Is your feature request related to a problem? Please describe. SOAP is a quasi-second-order optimization algorithm that runs AdamW in the eigenbasis provided by Shampoo.

Additional context The code is readily available here: https://github.com/nikhilvyas/SOAP/blob/main/soap.py.

[dario-coscia]

I would make this PR more general to upload new optimisers for PINNs. Torch is quite behind in the update from the PINN community. What do you think @GiovanniCanali ?

Examples include:

• Self-Scaled Broyden and Scaled BFGS: https://arxiv.org/abs/2501.16371
• Shampoo: https://arxiv.org/abs/2406.17748
• SOAP: https://arxiv.org/abs/2409.11321
• ENGD with optimizations: https://arxiv.org/pdf/2505.12149, https://openreview.net/forum?id=bKsZomnmqn&noteId=3aTz8N3dvj, https://arxiv.org/abs/2505.11638

[GiovanniCanali]

I would make this PR more general to upload new optimisers for PINNs. Torch is quite behind in the update from the PINN community. What do you think @GiovanniCanali ?

Examples include:

• Self-Scaled Broyden and Scaled BFGS: https://arxiv.org/abs/2501.16371
• Shampoo: https://arxiv.org/abs/2406.17748
• SOAP: https://arxiv.org/abs/2409.11321
• ENGD with optimizations: https://arxiv.org/pdf/2505.12149, https://openreview.net/forum?id=bKsZomnmqn&noteId=3aTz8N3dvj, https://arxiv.org/abs/2505.11638

Hi @dario-coscia! Yes, let's include new optimizers in this PR. Maybe we can open a discussion to collect suggestions, what are your thoughts about this?

[dario-coscia]

I think we can do everything here :)