Vincent Roulet comments

Results 102 comments of


                                            Vincent Roulet

Memory overflow using scale_by_radam

Right, thank you for catching this @carlosgmartin ! @HGangloff feel free to reopen if the matter was not solved but I think it should be good.

Intended usage of the Sophia optimiser

Thanks for catching this. You are absolutely right. The implementation of recent optimizers in the contrib folder are the responsibility of the authors adding them.

Memory leak in optax.radam

Hello @lukekulik, Thanks for catching this and giving a reproducible example. I think there is a simple fix for this: replace the jax.lax.cond in [here](https://github.com/google-deepmind/optax/blob/dd1dac479e8bb49f58fec98c0c34316cba9d2c99/optax/_src/transform.py#L764) by a jnp.where. So that...

Memory leak in optax.radam

Fixed by #974

Fix memory leak in radam optimizer

Perfect thank you again!

Performance issue: multi_transform and set_to_zero don't prevent computation

Hello @YanisJouanaud, Sorry for the delay on this one. I think the issue is that even if the optimizer is not used, its state gets transferred. The fast speed of...

Performance issue: multi_transform and set_to_zero don't prevent computation

Hello @YanisJouanaud, Can you tell me if this solved your issue?

Performance issue: multi_transform and set_to_zero don't prevent computation

I'm closing this issue, @YanisJouanaud feel free to reopen.

Feature request: GaLore optimizer

Hello @gil2rok, Thanks for pointing this out! Would you be willing to contribute with an implementation? If not, we'll add it to our todo list but any contributions are super...

Feature request: GaLore optimizer

That's great news! Yes, the workflow currently would be to 1. add this optimizer in the contrib folder, note that the PR you pointed out introduced some errors (like in...