Philippe Hansen-Estruch

icon indicating a website link https://philippe-eecs.github.io/website/

icon company UT Austin icon location Austin, TX icon location Interested in Generative Modeling, Reinforcement Learning, and Representation Learning

Results 8 comments of Philippe Hansen-Estruch

jax version

Hey, I've implemented a Jax version that I can release soon when I'm finished with my project :)!

optax.MultiSteps out of memory

I am having this issue as well for use in diffusion models

optax.MultiSteps out of memory

you're a king

jax version

Well "soon" is a relative term, but I just released it here! Includes an implementation of MAE and my own method. https://github.com/philippe-eecs/small-vision/tree/main

Comparison with self-conditioning proposed in Analog Bits, and basic two pass sampling baselines

The paper is also missing a cite for DiffAE, which uses a similar method to condition the model on a representation. Trains another diffusion model to sample this representation. Then...

RingFlashAttnFunc forward backward arguments mismatch

Yes I get this error: ``` RuntimeError: function xFuserRingFlashAttnFuncBackward returned an incorrect number of gradients (expected 17, got 13) ```

RingFlashAttnFunc forward backward arguments mismatch

Solved this problem by downgrading flash-attn to 2.6.2

[Bug] Attention to qkv should use QKVParallelLinear

Would TP be needed in the 14B parameter model? Wanted to finetune that on a new VAE I was writing up. Happy to test this for you but wanted to...