lightning-thunder icon indicating copy to clipboard operation
lightning-thunder copied to clipboard
verified publisher icon Lightning-AI

Allocate dQ, dK, and dV as a catted tensor to save a downstream cat in nvFuser.

Open wujingyue opened this issue 1 year ago • 0 comments

The description on the added compile option explains what this optimization does.

This optimization is disabled by default for now. I'll try to enable it by default or even always after #35 is merged and bookend is disabled by default.

wujingyue avatar Mar 23 '24 04:03 wujingyue