Consistent-DynamicNTKRoPE icon indicating copy to clipboard operation
Consistent-DynamicNTKRoPE copied to clipboard
verified publisher icon NormXU

Does consistent of Dynamically Scaled RoPE require sft?

Open dengxiaotian123 opened this issue 2 years ago • 1 comments

Does consistent of Dynamically Scaled RoPE require sft?

dengxiaotian123 avatar Nov 22 '23 11:11 dengxiaotian123

@dengxiaotian123 Yes. From my experiences, SFT the LLM with consistent dynamically scaled RoPE can improve the long context understanding.

NormXU avatar Nov 23 '23 03:11 NormXU