Hi-archer

Results 5 comments of Hi-archer

WARNING: tokenization mismatch 139 vs. 141

I have the same error as you, whether you used the author's cleaning code and then generated sharegpt_split.json, and reported the error during training with sharegpt_split.json data?

WARNING: tokenization mismatch 139 vs. 141

> > I have the same error as you, whether you used the author's cleaning code and then generated sharegpt_split.json, and reported the error during training with sharegpt_split.json data? >...

WARNING: tokenization mismatch 139 vs. 141

> > > > I have the same error as you, whether you used the author's cleaning code and then generated sharegpt_split.json, and reported the error during training with sharegpt_split.json...

Add scaled_dot_product_attention to replace flash attention

> I have set scaled_dot_product_attention as default when the torch 2.0 was installed. It should be as efficient as original. I used this code. It worked very well and also...

[Bug]: When running gemma2 7b, an error is reported [rank0]: RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasGemmEx( handle, opa, opb, m, n, k, &falpha, a, CUDA_R_16BF, lda, b, CUDA_R_16BF, ldb, &fbeta, c, CUDA_R_16BF, ldc, compute_type, CUBLAS_GEMM_DEFAULT_TENSOR_OP)` Set up according to the prompts: os.environ['VLLM_ATTENTION_BACKEND'] = 'FLASHINFER' print("Environment variable set for VLLM_ATTENTION_BACKEND:", os.getenv('VLLM_ATTENTION_BACKEND'))

> I also got segmentation error with flashinfer=0.0.8 after some requests. > > > One should install FlashInfer manually @orderer0001 > > But, currently it can only use one GPU....