slamandar

Results 5 comments of slamandar

Bert-large model not attaining ~65% accuracy even after training till 52k timesteps!

> We tuned the bert model very carefully with a very small lr (3e-6), but the price to pay is that it has to be trained for much longer. I...

Repo level concatenation of data

Regarding repo-level concatenation, I have a related question. In a batch, one sample may contain multi docs from different files, such as repo_a/file_a and repo_a/file_b. When concatenating these files into...

Repo level concatenation of data

Thank you for your prompt and detailed response! One last question. In one sample, is there a need for special token between the concatenated files? So that the model can...

Repo level concatenation of data

Completely understand. Thanks again for your quick response!

Was the base frequency increased, or do you rely on position interpolation via scaling?

I have the same confusion