mosaicml
Does it support Preference data (for training Reward / DPO)?
🚀 Feature Request
The preference data looks like this:
{
"chosen":
[
{"role": "user", "content": "abcd"},
{"role": "assistant", "content": "abcef"},
...
],
"rejected":
[
{"role": "user", "content": "abcd"},
{"role": "assistant", "content": "abcef"},
...
]
}
This data is used to train a Reward Model or DPO
I'm wondering if it's possible to use streaming for this kind of situation. And How? Thanks very much.
Yes it should work out of the box. Use MDSWrite to convert your preference data to MDS , and create a streaming dataset out of it. Use json as the encoding method. Let me know if you see any issue.
Yes it should work out of the box. Use MDSWrite to convert your preference data to MDS , and create a streaming dataset out of it. Use json as the encoding method. Let me know if you see any issue.
Thank you for quickly explain, I'll try it.
@ericxsun Wondering, have you tried the @XiaohanZhangCMU suggestion? Did it work?
