the douban testset

[heyuanw]

hi, I have questions about douban test data.

1. the pos/neg label is not 1:9 ?
2. I find that some context has all ten neg responses but no pos response, for such case, the denominator will be 0, thus how to calculate p1/10 result by using douban_evaluation.py?
3. I find that the testset len in data.pkl in douban is 6670, not equal to 10000 in test.txt, how to deal with this? is there any special point of these 6670 context?

thanks

[xyzhou-puck]

Hi,

q1. the pos/neg label is not 1: 9? No, it is not 1:9, because the label in the douban test-set is manually annotated.

q2 & q3 Those contexts in test-set that only has positive or negative candidate responses are discarded during evaluation, same to what SMN did.