About the generation results using all default configs of WoW

[Vonderland]

Thank you for your nice code! It is quite novel to consider sequantial relationships for knowledge selection. I have used your code and pretrained models for WoW seen/unseen inference (under the default setting without changing anyting), and find the results are different from the samples shown in the paper. The metrics are shown as below: seen {'accuracy': 0.27355072463768115, 'kl_loss': 0.3048178, 'knowledge_loss': 1.7232132, 'perplexity': 53.37007, 'rouge1': 0.19250293278265435, 'rouge2': 0.06835883860623956, 'rougeL': 0.1739447536411383, 'total_loss': 6.0052814} unseen {'accuracy': 0.18561958184599694, 'kl_loss': 0.27510664, 'knowledge_loss': 2.3412037, 'perplexity': 82.69272, 'rouge1': 0.16044149682040593, 'rouge2': 0.04274795745293827, 'rougeL': 0.1446172269936499, 'total_loss': 7.0314417} It is a little different from the results you provide in readme.

I have also compared the selection and generated results of the examples in the paper, but they are quite different:

31 (seen). (context) truth ##fully , anything with cheese is the best (knowledge_gt) [CLS] che ##dda ##r cheese _ _ knowledge _ _ it is the second - most popular cheese in the us ( behind mo ##zza ##rella ) , with an average annual consumption of (knowledge_pred) [CLS] bread ##ed cut ##let _ _ knowledge _ _ chicken fingers or chicken tender ##s are an american dish prepared by bread ##ing and deep fry ##ing the pe ##ctor ##alis minor muscle (gt) especially che ##dda ##r cheese ! it ' s the second most popular cheese in the us ! (pred) i love cheese fingers .

and the results from paper are:

knowledge_pred: Pizza is a traditional Italian dish consisting of a yeasted flatbread typically topped with tomato sauce and cheese and baked in an oven. (from “Pizza” Wikipedia page)

pred: i love pizza too ! it ’ s a traditional italian dish consisting of yeasted flatbread typically topped with tomato sauce and cheese

Should I change some config (like beam search size) to get the results closer to the paper?

Looking forward to your reply :)