atulkum
Why is the model outputting UNK tokens? Shouldn't it be able to point to unkown words from the input?
From: https://github.com/abisee/pointer-generator/blob/master/beam_search.py#L111
In decoding words, we change the token id to the unkown id if t<vocab.size(). So if the decoder is pointing to that particular token it produces [UNK] in output. Is it correct? Following the paper it seems that the decoder should be able to point to that token and copy it, instead of copying the unknown token. I think it's the whole purpose of the pointer-generator model to handle oovs. But from some experiments in decoding I see that the models often outputs some unknown tokens. I tried replacing the 50k vocabulary to the full vocabolary but I get cuda device asserted errors.
There is still a UNK token in the fixed vocab. Pointer-generator can reduce the UNK outputs, but cannot prevent model from outputting UNK. In some timesteps, if the p_gen is high and the probability of the UNK of the fixed vocab is large, the model will still output UNK.
Pointer-generator use the same vocab at encoder/decoder input and decoder target, and the vocab is constructed from train-data (no val/test-data). If the vocab can cover all words of corpus (train,val,test), the UNK token can be excluded.
By using two vocabs (source vocab and target vocab), UNK token may be excluded in target vocab.