Question about the loss calculation of pet

[Chunngai]

Hello.

I am not clear about the loss calculation of pet during training (as described in Section 3.1 in the paper: Exploiting Cloze Questions for Few-Shot Text Classification and Natural Language Inference).

Here's my understanding (based on the Yelp dataset). Assume that:

• PLM = BERT
• P = It was [MASK]. [X]
• V = {1: "terrible", 2: "bad", 3: "okay", 4: "good", 5: "great"}

And that:

• X = The movie is interesting.
• y = 5

Based on my understanding, the loss is calculated as follows:

1. Create the prompt: P(X) = It was [MASK]. The movie is interesting.
2. Feed P(X) into PLM, and PLM outputs the probability of each token in the vocabulary: probs = [p_1, p_2, ..., p_N], where N is the vocab size and len(probs) == N.
3. Take the probability of the five words corresponding to the five labels: probs_of_five_labels = [p_{terrible}, p_{bad}, p_{okay}, p_{good}, p_{great}] (here len(probs_of_five_labels) == 5)
4. Apply softmax: normalized_probs_of_five_labels = softmax(probs_of_five_labels).
5. Use cross_entropy to calculate the loss between normalized_probs_of_five_labels and the one-hot vector of the label one_hot = [0, 0, 0, 0, 1]: loss = cross_entropy(probs_of_five_labels, one_hot)

Am I right?

[Chunngai]

Additionally, I could not find the loss calculation code. Could you kindly tell me which file stores the loss calculation code so that I can better understand the calculation process?

[cylnlp]

I wonder how it works when we have multiple verbalizers... Given a V = {[1: "bad", "terrible"];[2: "good", "excellent"]}, and an input "The movie is interesting." with label "2". How would Pet be trained? Would it augment one training example into "It was good. The movie is interesting." and "It was excellent. The movie is interesting."?

[Faith-Uchiha]

Additionally, I could not find the loss calculation code. Could you kindly tell me which file stores the loss calculation code so that I can better understand the calculation process?

Hi, have you figured out how where the calculation code is ? i guess the writer put it in some helper, but i'm still confused