How can one un-BERTify when last_four_layers_embedding = True?

[yu202147657]

These two lines are added when last_four_layers_embedding=True:

# cast layers to a tuple and concatenate over the last dimension
cat_hidden_states = torch.cat(tuple(last_four_layers), dim=-1)

# take the mean of the concatenated embedding vectors over the token dimension
text_embedding = torch.mean(cat_hidden_states, dim=1)  

However, once you take the mean, how would you recover the original text from the embeddings produced?