openai
few shot classification with unknown class
I have a very small dataset i.e. 10 images each class and total 500 classes. During inference, there is also a possibility that an image might come that does not belong to any of these 500 classes, and I have to identify it as well. How do I go about this? I have tried cosine similarity, but it is not very robust in finding the unknown class. Also, due to small images per class, the model overfits.
I solved the problem of unknown class through CLIP model using this approach. I used its image embeddding as well as the text embeddings and added them together [torch.add()] and saved them locally as a .pt file. The text embedding was generated with either some words written on the object or a sentence describing the object. During test time, I generated only image embedding through CLIP and compared the cosine similarity. If it was below a certain threshold, I marked them as unknown or outlier. This gave me 100% accuracy for my usecase. You can use huggingface's openAI clip (giant variant worked best).