pierlj
K-shot Sample Stratagy for Novel Classes
Hi. How do you sample K examples for novel class? Can you describe the stratagy detaily? And the sampling stratagies are the same for different dataset (natural and aerial)?
Hi, thanks for your interest in this work.
Specifically, a table is created before training to list the ids of all images which have at least an instance of each class. This allows drawing K examples for each class during training. The support images for base classes are picked randomly from this table each time we need to compute support features. However, for novel classes, only K examples are available overall and only K examples are kept for the whole training. This is implemented by setting the same seed in a PyTorch RNG before sampling of the support images. This is an overcomplicated way to ensure that the same K examples will always be drawn. A much simpler approach would be randomly keeping only K ids in the class table when it is built.
Regarding the exact number of examples, this is a tricky question as many works leverage all annotations from K support images. This way, the number of annotated instances of the novel classes may not be constant. In this work, we keep only one annotation for each support image, ensuring to have only K annotated instances of each class. However, this may provide wrong supervision during novel class training as actual objects will be classified as background simply because annotation is not available.
Hi, thanks for your interest in this work.
Specifically, a table is created before training to list the ids of all images which have at least an instance of each class. This allows drawing K examples for each class during training. The support images for base classes are picked randomly from this table each time we need to compute support features. However, for novel classes, only K examples are available overall and only K examples are kept for the whole training. This is implemented by setting the same seed in a PyTorch RNG before sampling of the support images. This is an overcomplicated way to ensure that the same K examples will always be drawn. A much simpler approach would be randomly keeping only K ids in the class table when it is built.
Regarding the exact number of examples, this is a tricky question as many works leverage all annotations from K support images. This way, the number of annotated instances of the novel classes may not be constant. In this work, we keep only one annotation for each support image, ensuring to have only K annotated instances of each class. However, this may provide wrong supervision during novel class training as actual objects will be classified as background simply because annotation is not available.
Hi, thanks for your interest in this work.
Specifically, a table is created before training to list the ids of all images which have at least an instance of each class. This allows drawing K examples for each class during training. The support images for base classes are picked randomly from this table each time we need to compute support features. However, for novel classes, only K examples are available overall and only K examples are kept for the whole training. This is implemented by setting the same seed in a PyTorch RNG before sampling of the support images. This is an overcomplicated way to ensure that the same K examples will always be drawn. A much simpler approach would be randomly keeping only K ids in the class table when it is built.
Regarding the exact number of examples, this is a tricky question as many works leverage all annotations from K support images. This way, the number of annotated instances of the novel classes may not be constant. In this work, we keep only one annotation for each support image, ensuring to have only K annotated instances of each class. However, this may provide wrong supervision during novel class training as actual objects will be classified as background simply because annotation is not available.
Yeah, the sampling is an overcomplicated problem. Thanks for your patient reply. But I'm sorry to say that I'm still confused about the technique details. Is the method you described included in your code? If not, would you like to share the sampling code?
Yes it is included in this repository, most of the related code is inside data/data_handler.py, data/task_sampling.py and data/example_selector.py.
The DataHandler class is a master class that manages most of the data processing, from the sampling of the example with the ExampleSelector class to the selection of the classes for each episode with the TaskSampler. Note that the actual sampling procedure is done inside a sampler class: SupportSampler in data/samplers/fs_samplers.py but the filtering of the annotations is done directly into the dataset class when each image is loaded.
I agree the sampling is complicated, yet the way I implemented it inside this repository could be improved.
the sampling is complicated, yet the way I implemented it inside this repository coul
Hi, I read the code. You use the 'rng_same' mode to sample novel examples. So, it brings up the problem you mentioned above. To keep K examples of novel classes, you drop the novel object annotations when it beyonds the K-shot number in the query-support finetuning dataset. I think it harms the performance because the ratio of unannotated novel objects( sample noise to some extent) and K-shot annotated novel objects can be very large.
I totally agree, I actually conducted some experiments about it and it does hurt the performance. I think it is a better choice to keep all annotations for the K support images. Unfortunately, the existing literature is not clear about this design choice, some works keep only 1 annotation per image, others keep all and some do not even mention it.
If you want to use this repo, you should probably comment out the lines that filter the annotations in the dataset class.
I totally agree, I actually conducted some experiments about it and it does hurt the performance. I think it is a better choice to keep all annotations for the K support images. Unfortunately, the existing literature is not clear about this design choice, some works keep only 1 annotation per image, others keep all and some do not even mention it.
If you want to use this repo, you should probably comment out the lines that filter the annotations in the dataset class.
The problem bothering me for a while. In a natural image dataset, it is easy to find an image only contains one class object, so the exact k-shot dataset can be constructed based on these images. However, the aerial image always contains many objects in one image, so it is difficult to build a such k-shot dataset in a heuristic method.