Matching Feature Sets for Few-Shot Image Classification
Arman Afrasiyabi, Hugo Larochelle, Jean-François Lalonde, Christian Gagné


Published in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2022


In image classification, it is common practice to train deep networks to extract a single feature vector per input image. Few-shot classification methods also mostly follow this trend. In this work, we depart from this established direction and instead propose to extract sets of feature vectors for each image. We argue that a set-based representation intrinsically builds a richer representation of images from the base classes, which can subsequently better transfer to the few-shot classes. To do so, we propose to adapt existing feature extractors to instead produce \emph{sets} of feature vectors from images. Our approach, dubbed SetFeat, embeds shallow self-attention mechanisms inside existing encoder architectures. The attention modules are lightweight, and as such our method results in encoders that have approximately the same number of parameters as their original versions. During training and inference, a set-to-set matching metric is used to perform image classification. The effectiveness of our proposed architecture and metrics is demonstrated via thorough experiments on standard few-shot datasets---namely miniImageNet, tieredImageNet, and CUB---in both the 1- and 5-shot scenarios. In all cases but one, our method outperforms the state-of-the-art.

Paper and Supplementary Material

Arman Afrasiyabi, Jean-François Lalonde, Christian Gagné
Matching Feature Sets for Few-shot Image Classification.
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022
(paper hosted on ArXiv)


Video [Slides]




click on the figure to see .pdf version.


This project was supported by funding from NSERC-Canada.