Portmanteau Vocabularies for Multi-Cue Image Representations

The success of the bag-of-words framework is highly dependent on the quality of the visual vocabulary. In this work we investigate visual vocabularies which are used to represent images whose local features are described by both shape and color. To extend BOW to multiple cues, two properties are especially important: cue binding and cue weighting. A visual vocabulary is said to have the binding property when two independent cues appearing at the same location in an image remain coupled in the final image representation. The property of cue weighting implies that it is possible to adapt the relevance of each cue depending on the dataset. The importance of cue weighting can be seen from the success of Multiple Kernel Learning (MKL) techniques where weights for each cue are automatically learned.

We describe a novel technique for feature combination in the bag-of-words model of image classification. Our approach builds discriminative compound words from primitive cues learned independently from training images. Our main observation is that modeling joint-cue distributions independently is more statistically robust for typical classification problems than attempting to empirically estimate the dependent, joint-cue distribution directly. We use Information theoretic vocabulary compression to find discriminative combinations of cues and the resulting vocabulary of portmanteau1 words is compact, has the cue binding property, and supports individual weighting of cues in the final image representation.

The effect of alpha (cue-weighting) on Portmanteau clusters. Each of the large boxes contains 100 image patches sampled from one Portmanteau word on the Oxford Flower-102 dataset. Top row: five clusters for alpha = 0.1. Note how these clusters are relatively homogeneous in color, while shape varies considerably within each. Middle row: five clusters sampled for alpha = 0.5. The clusters show consistency over both color and shape. Bottom row: five clusters sampled for alpha = 0.9. Notice how in this case shape is instead homogeneous within each cluster.

Code Available

Example code to construct portmanteau vocabularies method is available here: (Portmanteau Vocabularies code )

Data Available

Data used for the Demo code: (Data Available )

The final histograms of Bird-200 and Flower-102 data sets used in the paper: (Final Histograms )

Literature

Fahad Shahbaz Khan, Joost van de Weijer, Andrew D. Bagdanov and Maria Vanrell Portmanteau Vocabularies for Multi-Cue Image Representations , Proc. NIPS 2011 , Granada, Spain, 2011. (Poster)