Questions to ask

7.18.11

One method to dealing with multi-class classification/image retrieval is incorporating the user into the task. Particularly interesting is with the approach of asking the user questions which he/she can readily answer but the computer cannot.

Animals

A simple example would be using the “Animals with attributes” dataset where every picture has a set of attributes which are all equally challenging. These include things such as: Is it X (where X is the color of the animal), does it have stripes, does it like water, does it eat X, and etc&… These are complex which require prior knowledge but that can also quickly improve classification/retrieval accuracy. [1]

Coins

Coins are very similar to the “Animal with attributes” dataset. Some basic questions include: Does the obverse contain an animal, what is the pose of the obverse figure, how many beings are on the obverse side, and &etc… As it can be see, while questions can be made, they are no longer boolean. While they could be reduced to boolean questions it would also create dependencies between the questions as well as drastically increasing the total number of attributes. For this reason a good system would have to be able to incoreporate non-boolean questions for it to be practical. Also many of the answers to these questions are similar for instance some figures are said to be “charging” while others are “running”. These similarities if not taken account of would make any system impractical. Some understanding of attribute similarity is important to allow for both user and ground-truth errors that may exist.

[1] Learning To Detect Unseen Object Classes by Between-Class Attribute Transfer. Lampert, Nickisch, Harmeling. 2009