Back to the coins, there are a few intuitions which seem important to be noted.First, coins are ultimately annotated  by students who copy from primary sources written by experts.  Second, ideally the expert uses a more precise vocabulary than the student (where “precise” is a metric not yet defined). Third, ideally the entropy present in the answers given by experts is far higher than that of students. That is, students answer questions in more general terms than experts such that an expert’s answer is far more helpful in classifying a coin than a student’s answer. The problem arises due to the fact that coins are annotated by experts and no graceful method for generalizing the questions/answers exists implicitly in the dataset. All questions trivially present in the database such as “Name the figure on the reverse of the coin from this list” has no clear general form. That is, not without knowing about linguistics because the general form of the question could be “Is the figure on the reverse of the coin a male or female or other?”

One way of generalizing questions/answers is to use WordNet. This is not a trivial issue, but has been researched at length. Instead of asking the question “Figure on the obverse of coin” and offer over 100 answers to choose from, more abstract questions could potentially be used by abstracting the answers, and then further refining the answer as needed. Bellow is a tree of the abstraction of answers where the green circles represent actual answers encoded in the dataset and the black circles represent abstract answers.

Many terms are not included in this chart, and many of the generalized answers are not intuitive or what questions can be made such that the answers make sense. These are points of further research. What does become apparent is that several answers which are encoded in the dataset can be generalized into much broader terms. Example: instead of having answers “Ares” and “Luna” side by side its more intuitive (a metric that needs further defining) for a student to be first asked about “Greek deities” and “Roman deities”.

1. “Visual Recognition with Humans in the Loop”, Branson, Wah, Schroff, Babenko, Welinder, Perona, Belongie. 2010
2. “Exploiting ontologies for automatic image annotation”, Srikanth, Varner, Bowden, Moldovan. SIGIR 2005.

Let’s assume for any given question, there is a discreet set of possible answers, one of which is the “correct” one. In prior work if the wrong answer is given, it can drastically skew the accuracy of results. This is particularly the case in the Coin dataset. If a coin is annotated to have a “young male” on the obverse side but the user mistakes it for “Zeus” this will make identifying the coin almost impossible. Current systems recover from such situations by asking the user more questions as to dampen the incorrect answer. This is however clumsy as the more questions that are asked, more likely is there a chance for incorrect answers to be given (or rather, there is a fixed error rate for all questions).

• “young male” vs. “boy”. WordNet can be used to find similarity of single words, not phrases. How to address phrase similarity is a problem.
• “man” vs. “Zeus”. WordNet does not have proper nouns. This seems a problem suited for ontological studies.
• “Zeus” vs. “Jupiter”. This is the most curious issue because it crosses several problems. First it is similar to the ontological problem as both could be thought of as “man” and thus share some similarity. Or some equivalency could also exist, like a pseudonym. Second, and much more interesting, there exists no historical/mythical genealogical database similar to WordNet. It would be very helpful if similarity between “Henry II” and “Henry III” could be calculated because that is much more similar to “Buddha” who shares no historical or mythical connection.

Animals

A simple example would be using the “Animals with attributes” dataset where every picture has a set of attributes which are all equally challenging. These include things such as: Is it X (where X is the color of the animal), does it have stripes, does it like water, does it eat X, and etc&… These are complex which require prior knowledge but that can also quickly improve classification/retrieval accuracy. [1]

Coins

Coins are very similar to the “Animal with attributes” dataset. Some basic questions include: Does the obverse contain an animal, what is the pose of the obverse figure, how many beings are on the obverse side, and &etc… As it can be see, while questions can be made, they are no longer boolean. While they could be reduced to boolean questions it would also create dependencies between the questions as well as drastically increasing the total number of attributes. For this reason a good system would have to be able to incoreporate non-boolean questions for it to be practical. Also many of the answers to these questions are similar for instance some figures are said to be “charging” while others are “running”. These similarities if not taken account of would make any system impractical. Some understanding of attribute similarity is important to allow for both user and ground-truth errors that may exist.

[1] Learning To Detect Unseen Object Classes by Between-Class Attribute Transfer. Lampert, Nickisch, Harmeling. 2009

Sample images from the Caltech-101 dataset. Link

These datasets while difficult, are relatively easy when compared to the multi-class datasets. This is because images belonging to a particular category can be visually different from the other images. For instance a soccer ball is very different from an airplane when color histograms are compared.

Birds and flowers have recently become popular datasets because while the images can still be very different, they are more similar than prior datasets. For this reason they belong in the multi-class type of classification.

Flowers dataset. Link

Caltech-UCSD Birds 200. Link

Coins dataset. Link

As it can be seen, coins is perhaps an extreme example of the multi-class image classification problem. While a numismatic expert may argue that coins from different regions of the world could be split by category, and denominations within the region are actually classes, for the untrained eye (ie. computers) they are pretty much all the same. Unlike flowers where color and petal shape can aid in the classification process coins have neither. Also coins have no “poses” as is the case with birds which has been shown to be an important feature. These omissions have all been used in the recent past in creating state-of-the-art classification algorithms for the Bird and Flower datasets. While birds and flowers have a relatively small number of visual characteristics coins have complex scenes that have been painstakingly catalogued by humans.

• Fixing any disk errors
• Fixing any permission errors
• Reinstalling Snow Leopard
• Reinstalling Chrome

Parallel Python

Parallel Python (PP) is a great start in getting code to run in several computers. It works on a versatile client/server model, where generic servers must be launched to do the actual computations and the user must right a client which interacts with the several servers. The nice thing about this module is how fast you are up and running. Basically you pass arguments, and a function pointer to PP and it handles the distribution and gathering of results. Load balancing, passing extra data, and anything that can’t be done with a single function will be outside the realm of this module.

MPI4Py

MPI is something that I never used, the documentation is sparing, and overall very mysterious. That said, the potential is great. There are several MPI implementations for Python, and I picked this one because it seemed the easiest to install and start using. Essentially when using MPI4Py your script is called in parallel on multiple CPU cores and/or computers and the MPI element makes sure that all the instances talk to each other. What is cool is that variables in your program are both local to an instance, but at the same time can be shared seamlessly with your other instances. The documentation is very good, but the actual MPI standards are very odd and worth learning more about particularly because it can be used with computer clusters.

The setup

My main computer runs Snow Leopard and I have become a stickler to running all my applications in 64bit. Not for performance gains (although this does enable the use of OpenCL which  is 64bit only), but for consistency. Actual computations are left for my schools computer cluster which uses some linux distribution and are also 64bit machines.

• Python-2.6
• Matlab 2010a on the mac (64bit)
• Matlab 7.6 on linux (32+64bit)

Great, but what about my old code

The first hurdle to the transition is making all that old Matlab code work. Once in 1999 someone discovered this problem and wrote PyMat which interfaced Python and Matlab by using Matlab’s shared libraries. While perhaps it still works, the code felt very antiquated. It has been over 10 years since the last update! So I wrote a new version from scratch which was a delight for learning more of the internals of Python.

The nice thing about the module I wrote is that is allows for all matlab functions to be called directly so that:

X = matlab.rand(numpy.array([100 100]), 1)

The following snippet calls Matlab’s native rand function and the result is stored in a numpy array in X. The 1 at the end of the snippet tells the python interface to expect a single return value. In the future this may not be required.

A difficulty in doing this interface is that each Matlab version has slightly different set of function names. This makes the interface have to be slightly tweaked for each version.

Functions that don’t exist

It surprises to see how many helpful functions Matlab has. Particularly saving and opening images and other assorted data types. Numpy while very good doesn’t have a easy to use save routine nor can Numpy natively open arbitrary images. While naturally this is because these features are not part of the project’s mission statement, it does mean more work for the valiant researcher.

Quickly you will see that a handful of functions must be written to do these simple tasks, and while there are plenty of stable modules including PIL and Pickle for opening images and saving data (respectively), it does mean more steps between you and your work.

IDE

Finally by the end of the day you will begin to miss the ugly, Java-esc, Windows-esc and X11-est IDE that Matlab has featured for probably over a decade. Being able to inspect variables, even see the variable names being used, and the ugly but useful debug features is badly missed as well.

By the end

By the end you realize why Matlab exists and is widely used. However, unless someone ventures out and fixes the wholes that exist with Python as a Matlab alternative, people will never have an alternative. Unifying the tools that exist for Python and adding interoperability with Matlab are the first steps to making a really good alternative to the most widely used piece of software in the sciences.

A fork of OpenCV 2.0 can be found at https://code.google.com/p/opencv-cocoa/ which has both the Cocoa and Quicktime updates. This is completely tested so use it with caution. Any help and suggestions are greatly appreciated.

Because Cocoa functions strictly inside a MVC framework there is a small change that you must make at initializing HighGUI that is found here. It boils down to the fact that your code cannot run in the main thread, because Cocoa needs to run there. So to update any existing projects you must remove your code from the main function and place it in an auxiliary function which is called by cvInitSystem.

I have not yet made any benchmarks, so there is a chance that using Cocoa and QTKit at either 32 or 64 bit is not imperative for improving performance of your code. The main purpose is to allow the usage of other libraries in conjunction with OpenCV which already work at 64bit and may depend on Cocoa.

It is extremely easy to write OpenCL kernels (although there is room for improvement) and here is the warping kernel I have written.