Humans intuitively know what a chair is. You have seen many examples from an early age and learned what makes up a chair from these examples: legs, seat, backrest. You have internalised these characteristics in such a way that you no longer have to parse the individual components of a chair to know that 'this is a chair'. People have the ability to deep learn, a term often associated with artificial intelligence. But more about that later.
Film about the project (German)
Prof. Dr. Siegfried Handschuh has been Associate Professor at the University of Passau and Professor of Data Science at the University of St. Gallen Bernhard Bermeitinger (computer scientist), Maria Christoforaki (computer scientist) and Simon Donig (historian) describe how they taught the formal language of classicism in the Neoclassica Maschinen project.
Playing the video will send your IP address to an external server.
First to Simon Donig. He is a historian interested in neoclassical furniture. Donig has acquired expert knowledge over several years. He knows more than a little about certain chairs and can assign their characteristics and peculiarities to certain epochs. For example, he knows that the formal language of a neoclassical chair refers back to the ancient world. It is simpler, straighter and more geometric than that of a baroque chair and has fewer floral decorations. He is now passing this knowledge on to a machine:
The historian Simon Donig passes on his knowledge to the machine by annotating this image.
The historian Donig annotates a photograph of a period room in the Metropolitan Museum of Art. The museum uses this form of installation to make contemporary spaces directly tangible. The picture is a replica of a room in the "Hôtel de Tessé", which stood in Paris in 1628. The wall panelling actually comes from the Hôtel, the exhibited pieces of furniture are also originals from that time, but not directly from the Hôtel. However, the furniture in the Hôtel could have looked the same or similar, perhaps including two armchairs or, in technical terminology: fauteuils.
For the machine to be able to understand the historian, it needs mediation from computer science. Under the direction of Siegfried Handschuh (*), the team at the Passau Chair of Digital Libraries and Web Information Systems has developed the Neoclassica project editor, which the historian Donig can use to teach the machine the neoclassical language of form. The computer scientist Maria Christoforaki has modelled the historian's knowledge and turned it into a form that machines can understand. Together they built an ontology, a kind of dictionary for a computer.
Playing the video will send your IP address to an external server.
Historical knowledge, translated into computer language
In the video, computer scientist Maria Christoforaki explains how she translated the work of historian Simon Donig into a form that can be processed by a machine.
The Passau ontology is based on the CIDOC Conceptual Reference Model, which many museums worldwide use to document their cultural heritage. It is a standard (ISO 21127:2014) for the controlled exchange of information in the field of cultural heritage. This should ensure that other researchers can also use the data annotated with the Neoclassica ontology.
Historical definitions feed artificial intelligence
"This ontology has the mark of Simon Donig," says Prof. Dr. Handschuh. It also has the mark of his research: the historian researched primary sources for this, including the "Cabinet Dictionary" from 1803, in which the English furniture manufacturer Thomas Sheraton compiled inspiring illustrations of chairs. The term fauteuil (armchair) appears again in this and is defined as follows: "Fauteuil, from the French, signifies a large chair." This definition is now also found in the Passau ontology and consequently in CIDOC.
The ontology thus models this expert knowledge and makes it accessible to the computer. However, it is only one component of the artificial intelligence used by the Passau team. The second component is an intelligent, adaptive system: an artificial neural network. This identifies and classifies images fed into the system.
According to Prof. Dr. Siegfried Handschuh, neural networks are essentially based on statistical methods: they try to estimate a mathematical function with many millions of parameters.
VGG19-Layout, Convolutional Neural Network
Schematic representation of the processing steps in a normal convolutional neural network:
An image is entered on the left. The matrix represents the colour values of the respective pixels. This neural network consists of 19 convolutional layers, which independently find and compress the information in the image. The output on the right is a probability distribution across the given classes. The example image shows the neural network is 74% sure that the entered image is a chair.
Source: Bernhard Bermeitinger
Broadly speaking, a neural network consists of several layers. On the left, the training example is entered into the input layer. The above image depicts a neoclassical chair. On the right, the best result from the output layer is that the image is recognised to contain a neoclassical chair with a high probability.
In between there are hundreds of hidden layers whose purpose is to recognise and extract different characteristics of an image. For example, the first level could learn to recognise the rough contours of a chair. The levels in between extract further features and the last level captures the smallest details and shapes that can be characteristic of the neoclassical language of form.
A neural network learns with the help of any number of examples; the more the better. Deep learning in a machine work in a similar way to the human brain: the artificial neural network identifies certain characteristics in the different layers and passes them on to the deeper layers. It can reduce characteristics to the essentials and generalise them. On the basis of this knowledge, it can calculate the probability of whether a previously unknown image also depicts a chair. An example of such an artificial neural network would be the search functions in Google Photos or Google Lense.
Not quite a match: Pre-trained by Google, the network recognises objects from the everyday life of today in images, such as a laptop. The supposed laptop is in fact a mirror from the dressing room of the French queen Marie Antoinette.
High match rate: This is the result after the already pre-trained network has been trained with Passau ontology.
Now the Neoclassica team has collected many images of neoclassical furniture – a total of 1246 files from renowned museums such as the New York Museum of Modern Art, the Amsterdam Rijksmuseum or the Hermitage in St. Petersburg. However, this is too little to train the network successfully. They sometimes have only five to 20 images available to make up a training example; usually it's 100 to 1000.
Network pre-trained by Google
In order to improve accuracy, the research team in Neoclassica uses a trick: it relies on the preliminary work carried out by Google, which has already trained the neural network regarding everyday knowledge. The artificial intelligence already knows the concept of a chair and recognises chairs in images. The pre-trained Google network, on the other hand, does not yet know neoclassical objects; instead, it assigns the forms to objects from our everyday lives. A laptop, for example. Or a microwave. However, trained with the historian Donig's knowledge, the network can even recognise neoclassical furniture in previously unknown images with a with high degree of reliability, as the computer scientist Bernhard Bermeitinger shows here:
The computer scientist Bernhard Bermeitinger demonstrates how the neural network can locate items of neoclassical furniture in photographs.
Such knowledge is of no interest to the internet giant Google, which specialises in digitising our everyday lives. The historian, on the other hand, is enthusiastic about the new tools: they could open up completely new opportunities for experts in the field of art history.
This collaboration with the humanities is also attractive for project manager Prof. Dr. Siegfried Handschuh, himself a computer scientist. He goes one step further: He wants to take the machine so far that it can not only recognise items of furniture, it can also describe them – and consequently check existing theories.
Playing the video will send your IP address to an external server.
* Prof. Dr. Siegfried Handschuh has been Associate Professor at the University of Passau and Professor of Data Science at the University of St. Gallen since the winter semester 2018/2019.