“Our major challenge is to equip artificial intelligence (AI) with common sense,” explains the researcher Barcelona Supercomputing Center (BSC), Joaquim More. The computer linguistics expert demonstrated just how incapable machines are of contextualisation in an image recognition test with Raphael’s work Saint George kills the dragon. “The helmet identified him as a motorcyclist’s helmet and the dragon at the horse’s foot as a puppy,” he recalls.
The reason for the interpretation errors made by the AI lies in the lack of data, as Joaquim Moré laments: “It recognizes people driving vehicles because it has more data from today’s everyday life, but no kite or horse to wear. ” in the context of the moment of work. “That’s what we’re trying to solve with the project. Saint George on a bicycle. We will add a cultural layer to the AI so that it makes the appropriate identifications,” he reveals.
Thanks to this initiative, which aims to make machines interpret the images, artificial intelligence provides for the first time descriptions of images of cultural heritage, taking into account the temporal context of the creation of works of art.
The project Saint George on a bicyclewhich is coordinated by the BSC in cooperation withEuropeana Foundationtrains these artificial intelligence models using natural language processing algorithms and deep learning. “The key is to collect very powerful databases. However, we entered the project not knowing that there is hardly any art as there are some very limited datasets, it is not extensive or of good quality,” warns the BSC researcher and coordinator of Saint George on a bicycleMaria Cristina Marinescu.
In order for the machines to have the amount of data necessary to train the AI model, they turned to citizen science, which is common for projects in other areas such as the environment. They went to the portalcrowd sourced Zooniverse, which is still active the project campaign Saint George on a Bike, where more than 700 volunteers have already registered. One of these people who collects descriptive notes is the researcher of the NOVA University of Lisbon Giulia Sonetti, who, in addition to citizen science, is also a social volunteer of the association By the age of cycling sense, with which he picks up elderly people from dormitories in order to transport them by bicycle. “I like working together and even more so when it comes to collecting scientific data,” he says happily.
Another Saint George on a Bike volunteer is the artist Júlia Pérez, who has previously worked on other projects related to the technological world such as: B. the creation of virtual three-dimensional models of sculptures. “In the visual arts, they teach us the requirements of a visual construction, with elements like light structuring a painting. In the beginning it was difficult for me to describe the picture without considering these elements. All this information that humans involuntarily perceive and that evoke an emotion in us are layers that can be integrated into AI in the future, although for now we will limit ourselves to describing the images,” admits Pérez. BSC researcher Maria-Cristina Marinescu clarifies that for now their focus is on “training a model so that if you show them a photo other than an artwork, they can accurately describe it based on the data we see What is seen on it may have been introduced by other descriptions, but not its symbolism
The system manages to draw inferences using a language model. Moré gives an example: “We use ingenuity to enable him to infer that a person sitting on a horse in a painting is a rider. If he also wields a sword, he is a knight.”
To collect these descriptions, BSC researchers focused on illustrations of 16,000 works from the 12th to 18th centuries obtained from museums and supplements. You focused on this period, “mainly because there is no such thing as abstract painting,” says Marinescu. “We already have more than 11,000 descriptions of 3,000 artworks. Volunteers are leaving at a rate of 150 entries per day. It is important to have several different descriptions for each image in order to have higher quality data,” the project coordinator points out.
The project will have multiple applications. “Besides generating rich, high-quality cultural heritage data for people to access openly, it will help museums to better organize and analyze their heritage and launch creative initiatives,” says its coordinator Maria-Cristina Marinesco. “Also so that artists can search for iconography to make groundbreaking works iconographic. For example, Jesus Christ never seems to eat,” adds BSC researcher Joaquim Moré.
So that citizens can understand the possibilities offered by the use of artificial intelligence for image recognition, the extraction of relationships between thousands of images and the possibility of organizing virtual exhibitions based on the relationship between works of art that Saint George on a bike project has published an informative video.
Beyond art, “the same system can be explored for other applications, as it represents a first step towards contextual image recognition, from which artificial intelligence can go beyond what it is currently doing with capchas full of traffic lights and cars and little else ‘ concludes More.
reduced by 50 percent
Exclusive content for subscribers
read without limits