The last clue that exists of Medicine, Jurisprudence and Philosophy, the so-called Paintings of the Faculty, by Gustav Klimt, is from 1945. The three great works were in a castle in Austria that, the day before the end of World War II, the Nazis had burned before the Red Army confiscated an artistic heritage that they had themselves plundered all over Europe. In that fire, more works by the Austrian artist disappeared, it is not known how many, what seems more than proven is that these three pieces from the end of the 19th century, measuring more than four meters, succumbed to the flames. The only trace that remained were black and white photographs until recently Google Arts & Culture and the Belvedere Museum in Austria have resurrected the pieces and their brilliant colors thanks to artificial intelligence.

The work has been done by an algorithm that has been trained by specialists from both institutions after exhaustive chromatic research. The starting point, since only black and white images remained of the three pieces, was a fragment of color that is still preserved from the Medicine. From there, 80 images of Klimt paintings were collected from different institutions. With this first information, the robot would learn through trial and error a bias towards the colors of its work. The experts resorted to works such as Beethoven Frieze, where golden snakes appear similar to those surrounding the three women at the forefront in Jurisprudence, they explain in the Google Institute.

But an algorithm of this type needs more food to continue learning, about 5,000 images on average to assimilate an object, so they also showed him a million photos of things in the real world, including people, animals and buildings; and 91,749 works of art already stored by Google Arts & Culture (the platform houses pieces and documentation from more than 2,500 cultural institutions from 80 countries). “This allows the machine learning model to assimilate the limits of objects, textures and compositions common in works of art,” say Google experts.

Not only did they use images, Klimt experts such as Franz Smola, from the Belvedere Museum, did a research work gathering academic and journalistic documents of the time. Smola used, for example, excerpts from the journalist Ludwig Hevesi’s chronicles on the Jurisprudence, which was exhibited in 1903: “Three terrifyingly beautiful avenging goddesses, with golden snakes in their hair.” Six years later, in another chronicle, Hevesi returns to treat the painting: “A hell of luxury, where the golden instruments of torture are encrusted with diamonds and the martyrs bleed rubies.” The precision in the details of these texts and, above all, in the description of the colors, has been as decisive for learning the robot as the images. “To make the paintings historically accurate, we guided the algorithm with Dr. Smola’s research. If we know that a certain object has a specific color, we add that color directly to black and white photos ”, they explain at the Google Institute.

With all this information, Emil Wallner, an engineer at Google, spent almost six months programming the artificial intelligence code with which the algorithm works so that it could generate color predictions according to Klimt’s work. In the next step in the process, Smola and the lab team reentered. The robot does not manually color the paintings, but instead does a statistical analysis of Klimt’s existing artwork and learns to mimic the coloring style. Therefore, with the chromatic references that the algorithm had generated, the specialists carefully inserted the colors in the three paintings of the Austrian master.

“The result was surprising because we were able to color even the parts about which we had no information,” explains Smola. “When I first saw the green sky of PhilosophyI exclaimed, ‘What is this?’ I was amazed because I assumed it would be blue. It was a special emotion, something I will never forget, ”Wallner says. The sky was emerald green, as some newspaper documents that described greenish tones in that part of the painting had already advanced.

In this way they revived the works with which the painter caused a scandal – another throughout his career – in the academic institution. In 1894, the University of Vienna commissioned Klimt and artist Franz Matsch for allegorical paintings depicting these three disciplines. An opportunity that the former took advantage of to criticize “the narrow-mindedness of the Austrian state and society.” To achieve this, he deployed his imagination through mythological concepts such as naked women trapped in snakes or pregnant women, children, skeletons, spectral beings with long hair, all brilliantly finished with the characteristic gold of Klimt. The result was the offense and anger of the experts, who described the pictures as pornographic and offensive.

Researchers at the Gustav Klimt Foundation recall that after being required on several occasions to review his creations, the painter finally removed the pieces, returned the money they had given him and sold them to Serena Lederer, a wealthy Jew and collector of the works by Klimt, a Vienna resident who was a victim of Nazi looting. His collection ended up in the castle that the SS officers burned down. Now revive on the internet. The result has become a 360º augmented reality experience that is part of Klimt vs. Klimt – The Man of Contradictions.