Michela Vignoli, Doris Gruber, Rainer Simon, Axel Weißenfeld: Impact of AI: Game-changer for Image Classification in Historical Research? (original) (raw)
Related papers
Deep learning for historical books: classification of printing technology for digitized images
Multimedia Tools and Applications, 2021
Printing technology has evolved through the past centuries due to technological progress. Within Digital Humanities, images are playing a more prominent role in research. For mass analysis of digitized historical images, bias can be introduced in various ways. One of them is the printing technology originally used. The classification of images to their printing technology e.g. woodcut, copper engraving, or lithography requires highly skilled experts. We have developed a deep learning classification system that achieves very good results. This paper explains the challenges of digitized collections for this task. To overcome them and to achieve good performance, shallow networks and appropriate sampling strategies needed to be combined. We also show how class activation maps (CAM) can be used to analyze the results.
Deep Learning Approaches to Classification of Production Technology for 19th Century Books
2018
Cultural research is dedicated to understanding the processes of knowledge dissemination and the social and technological practices in the book industry. Research on children books in the 19th century can be supported by computer systems. Specifically, the advances in digital image processing seem to offer great opportunities for analyzing and quantifying the visual components in the books. The production technology for illustrations in books in the 19th century was characterized by a shift from wood or copper engraving to lithography. We report classification experiments which intend to classify images based on the production technology. For a classification task that is also difficult for humans, the classification quality reaches only around 70%. We analyze some further error sources and identify reasons for the low performance.
Applying Computer Vision Systems to Historical Book Illustrations: Challenges and First Results
2020
Digital humanities still need to unlock the potential of images anlysis algorithms to a large extent. Modern deep learning images processing can contribute much to quantify knowledge about visual components in books. In this study, we report on experiments carried out for historical print. The illustrations in books offer much for humanities research. Object recognition systems can identify the portfolio of objects in book illustrations. In a study with several hundreds of books, we applied systems to find illustrations and classify them. Results show that persons are shown in illustrations within fiction books with a higher frequency than in non-fiction books. We also show the classification results for an analysis of the printing technology. This expert task can still not be perfectly modeled by a CNN. A class activation map analysis can be used to analyze the performance qualitatively.
PEARC '20: Practice and Experience in Advanced Research Computing, 2020
Art historians are interested in possible methods and visual criteria for determining the style and authorship of artworks. One approach, developed by Giovanni Morelli in the late nineteenth century, focused on abstracting, extracting and comparing details of recognizable human forms, although he never prescribed what exactly to look for. In this work, we asked what could a contemporary method like convolution networks contribute or reveal about such a humanistic method that is not fully determined, but that is also so clearly aligned with computation? Convolution networks have become very successful in object recognition because they learn general features to distinguish and classify large sets of objects. Thus, we wanted to explore what features are present in these networks that have some discriminatory power for distinguishing paintings. We input the digitized art into a large-scale convolutional network that was pre-trained for object recognition from naturalistic images. Because we do not have labels, we extracted activations from the network and ran K-means clustering. We contrasted and evaluated discriminatory power between shallow and deeper layers. We also compared predetermined features from standard computer vision techniques of edge detection. It turns out that the deep network individual feature maps are highly generic and do not easily map onto obvious authorship interpretations, but in the aggregate can have strong discriminating power that are intuitive. Although this does not directly test issues of attribution, the application can inform humanistic perspectives regarding what counts as features that make up visual elements of paintings.
Extracting and Analyzing Deep Learning Features for Discriminating Historical Art
Practice and Experience in Advanced Research Computing, 2020
Art historians are interested in possible methods and visual criteria for determining the style and authorship of artworks. One approach, developed by Giovanni Morelli in the late nineteenth century, focused on abstracting, extracting and comparing details of recognizable human forms, although he never prescribed what exactly to look for. In this work, we asked what could a contemporary method like convolution networks contribute or reveal about such a humanistic method that is not fully determined, but that is also so clearly aligned with computation? Convolution networks have become very successful in object recognition because they learn general features to distinguish and classify large sets of objects. Thus, we wanted to explore what features are present in these networks that have some discriminatory power for distinguishing paintings. We input the digitized art into a large-scale convolutional network that was pre-trained for object recognition from naturalistic images. Because we do not have labels, we extracted activations from the network and ran K-means clustering. We contrasted and evaluated discriminatory power between shallow and deeper layers. We also compared predetermined features from standard computer vision techniques of edge detection. It turns out that the deep network individual feature maps are highly generic and do not easily map onto obvious authorship interpretations, but in the aggregate can have strong discriminating power that are intuitive. Although this does not directly test issues of attribution, the application can inform humanistic perspectives regarding what counts as features that make up visual elements of paintings.
AI and Cultural Heritage Image Collections: Opportunities and challenges
2021
This paper contributes to the discussion about the opportunities and challenges of applying computer vision and machine learning to archival image collections of significant cultural heritage value. We explore these questions from an institutional perspective. Our case study is a pilot project developed at Dumbarton Oaks, a research institute and library, museum, and historic garden affiliated with Harvard University and located in Washington, DC. The project focused on a collection of 10,000 images of Syrian monuments in the institution's Image Collections and Fieldwork Archives (ICFA). Drawing on that project, as well as the broader landscape of AI-based categorisation efforts in the fields of art and architecture, we will share our insights on the potential of AI to facilitate and enhance archival image access and recording. Many of the Syrian sites in the Dumbarton Oaks collection have been inaccessible to researchers and the public for over a decade and/or have been damaged or destroyed. The pilot project, undertaken in 2019-2020, was a collaboration between Dumbarton Oaks; a commercial tech partner, ArthurAI Inc.; and a computer science research team from the University of Maryland. For Dumbarton Oaks, the primary goal was to explore whether AI can improve the speed and efficiency of sharing collections and allow for more sophisticated curation by subject experts who, thanks to automation, would be relieved of the burden of rote processing. For the technology partners, the experimental value of the project lay in the availability of a collection that could be shared open access (no privacy or copyright issues) and was focused enough to yield a domain-specific training set. The methods and techniques explored included multi-label classification, multi-task classification, unsupervised image clustering, and explainability. Image classification. Deep learning. Convolutional neural networks. Cultural heritage. Digital humanities.
Deep Learning for Classification and as Tapped-Feature Generator in Medieval Word-Image Recognition
2018 13th IAPR International Workshop on Document Analysis Systems (DAS), 2018
Historical manuscripts are the main source of information about past. In recent years, digitization of large quantities of historical handwritten documents is in vogue. This trend gives access to a plethora of information about our medieval past. Such digital archives can be more useful if automatic indexing and retrieval of document images can be provided to the end users of a digital library. An automatic transcription of the full digital archive using traditional Optical Character Recognition (OCR) is still not possible with sufficient accuracy. If full transcription is not available, the end users are interested in indexing and retrieving of particular document pages of their interest. Hence recognition of certain keywords from within the corpus will be sufficient to meet the end users needs. Recently, deep-learning based methods have shown competence in image classification problems. However, one bottleneck with deep-learning based techniques is that it requires a huge amount o...
2022
Jointly funded by the British Arts and Humanities Research Council (AHRC) and the Laboratoire d’excellence « Les passés dans le présent », The Early Conflict Photography 1890—1918 and Visual AI (EyCon) project aims to harness AI-reliant computer vision tools to analyze a large corpus of conflict photography, particularly concerning colonial warfare. The research group is partnered with several UK and French archives and museums, including La Contemporaine, which notably holds the fonds Valois, containing the photographs produced by the Section photographique de l’armée (SPA) during World War One. Using both Application Programming Interfaces (APIs) for already-digitized collections, as well as scanning of collections that have not already been digitized, Eycon has assembled a very large corpus of images, and is now advancing to the computing stage. Computation involves training a neural network to recognize various features of the images in the corpus, with the aim of enriching their associated metadata, increasing their navigability, and making them more useful for researchers and the public. This short report explains the project’s goals, with a focus on the fond Valois. In particular, it considers how machine learning holds the potential for new contributions to existing scholarship on photography, personal presentation, and emotion, as well how the SPA attempted to shape the image of the war for domestic and international audiences.
Recognizing Characters in Art History Using Deep Learning
Proceedings of the 1st Workshop on Structuring and Understanding of Multimedia heritAge Contents - SUMAC '19, 2019
Figure 1: Art historical scene depicting the iconography called Annunciation of the Lord (left [10], right [32]). Mary and Gabriel are the main protagonists. We can clearly see the differences in the background, in the artistic style, in the foreground, in the objects, their properties, and the use of color.