Background:
Supported by the ARC-iJAC, the research led by Dr. Ewa Machotka and Dr. John Pavlopoulos (Stockholm University) has pursued the large-scale digital geospatial exploration of places depicted in Japanese early modern ukiyo-e landscape prints through Natural Language Processing (FY 2021). Their follow-up project aims to apply NLP technology to inscriptions on ukiyo-e landscape prints to facilitate a large-scale exploration of textual information featured in those prints (FY 2023).

Project leader: Dr. Ewa Machotka (Dept. of Asian and Middle Eastern Studies, Stockholm University)
Project manager: Dr. John Pavlopoulos (Dept. of Computer and Systems Sciences, Stockholm University)
Project members: Konstantina Liagkou, Panagiotis Papapetrou, Marita Chatzipanagiotou

Thank you very much for your time today. Could you please tell us the motivation for your ARC-iJAC research project Natural Language Processing for a Geospatial Exploration of Japanese Ukiyo-e Prints?

Machotka: The last several decades saw the rise of interest in the concept of Global Art History, understood as a heterogenous transnational and critical study of the world's cultural production. One of the challenges of this new research direction is the question of how to acknowledge the conceptual and material heterogeneity of artistic production across the world in a way that does not support a universalist understanding of cultures. This concern prompted our research. We saw that Japanese early modern landscape prints, as globally recognizable non-Western pre-modern artifacts, offer a critical testbed for considering these issues.

We know that these prints are often defined today as fūkei-ga, or landscapes. However, we should not forget that the notion of fūkei is a modern cultural translation entangled with the ideology of modernization and colonial power. Originally these images were largely defined as meisho-e or 'images of famous places', and they are rooted in poetic rhetorical figures that tie seasonal images with either actual or imagined places. So, to understand meisho-e prints and their social function at the time of their production, we have to understand what places were depicted (i.e. considered culturally significant) and how these geographical locations were represented and mediated by the prints. We wanted to identify a general pattern in this mediation, which can be done easier at a large scale instead of at the level of individual prints.

Considering the richness of the corpus that includes thousands of objects, we thought that recent advances in Natural Language Processing (NLP) could effectively help us to take the first step in this study, namely the geolocation of places depicted in prints, and identification of their distribution across time and space. Our exploratory mixed-method analysis has so far delivered promising results. We developed a novel application of NLP for the Digital Humanities that demonstrates the transformative potential of AI for the study of Japanese early modern prints and Art History at large.

How have the ARC-iJAC resources supported you in realizing this project?

Liagkou: To put it plainly, our research would not be possible without the Ukiyo-e Portal Database being developed by and hosted at the Art Research Center at Ritsumeikan University. First, the Portal Database offered us access to print collections kept in different museums around the world. If this feature sounds trivial, please note that not all museums freely share their collections online with the public.

Second, the ARC Ukiyo-e Portal Database offers access to an extremely rich corpus. When we started our research, the Portal Database hosted 678,429 prints kept at 28 institutions in Japan and abroad, and it is continuously growing. Hence, it offers access to a very large corpus of ukiyo-e prints facilitating 'distant viewing' or a macro analysis of the prints, thus also enabling a diversity of analytic tasks.

Third, the Portal Database features not only high-quality visual data itself (delivered in a standardized protocol) but also rich and high-quality metadata facilitating different kinds of explorations and analyses. In the context of our project, we especially appreciated image-content-related transcription of inscriptions on prints which often mention names of the places depicted in the prints. We identified these mentions with NLP, geotagged them, and then visualized them on a map.

As part of your research project, you developed an online application called Ukiyo-e Distant Viewer. Could you briefly explain its merits/ purpose?

Pavlopoulos: The Ukiyo-e Distant Viewer aims to facilitate the geolocation and visualization of recognized place-name entities found in ukiyo-e prints, enabling users to identify culturally significant places and explore their spatial distribution. This analysis covers thousands of images across Japan and provides a large-scale perspective on early modern landscape imagination.

It is important to note that our focus is not on individual prints or print series, but rather on identifying trends and changes across time.Ultimately, this tool will enable us to trace the chronological development of this imagination and gain insights into its cultural and historical significance.

How do you feel about the execution of the project? Have you come across any particular challenges?

Liagkou: Our exploratory mixed-method analysis has so far been successful. First, by employing the NLP approaches such as transfer-learning and Named Entity Recognition (NER) and applying our fine-tuned recognition model on a large dataset of prints, we provided a use-case of how a macroanalysis of a visual dataset can be undertaken in art historical research of Japanese visual culture.

Machotka: We also identified a number of methodological challenges. As we know, the field of Spatial Art History--combining Geographical Information Systems (GIS), NLP, and Corpus Linguistics--has advanced in the past few years. However, although these tools perform well on modern datasets, it is not the same for historical materials.

We can encounter several problems, such as OCR errors, difficulties related to place reference identification, and place reference disambiguation (related to language changes over time), among others. The situation is even more complicated in the case of mapping meisho-e prints due to the ambiguity of the depicted visual motifs. We need to note that place identification in prints is not always facilitated by iconography or visual motives but by the image-content-related inscriptions printed in the images that often feature place names.

So, the geolocation of these sites requires the reading of inscriptions. And transcription of inscriptions is one of the main obstacles for art historians interested in a large-scale analysis of the prints. This is due to the complexity of the Japanese early-modern writing system, problems with adequate identification of place names, and material aspects of a print (e.g. color scheme and preservation state). We plan to address some of these issues in our next project, which focuses on exploring possibilities of using computational tools for the automated transcription of inscriptions on prints.

Your next project, AI-powered Text Recognition of Inscriptions in Japanese Ukiyo-e Prints, will continue to utilize Natural Language Processing (NLP) technology to study the landscape prints (meisho-e) in the ARC Ukiyo-e Portal Database. What is the significance of this project?

Pavlopoulos: In this project, we will focus on resolving the linguistic problems related to transcribing the inscriptions on prints enabling geolocating of places mentioned in these inscriptions. We will investigate the development of computational tools for the automated recognition of the text of inscriptions on prints rather than using already-transcribed inscriptions provided in the database.

As we established previously, NER can be used to successfully extract the place names from inscriptions on ukiyo-e prints. However, the tool requires transcribed digital metadata to generate information, while many museum collections lack reliable transcriptions of inscriptions on prints.

Optical character recognition and handwritten text recognition (HTR) can be used to recognize the text from an image. Due to the technological, formal, and linguistic characteristics of ukiyo-e print inscriptions, which do not use a standardized writing system or movable type, we hypothesize that handwritten text recognition could be effectively applied to inscriptions on Japanese prints.

We expect the recognized text to contain errors, and we will investigate the accuracy of extracting place-named-entities from the recognized (not transcribed) text. This multimodal methodological challenge requires testing on non-transcribed inscriptions on prints, and our study will facilitate this.

How did you first connect with the Art Research Center (ARC)? / How did you hear about the International Joint Digital Archiving Center for Japanese Art and Culture (ARC-iJAC)?

Machotka: As a scholar of Japanese art history educated in Japan, at Gakushūin University in Tokyo (thanks to the doctoral fellowship issued by MEXT), I have been well aware of the pioneering contribution of the Art Research Center (ARC) and the International Joint Digital Archiving Center for Japanese Art and Culture (ARC-iJAC) to the digitization of Japanese cultural artifacts, and computational analysis of Japanese art.

I also had the honor and pleasure to meet the leading ARC researchers, Prof. Akama Ryo, Prof. Suzuki Keiko, Prof. Yano Keiji, Dr. Matsuba Ryoko, and many other colleagues at different academic events in Europe and Japan. So, I have been aware of the important work done by the ARC, its faculty, and the research value of the ARC databases for a long time. And I have to admit that I probably would have never started my own research adventure with Digital Art History if not for the ARC and its ground-breaking work.

Is there anything else you would like to comment on or highlight?

Machotka: We would like to stress one important issue, which we think is the key to the success of Digital Humanities, and can push the frontiers of research in (Global) Art History. It is the need and value of collaboration across various disciplines, institutions, and national borders.

Communication between researchers, exchange of experiences, sharing knowledge and good practices is the key to knowledge production. Digitization processes are going fast, and many museums invest in building their digital databases and sharing their collections with the general public. But to benefit from this incredible work, we would also like to see a strengthening of analytical aspects of Digital Art History, using computational tools not only to offer wider accessibility to museum collections but also to facilitate analysis and a better understanding of art objects.

So, we would like to encourage other art historians and computer scientists to explore possibilities for collaboration. Interdisciplinary work across distant disciplines like ours is not an easy task, as we need to learn to understand and respect our divergent research approaches. But it has been a rewarding experience for our team and brought out new findings that can move our disciplines forward.

(This interview was conducted by Yinzi Emily Li.)