ART RESEARCH CENTER

Professor Tanaka, thank you for your time today. Firstly, could you please tell us how you joined the Art Research Center at Ritsumeikan University?

Prof. Tanaka: In 2004, the College of Information Science and Engineering was founded at Ritsumeikan University. I was appointed to the Department of Information Science and Engineering in 2002 as a founding member.

Coincidently, the Art Research Center had been selected for the 21st Century Center of Excellence (COE) Program "Kyoto Art Entertainment Innovation Research" by the Ministry of Education, Culture, Sports, Science, and Technology (MEXT) in the same year and so, the joint research started.

Can you please tell us more about your area of research?

Prof. Tanaka: I conduct research on the ultra-high-quality visualization and visual analysis of cultural heritage to digitally preserve valuable cultural properties in Japan and abroad.

By utilizing the latest 3D scanning technologies such as laser scanning and photogrammetry techniques, we can obtain large-scale point-cloud data of cultural assets.

We have proposed a novel method of see-through visualization applicable to point cloud data. This method enables us to visualize the complex inner and outer structures of tangible cultural heritage.

Can you please elaborate on this a little further?

Prof. Tanaka: Conventional digital archiving methods, such as photography, visualize the appearance of three-dimensional cultural properties the way they are seen. However, while photographs and videos are certainly a realistic representation of cultural properties, they do not adequately reflect the entirety of them.

Therefore, capturing an image with a camera is insufficient for building digital archives of tangible assets, particularly large-scale cultural assets.

Temple compounds, for example, are vast and often made of complex internal structures. Hence, when you conduct laser scanning, the point clouds obtained the points ranging from hundreds of millions to billions of them. Therefore, capturing an image with a camera is insufficient for building digital archives of tangible assets, particularly large-scale cultural assets.

Our method of see-through visualization, which is applicable to massive point-cloud data, helps to visualize these complex shapes and internal structures, making it easier for us to understand the cultural properties as a whole.

Since we are talking about massive amounts of point-cloud data here, visualizing cultural heritage is basically about visualizing Big Data.

Can you please give us an example of your projects?

Prof. Tanaka: As part of the project to digitally archive the Gion Festival, one of the biggest annual festivals in Japan and a UNESCO Intangible Cultural Heritage, we created precise 3D see-through visualization models of two of the festival floats - the Hachiman-yama and the Fune-hoko.

You can observe the floats from different viewpoints and explore the complex inner structures from a semi-transparent exterior.

Furthermore, we have digitally recorded the assembly of festival floats - a process that usually takes three days. As cultural objects are created along with cultural events, visualizations that can reflect the time development of events are also meaningful.

So, based on our 3D scanned data, we made a time-series visualization of the assembly process of festival floats. Therefore, we have digitally preserved the knowledge on the assembly process that had until then only been passed from generation to generation through word of mouth.

Last year, the Gion Festival had largely been scaled back due to the pandemic. Are you going to continue your research on the festival, and if so, are there any concrete plans?

Prof. Tanaka: Yes, there are. Our software application is already faster compared with existing commercial software to process big data since we use the parallel computing technique - breaking down the big data into several smaller groups and treating them one by one in parallel.

With our recently developed stochastic noise transparentization method, I would like to realize ultra-high-definition computer graphics (CG) and construct an ultra-high-quality virtual reality (VR) of the parade of festival floats within a more dynamic townscape of Kyoto based on the richness of the original data.

Without simplifying or abandoning the data as in the conventional methods, we can keep their original preciseness.

You are also applying the stochastic noise transparentization method to another large-scale project - the digital archiving for the Borobudur Temple Compounds in Indonesia.

Prof. Tanaka: Yes, we do. Borobudur is a UNESCO World Heritage Site, and we are the first foreign scientist team to conduct a 3D scanning of this vast archaeological site in collaboration with the Indonesian Institute of Science (LIPI) and the Borobudur Conservation office.

Therefore, this is a significant international joint research project for the Art Research Center. The Asia-Japan Institute of Ritsumeikan University is also involved in this project.

We plan to create a next-generation, ultra-high-definition virtual reality tour of the Borobudur Temple Compounds that will be made available on the Internet and in local community centers in Indonesia.

Our newly proposed noise transparentization method should be useful to achieve this target since it works well for large-scale laser-scanned point clouds.

Could you please explain how this method works?

Raw laser-scanned point clouds commonly contain "noise" which negatively impacts the visualizations, making the objects look unsharp. For instance, Borobudur lies in the jungle close to the equator, so that strong sunshine and light reflections cause a lot of measurement noise.

By applying the noise transparentization method, we can make the measurement noise transparent, i.e. invisible, instead of eliminating it.

Thus, we can keep all the original points of the measurement data to maintain accuracy while avoiding the fuzziness in the images created.

We can realize precise, high-quality see-through visualization of large, complex 3D cultural heritage objects such as Borobudur by applying this method.

What is particularly fascinating about the Borobudur project?

Prof. Tanaka: It may be interesting to note that Borobudur is a multi-layer temple construction that contains parts invisible to the human eye. Some of the over 2,600 Buddhist reliefs were buried or hidden behind stone walls when reinforcement works were taking place during the Dutch rule in the 1890s.

Fortunately, the Dutch took gray-scale photographs of the reliefs before they became hidden, and we have been using these to visualize the complete structure.

First, we digitize the visible parts into point cloud data by photogrammetry scanning. By integrating GIS technology, we can keep the correct sequence of panels.

With AI technology, we then perform a 3D reconstruction of the hidden parts into point clouds based on the photographs. By using deep learning, we can visualize the original appearance of the reliefs with a 95% accuracy of the reconstructed point cloud and establish a "complete digital archive" from an "incomplete" or partially damaged cultural heritage.

And this exactly - being able to reproduce what the human eye cannot see - is one of the strengths of digital archiving by integrating humanities with information technology.

In the future, I would also like to realize a dynamical time-series visualization of Borobudur's building process, just as we did with the assembly of the Gion Festival floats.

You are also a Science Council of Japan cooperative member and chairperson in a subcommittee. Can you tell us about your role?

Prof. Tanaka: The purpose of the committee is to establish new humanities-science integration research fields based on computer visualization in the ICT age.

I believe that visualization can be seen as a common language for people from different fields because anyone can understand images.

Even here at the ARC, we have experienced struggles to overcome the challenges that people from natural/ information sciences and social sciences often use "different languages". Through visualization, mutual understanding can be reached more easily.

To promote the Art Research Center's model for integrating humanities and information science, we hold a Science Council of Japan open symposium called "Visualization that contributes to the Creation of Scientific Knowledge" twice a year. We plan to finalize our proposal within a year or two and submit it to the Japanese government for future policy decision-making.

Prof. Tanaka, thank you very much for your time. We are very much looking forward to the further development of your projects.

(This interview was conducted by Yinzi Emily Li.)