A novel neural network for preserving cultural heritage via 3D image reconstruction

Relief carvings or relief sculptures are cultural heritage objects with figures that protrude from a background such as a wall or slab, creating a sense of depth. Commonly found at historical sites worldwide, these artworks are considered to be of immense historical and cultural value. Unfortunately, many such relief carvings at heritage sites across the world suffer from varying degrees of damage and deterioration over time. While modern 3D scanning and photogrammetry techniques can digitally preserve their current form, they cannot restore the original appearance of these carvings before damage. Additionally, traditional methods for restoring them are laborious, and require extensive manual intervention and specialized knowledge.

One promising method is 3D digital reconstruction of these reliefs from old photos, captured before the damage or deterioration. Unlike 3D sculptures or 2D paintings, reliefs have a shallow depth and are meant to be viewed from the front or either side. This means that a single image can effectively provide most of the information required for 3D digital reconstruction algorithms.

In a breakthrough, a multinational research team led by Professor Satoshi Tanaka from the College of Information Science and Engineering at Ritsumeikan University, Japan, along with Dr. Jiao Pan, from the University of Science and Technology Beijing, China, developed an innovative multi-task neural network for 3D reconstruction and digital preservation of reliefs using old photos. "Previously, we proposed a 3D reconstruction method for old reliefs based on monocular depth estimation from photos. Although we achieved 95% reconstruction accuracy, finer details such as human faces and decorations were still missing. This was due to the high compression of depth values in 2D relief images, making it difficult to extract depth variations along edges. Our new method tackles this by enhancing depth estimation, particularly along soft edges, using a novel edge-detection approach," explains Prof. Tanaka.

The team also included Prof. Liang Li from Ritsumeikan University and Prof. Xiaojuan Ban from the University of Science and Technology Beijing. Their study was presented orally at the international conference ACM Multimedia 2024, held in Australia in October, and was published in MM '24: Proceedings of the 32^nd ACM International Conference on Multimedia on October 28, 2024.

The proposed multi-task neural network performs three tasks, namely semantic segmentation, depth estimation, and soft-edge detection, which work together to enhance the accuracy of 3D reconstruction. The core strength of the network lies in its depth estimation, achieved through a novel soft-edge detector and an edge matching module. Unlike the conventional binary edge classification, the soft-edge detector treats edge detection of relief data as a multi-classification task. Edges in relief images not only represent changes in brightness but also variations in curvature, known as "soft edges." The soft-edge detector determines the degree of "softness" of these edges in relief images, enhancing depth estimation.

The edge matching module comprises two soft-edge detectors that extract multi-class soft-edge maps and a depth map, from an input relief photo. By matching and detecting differences between the two maps, the network focuses more on the soft-edge regions, resulting in more detailed depth estimation. Finally, the network optimizes a dynamic edge-enhanced loss function, which includes loss from all three tasks, and produces clear and detailed 3D images of reliefs.

The researchers applied this innovative model to reconstruct the hidden reliefs of Borobudur Temple. "The wall reliefs on the ground level of the Borobudur Temple, a UNESCO World Heritage Site in Indonesia, are covered by stone walls due to reinforcement work carried out during the Dutch colonial period and cannot be viewed. Our multi-task neural network successfully reconstructed these hidden sections of Borobudur's ground-level reliefs from surviving old photographs. Through computer visualization and virtual reality, our research now allows virtual exploration of these unseen treasures," says Prof. Tanaka, highlighting the potential impact of their work.

Discussing the future implications of these findings, he says, "Our technology holds vast potential for preserving and sharing cultural heritage. It opens new opportunities not only for archeologists but also for immersive virtual experiences through VR and metaverse technologies, preserving global heritage for future generations."