SCAN4RECO is a 3-year Research and Innovation Action (2015-2018) where 9 partners from 5 countries collaborate in developing ICT solutions for efficient restoration and prevention of degradation of cultural heritage objects. The need to preserve, provide advanced access to and understanding of cultural heritage is of utmost importance, especially when considering its wealth throughout Europe.

The SCAN4RECO project has entered the New Year 2018 strongly progressing towards the integration of all research reults produced over the past two years into a complete solution.

A number of research results have been produced, disseminated and demonstrated to public and in Open Access literature. The most interesting ones are described below alongside demostration videos outlining main concepts and with links for finding more infomation about those technologies.

We are happy to announce that the Oculus supporting version of the VR museum is now available. The design of the Scan4Reco Virtual museum comes from a blend of the principles learned from making museums in the physical world and the perspectives that are offered by the digital one. The visitor starts his tour at a virtual lobby. Structural elements such as walls, slabs and roof are in place to create a sense of reality. Access points such as information and “ticket” desks simplify the navigation. The penetration of “natural” light cre-ates familiarity with similar spaces in the physical world and gives a sense of environmental quality.

Scan4Reco Virtual Museum has been demonstrated at EuroVR conference at Laval Virtual Center in France on 12–14th of December 2017. Professionals and Researchers from Virtual Reality, Augmented Reality and Mixed Reality (VR/AR/MR) field had the opportunity to interact with the Virtual Museum, share ideas and give feedback. The audience was enthusiast by the possibilities of such museum, while testing navigation system and manipulating 3D scanned objects resulting from the Scan4Reco project.

Utilizing appearance and surface measurements on artificially aged samples one can model ageing phenomena over time. Architectures based on Conditional Generative Adversarial Networks (cGAN) have been implemented by CERTH-ITI in order to model degradation over time. In the illustrated example in Figure 1, a cGAN has been trained from RTI color images on bronzes samples at three different ageing rounds. The cGAN model can realistically infer the appearance of bronze as it deteriorates over time.



Application of ageing model on a bronze panel in time (left to right)

The cGAN model is comprised of two subnetworks, namely, a Generator and a Discriminator, where both are CNNs. The Generator takes as input the initial color image of a material and a target time, and generates a new image which is an estimation of the degraded material at that time. The goal of the Generator is to increase the error rate of the Discriminator, while the role of the Discriminator is to learn to distinguish between the real degraded images from the fake (i.e. generated by the Generator). As the accuracy of both networks is getting better during training, the quality of the generated images increases.



cGAN-based ageing model architecture with Generator-Discriminator as Convolutional Neural Networks (CNN)

Texture simulation can be a useful guide for conservators and restorers enhancing their intuition and the non-destructive interference. In this direction, an aging simulation method which models the impact of physical forces using a particle-based approach. Linear and not linear texture blending has been evaluated in the Texture Blending step. The realism of simulation is enhanced by using a non-linear texture blending. The non-linearity captures the natural increase of the decay material which are “nutrients” evolving in the natural environment. As a case study in texture simulation a metallic sculpture has been used. A similar approach to the decay simulation can be followed for restoration. During restoration and following the same framework, a predefined texture is used as the deposed material so as to restore the decayed region.


The figure shows a simulation architecture diagram. Starting from a list of vertices and faces, texture mapping with the initial texture is followed. Then, according to the blending procedure, either linear or non-linear, texture blending is performed. The texture simulated object is fed into lighting model to extract the final simulated Cultural Heritage artwork.

Description of results have been reported in public project deliverables:
D.3.2 A guide for multi-material palette preparation
D.3.3 Procedures and application of artificial ageing

Demo video of forward and backward mu-ton ageing simulation

The Global and Fast Coarse Scan developed by CERTH-ITI is used to produce a 3D digital replica of the original item using the data captured by the depth sensor. The computed digital surrogate provides a coarse approximation of the items surface and appearance and contributes to the proper documentation of the Cultural Heritage item. A complete pipeline for holistic 3D scanning of an artwork has been developed.



Firstly, the artwork is rotated by a controlled rotary stage with a constant rotation step (e.g., 5°) performing a complete rotation (i.e. 360°). In each rotation step the depth sensor captures a depth map and a color map of the rotated object. After a complete rotation, a series of color and depth maps are produced which correspond to the artwork's different multiple views. The extracted depth maps are processed so as to discard areas based on surface normal information and/or Sobel filtering. All the recorded views are transformed in the same coordinate system using the kinematics of the rotary stage. Then, surface reconstruction is performed on the accumulated point clouds in order to extract a watertight surface of the object followed by texture mapping performed on the meshed 3D model so as to reconstruct the texture of the artwork object. Color maps used in texture reconstruction are corrected using samples obtained by the UV-VIS spectrometer.

Demo video of Fast Holistic Surface Scaning

One of the main components of the Scan4Reco system is the 3D scanning of the object geometry. It serves as a pre-requisite for being able to visualize together multiple results from a variety of surface and penetrating scanning of small parts of the object. Object geometry serves also as a reference for simulated prediction of future degradations of the object over prolonged periods of time. Such changes involve both physical erosion of the surface as well as chemical changes that affect the steadiness of the object surface and give raise to speeding up of the object deterioration. Since such changes occur very slowly, having a very accurate and high-resolution 3D object representation is even more important.



Photogrammetry as a technology of making depth measurements from raster photographs, has been used since the birth of modern photography in fields such as topo-graphical mapping, architecture and archaeology. It is also ideal way to obtain 3D information in situations where it is not possible to use other 3D scanners, e.g. in inaccessible locations, due to extreme sizes etc. It is ideal for the recording of translucent surfaces like alabaster and marble. Due to the composite nature of the image capture, color and form can be extracted from the data. Until recently achieving highest resolution recording of surface for facsimile production and featureless, reflective and dark surfaces was not feasible. With recent software developments (e.g. by Pix4D Mapper, Autodesk ReMake and many other ones) it became possible through improvements to photogrammetry technology to become soon the dominant method for recording at risk cultural heritage in 3D and color.

The photogrammetric 3D modelling provides not only the precise representation of the object geometry, but offers also a reference for positioning partial scans from other modalities. It also captures the object condition at a time that can be then aged artificially through digital simulation. The 3D modelling provided by RFSAT uses up to 50MPixel raster images taken with high-overlap on a precisely-controlled grid using robotioc techniques, thus providing high number of matching features among many images. The precise positioning and orientation of the camera in three dimensions (repeatable to single centimeters) is achieved by using a computer controlled mechanical arm. Images are then processed either locally (rough model only, due to a limited computational power of the rack PC) and/or using remote processing server where it can take advantage of the high processing power boosted by CUDA cores of the multiple Nvidia GTX 1080TI graphics cards.



A number of 3D models produced by RFSAT, including those specifically made for SCAN4RECO and otehr research projects funded by European Commission through FP7, COST and Horizon'2020 programs, can be previewed in detail with a Gallery3D virtual reality application by RFSAT built using Unreal 4.18 engine.

Reference Publications:

• Artur Krukowski and Emmanouela Vogiatzaki, High Resolution 3D Modelling of Cultural Heritage, 12^th International Conference on non-destructive investigations and microanalysis for the diagnostics and conservation of cultural and environmental heritage (ART’17), Politecnico di Torino, 22-24^th of November 2017, Torino, Italy.
  
• Krukowski Artur and Emmanouela Vogiatzaki, User Interfaces and 3D Environment Scanning for Game-Based Training in Mixed-Reality Spaces, 6^th EAI International Conference (ArtsIT’2017), Interactivity and Game Creation , 30-31^st of October 2017, Heraklion, Greece.
  
• Artur Krukowski and Emmanouela Vogiatzaki, UAV-based photogrammetric 3D modelling and surveillance of forest wildfires, Workshop on “UAV & SAR: using drones in rescue operations”, ISA, Rome (Italy), 29^th of March 2017
  
• Krukowski, Artur; Vogiatzaki, Emmanouela, Horizon 2020 “Scan4Reco” Project" Project Overview, E-Space 3^rd International Conference Cultural Heritage: Reuse, Remake, Reimagine, Berlin, Germany, 21-22^th November 2016 (Session Networking Session of EC-funded Projects),
  

	Demo video of Automated High-Accuracy Modelling using 3D Photogrammetry
	RFSAT's Gallery3D Application

In order to facilitate all the functionalities proposed in the Scan4Reco project, CERTH-ITI faced the critical issue of amply generated data, but not a method of interpreting them. For this to be possible, an annotation process that would label the data with existing degradations was necessary. Therefore CERTH-ITI has developed a tool for the SCAN4RECO project that enables conservators and other relevant experts of the field, to locate and define the degradations, irrespective of the data format.



Using such a procedure, one can extract knowledge about the degradations and their specific data representations, and this knowledge will be subsequently used in the Decision Support System, in order to assist conservators. Figure above depicts the annotation process on a bronze statue, by designating areas on top of it and assigning degradations to them from a list.

In order to achieve high precision scanning of 3D CH models in the context of Scan4Reco project, a mechanical arm is deployed with sensors mounted on it. The motion planning model that we developed consists of two stages. The first stage is a node-based search on a triangular mesh of a 3D model surface. The path search is accomplished by the A* algorithm which retrieves the optimal geodesic path between a starting and a goal point.



For the second stage of the motion planning model, we deployed the Gazebo simulation interface of Robotic Operating System (ROS). ROS includes implementation of several sampling-based algorithms such as PRM, RRT, EST, SBL and KPIECE, that provide motion plans for the robotic arm. We used the Dynamic Time Warping (DTW) algorithm to merge these two stages. The different paths extracted from ROS are compared with the optimal geodesic path, estimated by the A* search, through DTW in order to leach into the optimal trajectory sequence for the robotic arm movement.

The Gazebo simulator precisely reflects the real-world case. The Motion Planning, the Inverse Kinematic computations and the optimal path estimation correspond to the actual robotic arm moving process. The robotic arm is provisionally used to demonstrate a conceptual mode of the motion planning engine developed.

Video demonstration of precise scanning with Robotic Arm

In the context of Scan4Reco, CRS4 and UNIVR have developed a novel multi-spectral reflectance transformation imaging (MS-RTI) framework for the acquisition and direct analysis of the reflectance behavior of heterogeneous artworks. Starting from free-form acquisitions, we compute per-pixel calibrated multi-spectral appearance profiles, which associate a reflectance value to each sampled light direction and frequency. Visualization, relighting, and feature extraction is performed directly on appearance profile data, applying scattered data interpolation based on Radial Basis Functions to estimate per-pixel reflectance from novel lighting directions.


Figures (a),(b),(c) are examples of RBF relighted images (with lx = 0.3,ly = 0.45) estimated with different values of R (0.1,0.3,0.6). Top left one shows dense reflectance maps obtained with corresponding values of R in a circled location. Small radii result in weighting mainly the closest image, large radii result in smoothed details and worst shadows definition. Small radii create sparse reflectance maps, too large create oversmoothed maps,(d) PTM relighting from the same light direction. Here shadows and light direction related information tend to disappear.

They demonstrated how the proposed solution can convey more insights on the object materials and geometric details compared to classical multi-light methods that rely on low-frequency analytical model fitting eventually mixed with a separate handling of high-frequency components, hence requiring constraining priors on material behavior.

The flexibility of our approach has been demonstrated in various case studies within Scan4Reco. In particular, we have illustrated in a publication the results obtained on two heterogeneous case studies, a painting and a dark shiny metallic sculpture, that showcase feature extraction, visualization, and analysis of high-frequency properties of artworks using multi-light, multi-spectral (Visible, UV and IR) acquisitions.

The method and related case studies have been presented at The 14th Eurographics Worhshop on Graphics and Cultural Heritage, the major scientific event on the application of computer graphics techniques to cultural heritage. Our contribution has won the best paper award.

Reference:

Andrea Giachetti, Irina Ciortan, Claudia Daffara, Ruggero Pintus, and Enrico Gobbetti, "Multispectral RTI Analysis of Heterogeneous Artworks", 14^th Eurographics Worhshop on Graphics and Cultural Heritage, pages 19-28, September 2017. https://doi.org/10.2312/gch.20171288

EG GCH 2017 Best Paper Award: http://vcg.isti.cnr.it/wggch/best.html

Demo video of Data calibration with RTITool

Demo video of H-RTI stack calibration with RTITool

Demo video of Freeform Acquisition of Calibrated RTI

Demo video of Multi-spectral RTI ring

Demo video of Multi-spectral RTI Dome

Since the beginning of the project, the SCAN4RECO consortium has published over forty (40) peer-reviewed articles in prestigeous journals, presenting them at several conferences and workshops, nearly thirty of those during the second year of the project. The list of those can be found at:

Project WEB site:

http://www.scan4reco.eu/scan4reco/content/publications

Open Access Repositories:

https://zenodo.org/communities/horizon2020-reflective7-scan4reco/

http://cordis.europa.eu/project/rcn/197123_en.html

Institutional WEB sites of project partners:

RFSAT: http://www.rfsat.com/index.php/en/results/publications.html

CRS4: http://www.crs4.it/vic/cgi-bin/project-page.cgi?acronym=Scan4Reco

NOTE:The SCAN4RECO project is fully compliant with Open Access guidelines of the European Commission with respect to dissemination of reseach output of projects funded by the Horizon 2020 research framework, as outlined at: http://ec.europa.eu/research/openscience/index.cfm?pg=openaccess