AI-based system for extracting and generating supplementary data from analog data carriers

Abstract

The digitization and restoration of analog media pose significant challenges for cultural and commercial stakeholders. Manual processes are time-consuming and resource-intensive. Artificial intelligence offers new potential for automated restoration, precise metadata extraction, and the preservation of authenticity, integrity, and data protection in response to the growing need for resilience against deepfake technologies. The objective of this work is the development of an Artificial Intelligence (AI)-based, web-driven media archive that digitizes analog carriers, detects manipulations, converts content into high-resolution video formats, and integrates multimodal analyses of image, audio, and speech. The application provides scalable services and an intuitive user interface, incorporating functionalities such as media management, color correction, restoration, and object detection. A particular focus is placed on frame-based restoration. Video material can be ingested as files, live streams, or via capture cards. Individual frames are analyzed for defects using a You Only Look Once (YOLO)-based model; damaged regions are subsequently restored using custom-developed algorithms. Automatically generated masks ensure precise defect treatment. Training with a dataset of 8,935 manually annotated High Definition frames resulted in a high level of detection accuracy. The results were validated using generated reference images as well as visual inspection to ensure that defects are effectively identified and no longer perceptible. Overall, the findings highlight the potential of AI-driven technologies for efficient and high-quality restoration and provide a foundation for scalable, future-oriented media archive solutions.