Enhancing live football broadcasts by eliminating camera operator distractions with AI

"Our new invention is an algorithm adapted to detect video operators," says KTU professor Rytis Maskeliūnas, one of the creators of the innovation.

Serhii Postupaiev, another members of the research team, highlights that the presence of cameramen in the frame is a frequent issue in football broadcasts due to the complex nature of live sports coverage and the numerous cameras around the stadium.

"The number of camera points in prestigious tournaments can start from nine and numerous overlapping views that contribute to the visual distraction issues. These issues severely constrain the cameramen team, as they constantly have to film the game while avoiding capturing each other, potentially leading to a loss of context in some game moments or making the broadcast less dynamic and immersive," explains Postupaiev.

Eliminating visual distractions

To address this problem and remove unwanted objects during a live broadcast, KTU scientists designed and implemented an end-to-end system.

For its functioning, the YOLOv8 model, a state-of-the-art object detection system known for its speed and accuracy, was employed. YOLOv8, which stands for "You Only Look Once," can detect and classify objects in images in a single pass, making it ideal for real-time events such as live football broadcasts.

"It works by dividing the image into a grid and predicting bounding boxes, class probabilities, and segmentation polygons for each grid cell. This enables it to identify and segment cameramen," says Serhii Postupaiev, who recently graduated from KTU with a Master's degree in Artificial Intelligence in Computer Science.

To train the YOLOv8 model to accurately detect and segment cameramen during football matches, a dataset had to be created.

"I created this dataset to include a diverse range of cameramen with different sizes, shapes, and types of equipment, captured under various conditions and at different stages of the game. Now YOLOv8 uses this dataset to identify where cameramen are in the video frames," adds Postupaiev.

As the inventor explains, this process was needed to create the foundation for the actual removal of the operators. For this purpose, video inpainting technology was used.

"The term inpainting in deep learning refers to the process of reconstructing lost or deteriorated parts of images and videos. Specifically, in this case, it is used for removing cameramen from football video broadcasts," says Postupaiev.

Artificial intelligence (AI) and computer vision-based technology analyses the video frames to detect unwanted objects such as cameramen and fills the removed areas with relevant background details. The modified frames are then streamed back to viewers, ensuring a more immersive and professional broadcast.

Maskeliūnas adds that on television servers, this algorithm could process the recorded image before it is broadcast on air with a delay of a few seconds from the actual captured moment, which is still considered a live broadcast. He believes that, as the equipment improves, AI will fill this time gap perfectly.

Shifting the focus from simply capturing the action

With this new technology, watching football matches at home will be significantly improved. One of them is a smoother viewing experience.

"The broadcast will feel more polished and professional without disruptions caused by cameramen appearing where they shouldn't. This improvement will reduce the number of cases where important moments of the game are missed due to distracting shots," highlights Postupaiev, who received his Master's degree with this project.

According to Postupaiev, further research in this area could usher in a new era of sports broadcasting, shifting the focus from simply capturing the action to creating a fully immersive and uninterrupted viewing experience.

"By implementing cameramen inpainting, broadcast companies can explore innovative camera angles, perspectives, and effects, bringing games to life in new and exciting ways," he says.

Additionally, cameramen inpainting can extend beyond live broadcasts to enhance pre- and post-match analyses, offline highlight reel processing, and restoration of archival footage.

"This could even breathe new life into old recordings of classic matches," adds a KTU graduate.

Invention is not limited to football -- it can be applied to other sports with similar broadcasting challenges. Dynamic sports like futsal and basketball, which require immersive broadcasts, can also benefit from this technology.

"This is yet another illustration of what modern AI applications can do. We often hear about medical applications, but here we have a consumer-oriented approach to editing images we don't like. In the future, such technology will be able, for example, to remove advertisements or replace them with other ones thus constantly updating the content with a level of precision that the human eye will not notice," says KTU Faculty of Informatics professor Maskeli?nas.