New services that could be enabled through the processing power of digital TVs and STBs.
Will processors jumpstart 8K?
System-on-chip (SoC) designers and home device makers are experimenting with AI techniques to upscale high-definition or even standard-definition programs in a way that looks natural and realistic.
Here’s how it works. Rather than stretching an image over a wider battery of pixels (causing pixelation) or filling in the new pixels by extrapolating information from neighbouring ones (blurring edges), an algorithm compares two video samples or images of a similar scene: a full-resolution version and a deliberately degraded one. By comparing the two and analyzing the missing data, the AI algorithms and processors can deduce how to reconstruct it.
The reconstruction techniques are then applied to the program you’re watching, filling in details that weren’t captured in the recorded media but might have been present in the real world when the program was filmed, which ironically could be an episode of “Law and Order” where police reconstruct a murder scene with a skull fragment and a burnt piece of paper.
As you can imagine, super resolution like this is no easy computational feat. The internal inference engine of a digital TV (DTV) or set-top box (STB) would have to have access to thousands of samples, as well as being able to differentiate between a snowstorm and an artsy fade out. It would also have to perform this feat in real time without setting off whirring fans. But as techniques like this get refined, it’s easy to see how they will fundamentally change home entertainment.
Programming innovations like this will also, I believe, change how consumers, developers, and manufacturers view the STB. Over 200 million DTVs get shipped worldwide each year, along with at least an equal number of STBs—a category that includes classic set-top boxes from cable or satellite providers and streaming media devices like the Roku Express, and some that combine both. IDC estimates that smart TVs and streaming devices alone account for over one-third of smart devices brought home worldwide.
Still, they are often just there. It can be difficult to remember which remote controls them. By harnessing the performance lurking inside these anonymous aluminium boxes, STBs could become a primary vehicle for new applications and services.
The availability of IP as a primary delivery channel and advances in content production, for instance, open the door to advances in object-based media. With object-based media, individual consumers can obtain an individualized version or a program ranging from conventional tweaks, such as subtitles or audio tracks, to branched narratives (e.g., ‘choose your own adventure’ storylines) and different camera angles in sports or personalized advertising. Various objects can then be combined in the cloud, in the device, or via a combination of the two.
Object-based media standards are still emerging and in the coming years we are likely to see new creative uses of this technology. It will also likely be a public/private experience with object-based programming available at home or at live sporting events.
Working from home during the COVID-19 pandemic has changed consumer behaviours, normalizing the at-home use of videoconferencing applications, which once belonged only in the office. In the home, these applications fit quite naturally in the living room. Smart cameras use immersive visuals that recognize hand and body gestures, enhance, or blur out the background, provide intelligent focus with face tracking, and even enable augmented reality (AR) filters.
Other screens – such as those for smartphones, tablets, and laptops – have had built-in cameras for a long time, but users must have reassurance that their camera-enabled TVs have appropriate levels of security and privacy-preserving technology. Arm is actively working in this area and addressing increased security demands.
Besides conventional gaming consoles, various gaming services are delivered through STBs, either natively in the device or via cloud-gaming providers like Google Stadia and Tencent Games for the China market. Cloud gaming will require STBs to enable video and command streaming and low-latency features. Meanwhile, native gaming platforms need STB devices with much improved central processing units (CPUs) and graphics processing units (GPUs) for better gaming experiences, but have the benefit of not requiring a fast, reliable broadband connection. Recent advances in mobile graphics also make high-performance gaming more accessible than ever.
Another trend we have observed during the COVID-19 pandemic is the increasing number of people exercising from home. To make these applications work, AI algorithms for body-position tracking will be required, alongside an always-on voice subsystem for voice command and digital assistant features.
To seamlessly enjoy these new experiences on STB devices, always-on voice control will be a particularly important feature. Voice UI, widely available already on today’s STBs, enables more complex applications and experiences, such as the fitness applications described above, as well as basic voice commands for an improved user experience (UX). Greater machine learning (ML) capabilities in STB devices could be important, particularly across CPU and GPU, advancing AI functionality to minimize power consumption in standby mode and providing a responsive experience.
New experiences will also mean more threat of piracy and attacks. With video processed by more engines in the chipset, the big challenge is ensuring that all the content is secure across the entire platform. This means protecting the content, operating system, applications, and assets from attacks. In addition, due to new applications on STBs, there is now far more user data stored on devices, such as personal information and assets for social media accounts, payment, and video. There must be more advanced security requirements to protect user data, with STB manufacturers looking at more sophisticated security protections across devices. This is likely to be similar to how we protect user data on today’s smartphones.
DTVs and STBs are opportunities hiding in plain sight. The large viewing area of modern DTVs, combined with the omnipresence of STBs provide hardware designers and software developers with a canvas for the smart home. Arm is providing the IP and solutions needed to enable the advanced capabilities and new use cases that we will soon see emerge.
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
Leave a Reply