Design And Security Challenges for VR

Virtual reality is no longer just for gamers, and as this technology is deployed in everything from health care to industrial training, the requirements for processing more data faster over a high-speed connection are growing.

Designing these devices continues to be a study in contradictions. They must be extremely low power, with a small enough batteries to make them comfortable to wear. But they also must be able to process large amounts of streaming image data in real time, and with a screen refresh rate high enough to prevent motion sickness. In addition, they must last long enough between charges to make it attractive to consumers. Also, they must stay relatively cool, because the device is worn over the face, and they need to be secure, include high-speed I/O and advanced communications technology — and still be affordable.

“VR applications require sophisticated and complex devices,” said Ron Press, director of Technology Enablement for the Tessent division of Siemens EDA. “To support the intense computing VR applications demand, the devices need a lot of processing power. Thus, devices for VR are often large with duplicate processors.”

In fact, prior to the 7nm node, VR pioneers commented that chip technology had not progressed enough to meet all of those goals. But as chips continue to scale, and as they are architected into advanced packages with customized and heterogeneous compute elements, there are enough performance and power improvements to make this technology much more useful for a variety of applications. That is borne out in the growth numbers. According to Allied Market Research, the worldwide market for AR/VR chips is expected to reach $7.76 billion by 2026 worldwide.

One of the big drivers will be 5G connectivity, which will enable faster download and upload speeds anywhere. Today, most VR systems are wired and/or connected over Wi-Fi. But as these devices increasingly are untethered with 5G, a variety of new applications will become possible, including automotive, architecture, manufacturing, digital marketing, entertainment, retail, tourism, and much more.

This opportunity has not been lost on systems companies and chipmakers. Leading technology companies involved with VR and VR chips include Apple, AMD Broadcom, HP, Huawei, Meta (Facebook), Microsoft, NVIDIA, Imagination Technologies, Intel, MEDIATEK, Qualcomm, and Samsung, among others.

Different realities
Virtual reality is just one of several related technologies under development. While VR is by far the most recognized term, it falls under the umbrella of extended reality (XR), which also includes augmented reality (AR) and a combination of both AR and VR called mixed reality (MR).

VR requires a head-mounted display (HMD) to experience a 3-D world generated by computer using video and audio signals. In contrast, AR overlays computer-generated content on top of the real object, often with clear glasses. AR can be used to train a technician to troubleshoot a machine. While the technician watches an augmented version of the real machine through the HMD, AR images or video can guide the technician step-by-step through the repairs.

MR, as the term suggests, mixes certain VR and AR elements. This hybrid model makes it possible for the digital and physical world to appear as one integrated 3-D environment. Potentially, as MR matures, the user can interact with the physical world digitally.

A new term, “the metaverse” has recently emerged to mean cyberspace or the digital world. It can be accessed through VR or AR.

In June 2022, the 1,000+ member Metaverse Standards Forum announced, for the first time, the development of open standards for the metaverse. Membership for joining the organization is currently free. One of its goals is to encourage collaboration on spatial computing. Such collaboration in areas such as interactive 3D graphics, geospatial systems, physical simulation and photorealistic content authoring will energize the growth of VR technology.

Designing a VR system
VR systems combine 3-D software and various types of sensors, including those used for motion, audio, and video, as well as an HMI/user interface. Design considerations include force feedback, conductivity, low power, and being compact and light enough to wear. Selecting the right chips and components for the design is important.

“Regarding SoC selection, it depends on the targeted end-product,” said Amol Borkar, director of product management, marketing, and business development for Cadence‘s Tensilica Vision and AI DSPs. “For making prototypes or manufacturing in small volume, there isn’t enough business justification to build an SoC in-house. This rule of thumb applies to vendors building headsets that are usually tethered to a phone or computer. They might be building an all-in-one headset where the main compute inside is still an application processor (AP) running on an Android or a Linux OS variant. In this case, many high-performance, off-the-shelf SoCs (or chiplets) with integrated GPUs and vector DSPs may suffice. Vendors can fabricate the circuit boards and interface with various sensors. There have been a number of designs by well-known vendors that are shipping products using such components.”

However, for companies heavily invested in VR, the trend is to build their own SoCs or systems-in-package. “Building in-house gives the vendor the freedom to design the hardware to match the specs and the KPIs to create a compelling solution,” Borkar said. “In such cases, there may be custom MCUs or DSPs that offload various portions of SLAM (simultaneous localization and mapping) for high performance and low energy consumption. That approach lets the GPU mostly focus on display and rendering, while the main CPU concentrates on housekeeping tasks. These companies then also develop the software stacks, SDKs, and framework to leverage all these components. This results in a rich ecosystem that enables developers to build high-quality applications, creating great user experiences.”

Over the past several years, significant advances in data compression makes it easier to move large amounts of data more quickly. “Using VESA DSC and VDC-M video compression enables AR/VR SoC designers to reduce video buffer size and video interface bandwidth, leading to significant power savings, lower EMI, and a smaller footprint — all while keeping latency extremely low,” said Simon Bussières, director of systems architecture at Rambus. “VESA DSC and VDC-M are visually lossless compression codecs, meaning that when used, the human eye detects no perceptible difference. Both codecs have been rigorously tested, and in 2021 additional research conducted by VESA validated the visually lossless performance of both codecs for stereoscopic 3D use cases, including for AR/VR applications.”

Perhaps the most challenging part is user interface and navigation. Users need freedom of movement, ease of control, and system responsiveness. Also, the HMD must be comfortable to wear without strain on the human body, and it has to have enough performance to prevent the user from becoming nauseous. The device input can be challenging because it involves multiple formats, such as voice, gesture, click, and haptics. Fortunately, for industrial and commercial applications, the motion sickness issue is minimal as users move around less than they do for gaming.

Fig. 1: Many components go into making of a VR HMD. Source: Nexperia (simplified block diagram).

For years, VR developers have used discrete components such as MCUs, various sensors, different wireless connection chips, and DSPs to get the job done. Recently, multiple functions are integrated into fewer chips. For example, Qualcomm has developed the Snapdragon XR2 5G Platform which supports 5G, 8K 360° video, 7 concurrent cameras, and AI, all in one chip.

Security issues
Security is another consideration. No matter how a VR device is connected, that connection makes it vulnerable to attack. Hackers potentially can use the HMD or VR devices as gateways to both home networks and enterprise systems/networks. Once the link has been established, hackers can steal private data and information. Security experts warn against hackers driving by the neighborhood to hack Wi-Fi routers. But with 5G, hackers can launch an attack anytime, anywhere, which increases the vulnerability systems.

“VR provides new ways to visualize data, and in many cases is more portable and provides more connectivity options than traditional displays,” said Mike Borza, scientist and principal security technologist at Synopsys. “For many consumer uses, the security concerns pertain mostly to privacy. When used in an enterprise context, there can be many security concerns such as the protection of proprietary intellectual property and data (such as design models of manufactured products). When used interactively as part of a human-machine interface, there are concerns about protection of the integrity and availability of data while maintaining low overall latency for precise control (e.g., an operator remotely controlling machinery). And in many other applications that are emerging in spaces with confidentiality, there are issues involving privacy and governance requirements (e.g., healthcare).”

Security experts warn about the importance of cybersecurity. “Like any other connected system, virtual reality systems can be a target for cyber threats,” said Steve Hanna, distinguished engineer at Infineon Technologies. “Protection falls into two categories — protocol security and product security. Protocol security protects data while it’s in transmission. For example, data sent to and from a VR headset should be encrypted and authenticated to protect confidentiality and ensure authenticity of the data. Product security protects the system from being hacked. For example, secure boot verifies that only authorized firmware can run on a VR headset. Without product security protection, a compromised headset could infect a laptop, or vice versa.”

That connectivity becomes more vulnerable to attack as these devices become more mobile. “Connectivity today is wired (mostly high-bandwidth USB Type C) or Wi-Fi,” said Borza. “Future Bluetooth versions will have the bandwidth to support these devices. Depending on the connectivity model, link-layer security will be sufficient when a one-to-one correspondence between the VR device and the data source exists. But in many cases, the end-to-end security model makes much more sense, particularly when VR is used to view and interact with remote entities. So we can expect to see a need for high-bandwidth IPsec and TLS secure communications.”

Others point to similar needs. “To safeguard VR with 5G capability, it is important to deploy end-to-end security,” said Gijs Willemse, senior director of Product Management at Rambus. “5G security goes beyond man-in-the-middle and side-channel attacks. You are now also dealing with a potential attack against your (cloud) services over the network. In addition to designing with security software/chips, the VR implementation will need to consider access to personal and private data, as well as advanced firmware updates, device and user authentication, and potentially real-time payments.”

Design tradeoffs
As VR is deployed across different markets and use cases, it is clear that one size will not fit all.  For high-speed gaming, competition in this market has been all about processing speed and latency. For a surgical training application, in contrast, clarity and accuracy are the priorities. An 8K display with force feedback that gives the surgeon the feeling of doing a real operation on a live person is most helpful.

Once the VR design goals are set, the next step is to establish the most suitable test method, model, or simulation to ensure all the parameters meet the goals.

“One common metric is motion-to-photon (MTP) latency,” said Cadence’s Borkar. “MTP measures the time or latency from when a movement occurs to when it is rendered and displayed on the headset’s screen. There are a number of variants to this metric, but the overall idea is the same for most, with the target range being 15 to 20 ms or less for a good VR experience. This is an important metric, as the VR experience is a complicated ‘stack’ going from the camera/IMU sensor to the system bus, followed by SLAM processing, rendering, display, and more. Each phase of the process contributes latencies. However, to help reach the performance target clever tricks can come into play at the various stages. For example, SLAM may use only IMU data for some frames to improve the refresh rate, as cameras are usually 30 to 60 fps. GPUs may use timewarps and spacewarps to render intermediate frames and prevent stalling the pipeline.”

On the test side, tools are being introduced to to target VR technology. “Recently, there has been wide adoption of packetized scan pattern delivery using, for example, the Siemens Tessent TestKompress Streaming Scan Network (SSN) technology for advanced devices such as VR,” said Siemens’ Press. “Instead of testing duplicate cores individually or trying to time a broadcast to all cores, it delivers the same packet skewed in time to them with an on-chip comparator inside each core. As a result, any number of identical cores can be tested without extending test time or taking more test development time. What is also unique to packetized scan pattern delivery is that as test requirements change, the cores selected to be tested in parallel can be optimized on the fly. This provides a lot of flexibility. As these VR designs evolve, the test patterns and solutions for cores can be completed, then the cores to test in parallel can be selected later. Once the cores are selected, the packetized data is processed to provide the patterns to the selected cores in an optimized way that is dramatically more efficient than traditional approaches.”

Conclusion
VR challenges include integration of many design parameters and safeguarding cybersecurity threats. Although they may vary depending upon the application, the parameters typically include software/hardware optimization, selection of VR chips or SoC, power management, HMI, packaging, performance, and testing/simulation. It is important to clearly define the design goals. Additionally, to minimize the risks of cyber threats, VR systems must incorporate the latest security technologies and know-how.

VR and related technologies will continue to improve. New applications will emerge. In a recent experiment, “HoloScenarios” allowed medical students, by wearing the Microsoft HoloLens mixed reality headsets, to interact with a multi-layered, medically accurate holographic patient. Cambridge University Hospitals NHS Foundation Trust (CUH) in the UK, in partnership with the University of Cambridge (Cambridge, UK), and tech company GigXR in Los Angeles, developed the project, which increases learning opportunities for medical students.

There will be many more such applications over the next decade as people learn how to apply this technology. After a long wait and lots of promises, the virtual world is beginning to take shape. 

Resources:

• XR Association
• The Metaverse Standards Forum (metaverse-standards.org)
• SLAM and DSP Implementation (cadence.com)