Khronos: Open Standards Powering The Future Of Embedded Vision

At the recent Embedded Vision Summit, one of the presentations was by Neil Trevett of The Khronos Group. This is a standardization body and is a non-profit. He said that his day job is at Nvidia. It has been creating standards for 20 years with a focus on low-level hardware acceleration APIs for 3D graphics, parallel computation, augmented reality (AR), virtual reality (VR), and machine learning.

Probably its most famous standard is OpenCL (Open Computing Language) which is a framework for writing programs that execute across heterogeneous platforms consisting of central processing units, graphics processing units, digital signal processors, field-programmable gate arrays, and other processors or hardware accelerators.

Here’s the “Where’s Waldo?” chart of members. Can you spot your company’s logo? Cadence is 4th down on the right-hand edge, by the way. It is an interesting mix of companies. AMD, Intel, Google, no surprise there. But Ikea? Target?

The standards often fit together like Lego bricks. At the bottom of the above diagram, the standards are lower-level APIs that give access to hardware. For example, OpenGL is 30 years old this month.

OpenGL is 30 years old in June. It was great back in the day, but today people want more direct access to the GPU hardware. Vulkan is much lower level, and so can do multithreaded applications. Vulkan is the only open standard, one is Apple, one is Microsoft, but Vulkan is everywhere.

An important standard in the context of vision and AI is SYCL, which is standard C++ doing everything with template libraries. It supports the OpenCL backend, of course, but not exclusively. It supports other backends too. So you can get from AI frameworks such as TensorFlow to implementations that run on pretty much any hardware. SYCL is quite a recent standard, with the first version launched in February 2021.

OpenVX is a high-level graph-based abstraction for efficient portable vision processing. I have written about this before. See my 5-year-old post See Further by Standing on the Shoulders of…OpenVX.

Khronos is doing an increasing amount of work in standardizing interfaces used in safety-critical applications such as avionics and automotive. For example, it is taking the Vulkan API and adapting it to become Vulcan SC (for safety-critical) by removing a lot of functionality that is only needed for games and making it deterministic (to give predictable real-time performance and consistent results). All the pipelines are compiled ahead of time, and in a cache, so it is possible to pre-allocate all the memory you will need. When you put your foot on the brake in a car is not the time to run garbage collection.

Neil ended with a plea for people to get involved. As I said at the beginning, the standards are open and royalty-free, so anyone can use them. But as anyone who has ever been involved in a standardization effort knows, there is a huge amount of work to create the standard, in many cases requiring a huge amount of very specialized knowledge. So if you have a lot of camera knowledge or safety-critical knowledge, there is a lot of activity in those two areas in particular today.