Synthesizing Computer Vision Designs To Hardware

Computer vision is one of the hottest markets in electronic design today. Digital processing of images and video with complex algorithms in order to interpret meaning has almost as many applications and markets as there are uses for the human eye.

The biggest problem that designers face is that the computer vision system requirements and algorithms change quickly and often. Even the target technology might change during the course of a project. This means that using a traditional RTL design flow is almost impossible. Starting over within a RTL flow every time there is a change causes unacceptable delays in the production schedule.

If managing change is not enough of a challenge, there are technical challenges as well. Many computer vision systems need real-time speed. For example, the radar and LiDAR sensor analysis system in a car cannot tolerate delays in decision making. Real-time speed can mean billions of calculations per second, yet designers are also trying to keep power consumption to a minimum. And to make matters even more difficult, camera resolutions and frame rates are expected to double every few years.

A look at CNNs
Computer vision systems must understand what they are “seeing” in order to make correct decisions. This understanding comes from deep learning which is commonly implemented using convolutional neural networks (CNNs). CNNs are first used for training the system by feeding in massive amounts of images that are processed on GPU farms in a lab or data center in order to tune the weights of the network. This iterative process provides many images of the same types of objects with different attributes like orientations, aspect ratios, and clarity, to allow the network to “learn” how to detect and classify these objects.

A CNN that is designed for hardware in the field uses inferencing to identify images. It runs convolutions using the weights identified from the training in order to detect and classify objects in any image. Because many computer vision systems are deployed in cars, industrial equipment, or consumer products, which cannot depend on a lab full of computers, inferencing solutions are the focus of hardware designers. A typical CNN architecture consists of layers of convolutional filters, pooling, and fully connected layers whose lowest levels of implementation are matrix multiplication and dot-product functions, which are a very good fit for high-level synthesis (HLS).

Getting to hardware
High-level synthesis, such as the Catapult HLS Platform, allows hardware design teams to quickly synthesize computer vision algorithms written in C++ code into RTL. This provides a solution for hardware designers that generates high-quality RTL from C++ descriptions that target ASIC, FPGA, and embedded FPGA solutions. The platform delivers the ability to check the design for errors before simulation, provides a seamless and reusable testing environment, and supports formal equivalence checking between the generated RTL and the original source. This ensures fast design and verification and delivers power-optimized RTL ready for simulation and RTL synthesis. By employing HLS, teams can:

• Handle late-stage changes: easily change C++ algorithms at any time and regenerate RTL code or target a new technology.
• Evaluate hardware: rapidly explore options for power, performance and area without changing source code.
• Accelerate schedules: reduce design and verification time from one year to a few months and add new features in days not weeks, all using C/C++ code that contains 5x fewer lines of code than RTL.

Addressing the challenges
Perhaps computer vision design teams are initially drawn to HLS because the Catapult HLS Platform generates correct-by-construction RTL from C++ descriptions. The tool builds concurrent RTL modules from C++ classes and functions that are synthesized into concurrent, clocked processes (no need for multi-threaded design and debugging). But, a closer look at the tool demonstrates why it is so useful for computer vision design:

• Automatically add interface protocols: instead of coding interface protocols in source code, designers can use Interface Synthesis to add a timed protocol to untimed C++ function interface variables. The tool supports typical protocols such as AXI4 video stream, request/acknowledge handshaking, and memory interfaces.
• Close timing: HLS adds time to the C++ algorithm using a process called scheduling. Based on the selected technology and clock frequency, scheduling examines the operations that the algorithm describes and decides which clock cycle they execute.
• Analyze performance bottlenecks: one of the key questions that C++ algorithm developers have is, “What happened during synthesis to RTL?” In particular, they need to evaluate performance and identify any bottlenecks in the synthesized design. To answer this question, Catapult provides a Schedule view for interactive analysis.
• Optimize memory: Catapult automatically infers memory from arrays and maps to ASIC or FPGA memories/registers. The designer has control over memory bandwidth and partitioning, including word width, packing mode, and controlling the number of memories.
• Share resources: to keep area to a minimum, Catapult automatically shares resources by looking for explicit mutual exclusivity in the code and by applying design constraints.
• Explore and optimizing power: using PowerPro technology, Catapult generates power-optimized RTL to converge rapidly on an ideal design solution. Designers can interactively explore micro-architectures using constraints to evaluate power, performance, and area tradeoffs.
• Leverage libraries: when designing computer vision systems, it not hard to notice that there are low-level building blocks instantiated many times within the same design, such as convolutional filters, interfaces, and pooling. The Catapult HLS Platform provides parameterized C++ libraries for these low-level, basic imaging operations.

The world of computer vision design is constantly changing, which means traditional RTL flows cannot keep up with schedules. In order to quickly respond to changes and to accelerate design and verification, high-level synthesis offers a path to synthesizing computer vision designs to hardware.

To learn more about computer vision and HLS, view our new whitepaper here.