Working With The NimbleAI Project To Push The Boundaries Of Neuromorphic Vision

At the end of 2022, the EU kicked off a cool project that aims to implement neuromorphic vision. But what is that? Let’s take a deeper look at the project and our contribution.

First, if you are not familiar with Codasip Labs, I want to mention this briefly. Codasip Labs is in fact our innovation hub where we explore new technologies and try to contribute to the technology of the future. With its very forward-looking focus, naturally the NimbleAI project was a great fit for us to get involved.

So, what is it? NimbleAI is a 3-year, 10 million Euros, research project collecting partners within commercial organizations and academia across the EU and the UK. The project is funded by the EU and aims to design a neuromorphic sensing and processing 3D integrated chip. This is all inspired by the detection of light in the eyes and the processing of visual information in the brain. Our contribution is to develop a customized RISC-V core for the project. As the European RISC-V leader with strong expertise in developing customizable RISC-V IP, we were of course happy to take on this task.

Technology inspired by biology

In summary, the ambition of this project is to improve the energy efficiency and performance of next-generation neuromorphic chips that sustain event-based vision. If everything goes according to plan, this will result in the creation of an integral neuromorphic architecture to efficiently run accurate and diverse computer vision algorithms for endpoint devices. These are typically constrained in resources and area. Use cases include hand-held and battery-powered medical imaging devices, smart monitors for autonomous vehicles, or even wearable eye tracking glasses.

Using biological systems as a “golden benchmark” for electronic systems is a recent technological trend. And inspired by biology, NimbleAI is leveraging the biological eye-brain system for its system architecture. The project is implementing an always-on sensing stage, highly specialized event-driven processing kernels and neural networks to perform visual inference of selected stimuli using the bare minimum amount of energy.

Developing a customized RISC-V core for neuromorphic computing

Our role within the project is to explore RISC-V ISA (Instruction Set Architecture) extensions for in-memory accelerators. The accelerators are provided by our partner CEA. RISC-V customizations help to improve the performance and reduce the power consumption of the neuromorphic chips. To achieve this, we make use of our Codasip Studio tools and the CodAL architecture description language. These tools make customization very straightforward. The core we develop can then be mapped on an eFPGA from our partner Menta SAS. This is similar to what we described in this recent article on extending RISC-V processors in the field to enable post-silicon customization.

The NimbleAI team gathered for a project meeting. (Source: NimbleAI)

Accelerating game-changing technologies

So within the NimbleAI project, our team is working on creating some of the next-generation RISC-V-based AI processors. This is surely a perfect example of the work we do within Codasip Labs. We created this innovation hub to support the development and commercialization of technical innovations. We focus on key applications including security, functional safety, and AI/ML. Our mission? To identify and build the technology that extends the possibilities of custom compute. We want to accelerate the time to market for uniquely better products with customized, domain-specific designs. At the end of the 3-year NimbleAI project, we expect to see major performance gains compared to the current standard of CPUs or GPUs processing frame-based video. The project will enable new functionalities and practical implementations to unlock more advanced AI and computer vision algorithms and applications.

Within Codasip Labs, we collaborate with universities, research institutes and other companies, for example through different EU-funded projects. We also have a University Program that supports students and researchers to advance solutions in different applications.