Deploying Artificial Intelligence At The Edge

By Pushkar Apte and Tom Salmon

Rapid advances in artificial intelligence (AI) have made this technology important for many industries, including finance, energy, healthcare, and microelectronics. AI is driving a multi-trillion-dollar global market while helping to solve some tough societal problems such as tracking the current pandemic and predicting the severity of climate-driven events like hurricanes and wildfires.

Today, AI algorithms are primarily run at large data centers, that is in the cloud. For this intelligence to be used at the edge, data must be transmitted to the cloud, analyzed there, and the results transmitted back to the edge – a device in the field of operation, whether it is a sensor tracking the strength of a bridge, a mobile phone, a medical implant, or an autonomous vehicle.

The problem with the current approach of using AI primarily in the cloud is that it consumes much energy and can introduce data transmission delays and security vulnerabilities. Yet the solution – making the edge itself more intelligent – has its own challenges because of stringent edge device size, cost, energy budget and compute power constraints. These challenges must be addressed to scale AI.

To explore edge AI challenges and solutions, SEMI’s technology stewardship platform – the Chief Technology Officer (CTO) Forum – on July 15^th gathered  CTOs from 22 leading microelectronics companies including Accenture, Advanced Micro Devices, Advantest, Applied Materials, Brewer Science, Dell Technologies, Dow/Dupont, EMD Materials, Entegris, Galaxy Semiconductor, GridMatrix, HPE, Imec, Lam Research, Microchip, Qualcomm, Quantum Semiconductor, Resilinc, Soitec, Teradyne, Tokyo Electron, and Veeco. Two distinguished keynote speakers, Dr. Evgeni Gousev from Qualcomm and Prof. Boris Murmann from Stanford University, shared their exciting ideas and joined a thought-provoking discussion with the CTOs. Following are key takeaways from the Forum.

Why edge AI matters

Sustainability: As AI applications proliferate, they require more computing power and consume more energy. Today, AI produces more carbon than the airline industry – this is not sustainable! Increasing edge intelligence enables low-power local data processing to reduce both energy consumption and greenhouse gas emissions.

Latency: One of AI’s exciting promises is the ability to support real-time decision-making. Minimizing latency (delay) is key to realizing this promise – an autonomous car seeing an obstacle or a medical implant sensing an anomaly must decide the next course of action instantly. Edge AI is a critical enabler of this capability since it can provide swift diagnosis and actionable intelligence.

Privacy and Security: As our lives become increasingly dependent on digital devices, not surprisingly, privacy and cyber-security are becoming paramount concerns. Individuals, corporations, and even entire nations have fallen prey to digital errors and malicious hacking. The ability to perform intelligent operations at the edge reduces the vulnerabilities in data transmission, improving both privacy and security.

Connectivity: 5G networks and gigabit ethernet have arrived but are far from widespread. Even in well-connected environments, the limitations of communications are apparent when a video call gets choppy or simply disconnects. These disconnects are much more pronounced in underserved communities and remote areas around the world. Edge AI operates independently of a communication network, and this can make a life-and-death difference – for example, in a remote rural clinic or an environmental conservation operation deep in a forest.

Diversity, Equity, and Inclusion: This is a subtle, often under-appreciated benefit of edge AI – it is inherently democratizing, as it gives control to the problem owner operating in the field.  Not having to depend on potentially expensive cloud services or the presence of communication networks broadens AI access to populations around the world while empowering a larger pool of innovators.

How to make edge AI happen

While the concept of Edge AI is not new, much effort is still required for practical implementation to realize its full potential. It is a huge challenge to pack significant computing and analytical capability in a tiny inexpensive device with a severely limited power budget. Edge AI needs the following enablers to advance at a rapid pace.

System-Level Approach: Both Dr. Gousev and Prof. Murmann emphasized that edge AI development requires holistic thinking, co-optimizing the entire hardware-software-system stack. The hardware must be specifically designed for energy-efficiency and to operate with milliwatt power budgets. This is in contrast to the current approach of using hardware that is just good enough. Researchers are exploring innovative approaches such as analog and in-memory computing. The software infrastructure and tools must mature, and energy-efficient algorithms that are edge-specific and adaptable to neural networks need to be developed. Since these edge devices will be ubiquitous, it is also extremely important to design security in the devices and eliminate ethical biases in algorithms upfront.

Smart Data Management: Since both the compute capability and energy budgets of edge devices are severely limited, efficient data management is essential. Edge AI algorithms and architectures require sparse or compressed data sets. Prof. Murmann provided an example of how to implement this practically – the volume of input data can be reduced by preserving information using gradients of pixels rather than actual pixels. He added that the quality of data used for model training also must be improved – for example, using public images from mobile phones often give the “wrong training” to the system due to faults and occlusions.

Ecosystem Development: Dr. Gousev highlighted the importance of building a robust ecosystem for edge AI. He described the activities of tinyML, the not-for-profit organization that he chairs. tinyML is broadly defined as machine learning architectures, techniques, tools, and approaches capable of performing on-device analytics for a variety of sensing modalities (vision, audio, motion, chemical, etc.) at the milliwatt (or below) power range, targeting predominately battery-operated devices. tinyML is creating a community of interest and building the awareness and ecosystem required for true edge AI devices to proliferate in the future.

Supply-Chain Enablement: Edge AI technologies require full-stack optimization, including supply-chain elements like packaging and materials. For example, AI system costs can be mitigated by using innovative packaging techniques like heterogeneous integration and 2.5D or 3D – in place of expensive system-on-chip (SoC) solutions. The materials industry must keep pace with both the growing demand for the volume and innovation required to enable edge AI. The microelectronics industry also must closely track Environment, Health and Safety (EHS) issues for materials manufacturing and global regulatory changes affecting technology development.

Inspiring and Developing Talent: Ensuring that the microelectronics industry has sufficient talent is perhaps the single biggest worry that keeps leaders up at night. Mitigating the talent shortage needs a multi-faceted approach. The industry must inspire students by focusing on applications for social good and sustainability. Compensation must be competitive and combined with early career internships. The industry must expand its search for talent beyond the usual suspects already in the technology world – it must make a special effort to reach out to underserved communities in high schools and middle schools. Inspiring this vast pool of students with exciting career opportunities and providing them with the needed educational resources and tools could be the foundation of a long-term solution to the talent shortage. For the short term, the industry must invest in training the existing workforce to become savvy in AI data techniques.

Edge AI clearly holds exciting promise of making intelligence available at our fingertips, on demand. Edge AI can work for social good by improving sustainability and diversity, equity, and inclusivity. However, success hinges on several enablers as described here. While companies will continue to innovate for competitive reasons, this alone will not suffice – it will take a village! SEMI offers several collaborative programs in the pre-competitive space that can help, including initiatives in Smart Data-AI, supply-chain enablement, and workforce development that are addressing edge AI challenges and can play an important enabling role. Working together, SEMI and tinyML can support the industry in realizing the promise of edge AI.

Tom Salmon is Vice President of Collaborative Technology Platforms at SEMI.

Pushkar P. Apte, Ph.D., is Strategic Technology Advisor and leads the Smart Data-AI Initiative at SEMI.