AI: A New Tool For Hackers, And For Preventing Attacks

Semiconductor Engineering sat down to discuss hardware security challenges, including new threat models from AI-based attacks, with Nicole Fern, principal security analyst at Keysight; Serge Leef, AI-For-Silicon strategist at Microsoft; Scott Best, senior director for silicon security products at Rambus; Lee Harrison, director of Tessent Automotive IC Solutions at Siemens EDA; Mohit Arora, senior director for architecture at Synaptics; Mike Borza, principal security technologist and scientist at Synopsys; and Mark Tehranipoor, distinguished professor in the ECE Department at the University of Florida, and co-founder of Caspia Technologies. What follows are excerpts of that discussion.

L-R: Caspia’s Tehranipoor; Rambus’ Best; Synopsys’ Borza; Synaptics’ Arora, Keysight’s Fern; Microsoft’s Leef; Siemens EDA’s Harrison.

SE: With so much focus on all things AI, where do you see new threat models emerging?

Fern: There are a lot of AI-specific attacks coming out, like data poisoning, [which is] adversarial input for things like large language models that are trying to jail-break out of the protections. If you ask it, “How do you build a bomb?” it’s not supposed to tell you certain information. Also, an important threat to AI systems is the confidentiality of the training data. That’s something that might not be a threat for other types of systems, but honestly, it comes down to the security basics of confidentiality, integrity and availability, and pretty much all the threats to AI systems can fall into those buckets. And while there are attack techniques that are specifically designed for AI, machine learning, deep learning systems, there are also the traditional techniques for attacking — for example, a physical attack on an embedded device. If you have your entire network with all the weights that might be proprietary, and someone might be able to use a flash extraction attack or read all the data directly from the flash chip, then they have your entire network. That’s an example of a traditional attack just being applied in the machine learning or AI context.

Harrison: The key elements of AI and attacks come down to two things. One is the attacks on the AI data that’s used for the training. We’re starting to see a number of initiatives now. If you think about AI a few years ago, people would like to get their hands on as much training data as possible, and didn’t really care where it came from. It was, “If it’s training data, it’s good data.” But now there’s a lot of movement in terms of how do you certify that data? There are organizations being put in place to make sure the data you’re using for training is trusted AI data. That’s key. But then you see also the physical attacks. It doesn’t take a huge amount of interference on an AI network to push the AI model in a certain direction, and you see a lot of people pushing this and attacking, and seeing in what direction they can get the AI hardware to misrepresent the information. We’ve moved from just pure hardware attacks to this new era of attacking the training data, as well, and the impact is increasing.

Best: There are two answers to this question. One is that with the emergence of AI systems, as you were describing, obfuscation as a security approach has been mostly dead for a very long time. Obfuscation is no longer a security approach, and whether you’re trying to hide something in camouflage or conceal it in code or in any other way, once AI is written, that’s done. But it’s not just the training data that’s at risk. It’s also the edge data. When the model is distributed into a system, ensuring the privacy of the model, the authenticity, and the freshness of that model is critical. There are security threats with data at rest, and data in use, when the data is going back and forth to HBM, or if it’s going off chip to DRAM. There are also side-channel leakages when that data is being used inside an AI core. It is leaking information about the computation of the use of that model. A Barracuda paper published a couple years ago showed a power analysis attack to extract a model in use out of an NPU system. So all of the standard approaches I worry about — data at rest, data in use, and the very high value of these models as they’re sitting there in edge-based deployment systems. It took tens of millions of dollars, in some cases, to synthesize a model, and if an adversary can recover it, they get to execute it in their NPU without spending tens of millions of dollars. So there’s a lot of value in security needing to protect those models.

Tehranipoor: Where we are with security? We’re in much better shape than we’ve ever been with regard to the industry, the companies, and the attention of customers to security issues. We no longer have to work very hard to convince folks. A lot of companies we work with at the C-suite level are very much informed of the security issue. So that’s good news. The bad news, obviously, is always cost, cost, cost, and they want to make sure that easy automation is available to them. With regard to AI, it’s a double-edged sword. For example, a couple of years ago we asked students to inject vulnerabilities into some designs. Back then it was manual, and each vulnerability easily would have taken a week or two. We wanted to do this on many designs. After several months we were able to inject vulnerabilities into a few designs, and at some point, in parallel, we were working with LLMs to see if LLMs could do it, and we figured out in a matter of hours how to inject vulnerabilities into 10,000 designs. The point is, it’s really powerful if you find a way to use it well. But at the same time, an adversary can use this in an offensive mode. GenAI can understand the limits of what’s detected today, and it can use this to be able to counter that. This means it’s really important that we understand how we could take advantage of the opportunities that it gives, but at the same time be mindful of the challenges. Additionally, with the number of agents that we’re going to be developing under GenAI to work together, if we don’t design it well with some rules and operation understanding, problems are going to come soon.

Arora: I second that. Adversaries can use LLMs to attack the binary and automatically suggest how they should be attacked. Side-channel attacks, like Scott mentioned, are pretty popular in terms of how you can use AI to reduce the development cycle. That would be an offensive use of AI. What you could do is vulnerability chaining, and that’s a key trend coming up now, which is where AI is used to look at all the deformities or limitations of a product and then chain it up. We are seeing time and time again that it reduces the attack cycle. I remember way back, working on PCI payment cards, there were standards that specified a new requirement for 10 hours of side channel protection. But that was in 2010. At the time they thought 10 hours is good enough to prevent attacks at a system level. But now with AI, the attack cycle actually shrinks, so you can do more value-added attacks. The same thing goes for Platform Security Architecture (PSA) Level 3, Level 4, with Arm. They typically take 35 days to see how much they can attack. But if you cannot do it much quicker, you can do much higher-value attacks later. That’s one of the key trends we’re seeing. At Synaptics we’re trying to run AI as part of the secure video path directly for low latency, but we have to be very, very careful, because that’s a trusted boundary, and if something has leaked, you’re getting violation of all the test boundaries.

Borza: The real situation with AI is that it massively increases the size of the attack boundary, and we haven’t seen an explosion in the attack boundary like that, with the possible exception of cloud computing. If you can get into a data center, there’s a lot of damage you can do by being able to roam from place to place, or sit adjacent to somebody else’s compute loads and reverse engineer those from inside the cloud. GenAI presents that opportunity, but it’s distributed. And not just GenAI, but AI systems, in general. There are edge devices that carry around with them a lot of neural models, whether they’re large language models or not, and those are coming under attack through side channels, but also through direct manipulation. People are trying to get into them and steal the model and manipulate the model. If you can change the model at the hardware level, you can do the same kinds of things that you could do if you were poisoning the training data. You’re changing the way the model behaves. This is an increase in the size of the attack surface, and it’s very difficult to deal with. There also are some specific attacks that are peculiar to AI systems — a lot of the side channel attacks where you use the AI as an oracle and get it to spill the beans on what it was trained on, how it was trained, what information you’re not supposed to be able to get out of the model. All of that’s there, and you have the opportunity to do that either through clever prompting, or through getting access to the hardware itself and the underlying data directly from the hardware. So the attack surface is ginormous. There’s some hope for using AI to do some of the red teaming. But if you’re doing red teaming with AI, that means, by definition, your adversaries are using it against you, as well. And if you’re not doing that, your adversaries will still be doing that anyway.

Leef: I wanted to comment on the notion of data set contamination. What we’re seeing more and more is that the fine tuning is happening to a lesser extent. People are adding value outside of the LLM — in other words, font augmentation with graph retrieval-augmented generation (RAG), and so on. What that means is the valued ID, or knowledge assets, are shifting back to the premises. They’re sitting in a local place. They’re not going back into the LLM, so there’s no leakage of secrets going through this unidirectional membrane of IP. To me, the biggest issue with security as a business has been poor economics. You have to explain to people why they have a problem before you can sell them a solution. It’s akin to selling vitamins, as opposed to life-saving drugs for life-threatening maladies. The objections to security sales pitch are, ‘We don’t have any security experts here, and nothing has ever happened.’ This is where AI could come into the equation and play an interesting role. I looked at this when I was at DARPA. There were several classes of customers that were interested in security. There were the massive merchant semiconductor companies that frequently had very large departments of people who did bespoke security for every one of their chips. Then, on the opposite extreme were edge IoT startups, which really cared about getting to market as quickly as possible and building eye candy for their smartphone. That leaves two parties in the middle. First are the large application-focused semiconductor vendors that focus on particular domains. They don’t really have a lot of security experts inside, so if you talk to the economic value decision-maker and try to sell security, this person is going to say, ‘I don’t have people who understand this. Finding them is difficult. They’re expensive.’ So it’s a non-starter. Second are the mil/aero constituency that view this as more of an art than a science, and they have several people hidden in the closet somewhere that they whip out for special government programs. To me, there is a barrier to entry for security, making a bridge from security to economic decision-maker inside the customer companies, and the one thing that can help here is AI. Historically, the EDA industry has sold a tool when combined with a human. Therefore, economic value was delivered. But you still needed a human who knows what to do with the tool, and the obvious attack vector here is, “Why do we need a human? What if it’s an agent?” So now, if you bundle a tool and an agent, all of a sudden you’re delivering on a silver platter the solution to the decision-maker that does not necessitate acquisition of highly differentiated, expensive specialists. If you truly get it to the point where, for some number of dollars, somebody who’s not an expert in a domain can push a button and get objective improvement in reduction of risk of being sued in life-endangering situations somewhere down the line, if all they have to do is pay $500,000 or $1 million to take this off the table, that actually makes sense. The thing that has been preventing adoption of this stuff is the necessity of having experts.