New Design For Trusted Data

Recently, I wrote about Fully Homomorphic Encryption (FHE from now on) which I think is going to be something big that you will hear lots about in the future.

Here’s the reason I think it is going to be big. The people who care the most about security, such as financial institutions, governments, and companies with huge amounts of valuable data (such as semiconductor companies, or social media companies) can’t just rely on network and data center security.

Lisa Porter is the U.S. Deputy Under Secretary of Defense for Research and Engineering. Here’s a quote from her late last year:

No matter how secure a computer network or the environment it’s used in may seem to be, users should just assume it’s compromised.
…
There is no such thing as a secure system. Users should assume the network is compromised already—either from outside or from bad actors on the inside.

In fact, that statement is so central to her thinking, that the piece I pulled the quote from is simply titled Assume Networks Are Compromised, DOD Official Urges.

Cornami

I talked to Wally Rhines, who is CEO of Cornami. He told me that he had been looking into whether the industry was doing anything with fully homomorphic encryption for the DoD. But wherever he went, people said it was 10 or 20 years away, requiring a million times the best Xeon or NVIDIA performance, so he reported back that he didn’t think it was going to happen, and the program was funded without any conflict with industry. Wally then ran into Gordon Campbell. He was the founder and CEO of Chips and Technology, which was the first fabless semiconductor company. You can read more about that bit of semiconductor history in chapter 4 of Dan Nenni’s and my book Fabless: The Transformation of the Semiconductor Industry (link is to a free pdf). He had started a software company, which is now a hybrid software/hardware company called Cornami (“a tsunami of cores”). I won’t give the whole history, but he told Wally that the one thing that they could do that was revolutionary was FHE. Wally said, “I’ve heard of it and I don’t believe you can do it.” But they already had an FPGA-based emulator of their chip up for 18 months. So Wally spent several months assessing the technology and decided it was real.

“This is revolutionary,” Wally said.

“Why don’t you join us?” was the reply.

So he did. His timing wasn’t great since he started in early March just as everyone went to work-from-home and venture capitalists stopped investing in anything new until they realized how much they would need for follow-on money for their existing investments. Plus VCs invested in way too many AI chips. As a VC said to me once, “VCs make sheep look like independent thinkers”.

Cornami is somewhat in stealth mode so Wally said he’d have to be a bit evasive. But some things have been disclosed. They presented at last fall’s Linley Conference, which I attended. Most of their presentation was about general AI and they didn’t mention FHE, and they were coy about the internals of their chip. But they did show some impressive performance on ResNet-50, namely 17 times faster than NVIDIA’s then-fastest GPU at 1/7th of the power, and 1/70 the latency.

Linley analyst Mike Demler’s conclusions were:

Since it started in 2014, Cornami has disclosed few details of its plan to build a configurable AI processor, but at the recent Linley Fall Processor Conference, it presented performance results based on an FPGA prototype. Running ResNet-50 inference on 224×224 images with half-precision (FP16) floating-point operations, the development system predicts an incredible 105,000 images per second for the initial product, which it expects to require only 30W. Although that result comes from cycle-accurate measurements rather than an actual chip, it’s equivalent to 16X the performance of NVIDIA’s V100 GPU. Cornami based its estimate on single-batch inference, making its results even more impressive because the V100 had a more favorable batch size of 128.

Although the company withheld most of the architectural details, the new processor will integrate an array of programmable cores connected by a proprietary network-on-a-chip (NoC), which allows dynamic reconfiguration to meet the compute requirements of individual neural-network layers. Cornami doesn’t describe its product as an FPGA, but we expect it’s applying some of the programmable logic technology the founders previously developed at Quicksilver Technology more than 15 years ago.

The design is driven by software innovation. Fred Furtek developed a unique programming model that is massively scalable solving current inefficiencies with multicore chips with fully independent streams. FHE (and machine learning) have vast libraries such as Microsoft Seal and HLIB from IBM (and PyTorch etc. for machine learning). So everything doesn’t have to operate at the C source code level. “That’s how you get a million times the performance in a chip that is only in TSMC 16nm, nothing special.”

The design manager is Syed Ahmed, who is ex-Cadence. I asked about the tool flow. The physical design and formal is all Cadence. Palladium? “We can’t afford an emulator so we had to roll-our-own FPGA for all the testing.” Wally admitted it was a challenge since when debugging chip logic an FPGA is really hard compared to an emulator.

The design is approaching tapeout. Meanwhile, Wally and Gordy are out knocking on doors looking for a lead investor.

We’re in a race with the verification team to see if they can complete the verification before we raise the money we need to pay for the tapeout.

The design scales linearly, so it is easy to get more performance just by adding another chip. “It’s like memory, it’s easy to add more.” The way to think about it is as a re-configurable systolic array. A Cornami chip is actually a configurable computational fabric made up of thousands of FracTLcores that can be scaled to millions across a system. These can be configured in various ways including dynamically creating a RISC processor for recursion, and other things that you can’t do in their FracTLcore environment, where you need a von Neumann processor. But it is dynamically reconfigurable under software control so you always have the right mix. Cornami actually introduced the software programming model, called TruStream first, but quickly decided to put it in hardware to maximize its value and achieve true linear performance with core count.

The market

Another thing Wally told me is that it is amazing how many companies have their own groups looking into homomorphic encryption (HME). Every finance group has something going already. Microsoft is introducing some HME for voting, since simply adding up election votes works re­­­­­­­asonably well on regular computers. If “data is the new oil” then FHE is fracking or something…the thing that unlocks the value of data. It lets you control exactly who can do what with your data in a way that they cannot get around, by never decrypting or exposing the underlying personal or confidential data even if computers, networks, and personnel are not to be trusted. So the potential is to solve two holy grail problems. First, take a lot of basic computer power and assemble it in a way that delivers limitless computer performance. Second, to take a lot of insecure technology and deliver limitless security and privacy on all the underlying data, whether it be personal data or other confidential data, by never decrypting the data.

Of course, I don’t know enough about the design to be certain everything is going to be great. After all, I thought the Wave Computing design was impressive a few years ago and they ended up going out of business (see my post Wave Computing: a Dataflow Processor for Deep Learning). The jury is still out on Cerebras (see my post HOT CHIPS: The Biggest Chip in the World, or see their keynote presentation almost a year later at DAC next month). Any number of the chips that show up at HOT CHIPS each summer look too good to be true and sometimes are. So time will reveal more. But Cornami has potential customers wanting to evaluate and adopt the technology: banks, medical, military…all with a reputation for moving slowly, but now wanting to go fast.