AI Moves Out Of The Cloud And Onto The Edge

The impact of AI to date, in the cloud, is undisputed, but the question we must answer going forward is whether we can only expect more of the same or whether there is a fundamental shift looming that will change everything. Today, we will explore historical data to find patterns repeated through the ages to help us see what I will attempt to prove is imminent.

A brief history of time… keeping in Japan

But first, a story! I was inspired by a recent trip to Japan to explore a small part of the rich culture and history of this ancient land, and that part was the keeping of time. Because in Japan the keeping of time is both ancient and almost mystical. Let’s take a look.

The earliest official timekeeping in Japan began in 671 AD, when Emperor Tenchi introduced a water clock (rōkoku). A Japanese water clock tells time using a continuous flow of water to raise a marked float and pointer, or by the dripping of water out of a container to measure the passage of time. Time was also measured using natural phenomena, such as the burning of incense sticks, with different scents or metal balls on a thread used to mark specific intervals or serve as alarms. These methods were used to regulate court and temple life.

Specifically, Japanese society operated on a “seasonal time system” (futei jiho), which consisted of a day and night based on dawn and dusk and further divided each of those into 6 periods, each of which was called an Ittoki (One Toki). Since the length of an Ittoki in the day or night varied by day and even by season, it was a very complicated time system in which an Ittoki always changed. During this period, time was publicly announced across cities and towns via large bells located at temples and castles, linked by a system to ensure synchronization, and this system spread across the country and endured for centuries.

In the 16th century, Jesuit missionaries introduced mechanical clocks from Europe. But there was a problem – western clocks were designed to measure 24 identical hours, while the Japanese system has 2 sets of 6 varying Ittoki. So what happened? Well, the Japanese clockmakers (tokeishi) ingeniously adapted Western mechanical clocks to this complex system, creating unique timepieces (wadokei)!

This all changed with the modernization of the Meiji Restoration in 1873 and the adoption of the Gregorian calendar and western timekeeping. And here’s where things get really interesting – this growth in adoption of western clocks led a young 21-year-old named Kintaro Hattori to open a clock and watch shop in 1881. 11 years later, he would buy an abandoned factory and establish Seikosha or “House of Exquisite Workmanship”. Some seventy-seven years later, in 1969, that same Seiko introduced the Seiko Astron, the world’s first quartz watch, which completely revolutionized the watch industry; the repercussions are still being felt to this day!

Cool story bro

Fascinating as that story may be, you may be asking yourself what it has to do with AI. Well, if we take this as a story about the democratization of a technology, we may be able to look at other technologies to see if they follow a similar pattern. And in fact, it’s very easy to find examples of technologies moving from centralized control to broad adoption. Here are a few:

• Power Distribution – Initially fueled by centralized power plants, energy distribution now includes microgrids, solar panels, and even EVs that provide power at night to individual houses
• Media & Content Distribution – What was once under tyrannical control of movie studios and record companies in the 1940s and 50s is now changed forever with streaming services, content on demand, and the ability for anyone to fire off a podcast and launch it worldwide the same day (perhaps I should launch my ‘History of Seikosha’ podcast!)
• Processing – Of course, we’re all familiar with early computers that filled entire rooms, and the unstoppable march to reduce the weight & volume and increase the access of computing from the first PCs to the first transportable PCs (my first luggable Compaq PC weighed a whopping 36 pounds!) to tablets and smartphones of today

I see patterns!

Fine, so technology sometimes gets broadly deployed. So what? Well, the next question to be answered is whether we can identify patterns in these technology shifts, because if so, we can apply them to AI to see if the same patterns are in flight. And as it turns out there are indeed patterns. The examples above, and many others, all follow this same basic pattern:

• Initial centralization for economies of scale
  • Things are expensive or rare or complicated so are centrally managed
• Growing frustration with limitations
  • This predictably leads to frustrations in terms of access, or use case, or other compromises
• Enabling technology breakthroughs
  • Despite the frustrations, we are stuck in the frustrated state until there is some kind of breakthrough. I was stuck on a 10 hour flight back from Tokyo because the transporter has not yet been invented; until there is an enabling technology, things are forced to remain in a compromised state
• Early adopters followed by normalization of access
  • The breakthrough will likely have rough edges, but the early adopters will smooth those out and make the technology absorbable by the masses
• Tipping point followed by ultimate market transformation
  • And then inevitably there comes a tipping point where the world just changes, and moves on

Hypothesis testing time

OK, so we have a pattern. Let’s test our pattern by applying it to our timekeeping story and see if it holds, because if it does, then we have a proven pattern we can apply to AI. Here we go:

• Initial centralization for economies of scale
  • The use of centralized castle or temple bells satisfies this aspect
• Growing frustration with limitations
  • The futei jiho was incredibly complex, and error prone, and yet brought with it severe penalties for those who incorrectly served the time. Plus the need to rely on the audio range of bells meant a network of bells at earshot range was required (clever though this early mesh system was, it was certainly a logistical challenge)
• Enabling technology breakthroughs
  • Enter the western metal mechanical clock to replace the organic timekeepers of yesteryear
• Early adopters followed by normalization of access
  • We’ve already shown how the wadokei were developed to bridge the disparate timekeeping worlds. And after the Meiji Restoration, the local clock-making industry took off. By 1936 Seiko were producing 2 million watches and clocks a year
• Tipping point followed by ultimate market transformation
  • The introduction of the world’s first commercially produced Quartz watch in 1969 leads to what is now known in the horology world as the “Quartz Crisis,” which totally upended the Swiss watch industry. How disrupted were they? Swiss exports collapsed by 50%, they lost their long-held worldwide lead, and there was a 2/3^rd reduction in both watch companies and employees that resulted from this upheaval

The “Quartz Crisis” – Upheaval in the Swiss watch industry.

Pattern matching the path of AI

And so here we are, with a specific historical example that is just one of many technology transformations from which we have found repeatable patterns that we fact-checked against our timekeeping example. We now have a migration pattern that we can apply to what’s happening in AI today, and if the pattern holds, then it would be illogical to assume that AI will not follow the same pattern. It’s time for the final pattern check!

• Initial centralization for economies of scale
  • The pioneering AI programming work was all done on x86 servers, and now clearly has migrated to GPUs. So. Many. GPUs. Like contributing to global warming masses of GPUs in massive data centers that come with their own nuclear reactors to keep them humming masses of GPUs
• Growing frustration with limitations
  • Privacy & Security – regulations like GDPR in Europe and HIPAA in the US limit sensitive data traveling across networks that can introduce multiple points of vulnerability. Many people aren’t comfortable, or in some cases are completely unwilling, to allow their data to leave their control and go on a trip around the world
  • Latency – The travel time from the edge (where the human sits typing, or the car is driving, or the robot is building) to the cloud and back after being processed can take seconds or even minutes. When a bunny hops in front of your car while it is doing 70mph on the highway, there are only milliseconds to respond
  • Bandwidth – High data transmission expenses and network congestion at scale can wreak havoc on reliability and budgets
  • Connectivity Dependence – Anyone flown on a plane lately and marveled at the stability of the internet connection? Yeah, me neither. Any system that fails in remote areas or during outages is inherently unstable and prone to total system failures
• Enabling technology breakthroughs
  • Given the enormous amounts of money at stake, folks have developed AI-specific architectures like NPUs, TPUs, and other forms of AI accelerators to increase the amount of compute horsepower or efficiency. We’ve also (finally) started to see efforts to reduce the workload itself, like quantization schemes, pruning and compression techniques, knowledge distillation from larger to smaller models, and other approaches to get to more efficient algorithms. Finally, the development of AI Edge specific IP allows for integration, leading to systems that were specifically designed for low-power edge environments, right-sized for the task at hand, and with reduced BOM costs
• Early adopters followed by normalization of access
  • The IoT explosion leading to billions of connected devices is clearly present, as is the unquenchable demand for AD/ADAS systems in automotive. Beyond that we are seeing market & segment specific products for the industrial, healthcare, smart city, and other sectors
• Tipping point followed by ultimate market transformation
  • Analysts are now looking forward, and not having to look too far, to see the tipping point they expect to hit, not “sometime in the not too distant future”, but by 2026-27. The CAGR for Edge AI devices is forecast to be between 21% and 37% and the market is forecast to be >$140B by 2034. The inevitable is here: Edge AI WILL be everywhere. Don’t believe me? Try this simple test – type the title of this piece “AI Moves out of the Cloud and Onto the Edge” into your favorite search engine. You’ll be besieged with analysts, thought leaders, evangelists, LinkedIn posts, vendor blogs, and more all saying the same thing. Like the little girl in front of the TV in the movie Poltergeist said, “They’re heeeere…”

Told you so

There is a saying that history doesn’t repeat, but it does rhyme. Our previous examples therefore are as close as one can get to proof that AI is moving, and moving fast, to the edge, and we need to do something about it. What is needed is technology designed with DNA specifically for the hostile environment of Edge AI device needs, like low power, high efficiency, incredible resiliency in the face of churning model architectures, freedom from vendor SW limitations, and more. Fortunately for you, Quadric offers just that:

• A processor architecture built from the ground up to handle the demands of Edge AI devices, not cobbled together blocks of legacy DSPs and CPUs kludged together with brittle blocks of HW accelerators
• An architecture that is aware of and was designed to overcome the devastating impact of having to move data from HW-centric MAC heavy operations to flexible compute and back, many, many times during the processing of a model. The exact same laws of physics challenges (mainly latency and power) we saw introduced by moving data from the edge to the cloud and back apply, but on a smaller scale, in an SoC. To meet customer demands, the latency associated with this data movement, and the power consumed, must be avoided. The Quadric Chimera architecture keeps the data in local shared memories as much as is physically possible, allowing MACs and customer operator code to access the same data on successive clock cycles, with no data movement required
• An architecture that is future proof, because this is the intractable problem in the Edge AI world. When silicon takes years to develop, and AI models churn in months, by definition your chip will be out of date before it exits the fab (probably even before you tapeout). That is, unless you have an architecture that allows for the addition of custom operators in an easy to program manner like C++ that runs at full tilt, guaranteeing that any future operators you need 5 years from now can be added to silicon you are shipping today and will run as efficiently as if they had been designed in from the start

HW can NOT keep up with AI model changes.

• An architecture that puts you in control of what models get prioritized, and how they are developed. Gone are the days when you had to go hat in hand to your accelerator vendor begging for them to add the models or custom operators you need, because their architecture is simply too hard for mere mortals to program or schedule efficiently. With Quadric’s approach, if a model isn’t ported, you can do it yourself, and add any custom operators you need directly! And even if your competition are using Quadric (and why wouldn’t they be?), you are still free to differentiate with the custom operators that you add and own

Have I made my case that technology democratization can and does happen? That patterns can be seen? That they are verifiable against multiple examples, even timekeeping from over a thousand years ago? That when applied to AI, they prove the move to the Edge is happening, now, and will change everything? And that Quadric is the best technology for driving that shift forward? If you don’t think so prove me wrong! Call us (1-844-GPNPU00) or visit quadric.io – we’d love to talk about your challenges and how we can help.