Geoff Tate

  (all posts)
Geoff Tate is the founder and CEO of Flex Logix. Tate has more than three decades of experience in technology. He is the former CEO of Rambus, and a current board director at Everspin Technologies. He received his BSc in computer science from the University of Alberta, and an MBA from Harvard Business School.

Author's Latest Posts

Advantages Of BFloat16 For AI Inference
By Geoff Tate - 03 Oct, 2019 - Comments: 0

Essentially all AI training is done with 32-bit floating point. But doing AI inference with 32-bit floating point is expensive, power-hungry and slow. And quantizing models for 8-bit-integer, which is very fast and lowest power, is a major investment of money, scarce resources and time. Now BFloat16 (BF16) offers an attractive balance for many users. BFloat16 offers essentially t... » read more

eFPGA Macros Deliver Higher Speeds from Less Area/Resources
By Geoff Tate - 05 Sep, 2019 - Comments: 0

We work with a lot of customers designing eFPGA into their SoCs.  Most of them have “random logic” RTL, but some customers have large numbers of complex, frequently used blocks. We have found in many cases that we can help the customer achieve higher throughput AND use less silicon area with Soft Macros. Let’s look at an example: 64x64 Multiply-Accumulate (MAC), below: If yo... » read more

AI Inference Memory System Tradeoffs
By Geoff Tate - 01 Aug, 2019 - Comments: 1

When companies describe their AI inference chip they typically give TOPS but don’t talk about their memory system, which is equally important. What is TOPS? It means Trillions or Tera Operations per Second. It is primarily a measure of the maximum achievable throughput but not a measure of actual throughput. Most operations are MACs (multiply/accumulates), so TOPS = (number of MAC units) x... » read more

TOPS, Memory, Throughput And Inference Efficiency
By Geoff Tate - 02 Jul, 2019 - Comments: 3

Dozens of companies have or are developing IP and chips for Neural Network Inference. Almost every AI company gives TOPS but little other information. What is TOPS? It means Trillions or Tera Operations per Second. It is primarily a measure of the maximum achievable throughput but not a measure of actual throughput. Most operations are MACs (multiply/accumulates), so TOPS = (number of MAC... » read more

Do Large Batches Always Improve Neural Network Throughput?
By Geoff Tate - 06 Jun, 2019 - Comments: 1

Common benchmarks like ResNet-50 generally have much higher throughput with large batch sizes than with batch size =1. For example, the Nvidia Tesla T4 has 4x the throughput at batch=32 than when it is processing in batch=1 mode. Of course, larger batch sizes have a tradeoff: latency increases which may be undesirable in real-time applications. Why do larger batches increase throughput... » read more

Neural Network Performance Modeling Software
By Geoff Tate - 08 May, 2019 - Comments: 0

nnMAX Inference IP is nearing design completion. The nnMAX 1K tile will be available this summer for design integration in SoCs, and it can be arrayed to provide whatever inference throughput is desired. The InferX X1 chip will tape out late Q3 this year using 2x2 nnMAX tiles, for 4K MACs, with 8MB SRAM. The nnMAX Compiler is in development in parallel, and the first release is available now... » read more

Multi-Layer Processing Boosts Inference Throughput/Watt
By Geoff Tate - 15 Apr, 2019 - Comments: 0

The focus in discussion of inference throughput is often on the computations required. For example, YOLOv3, a power real time object detection and recognition model, requires 227 BILLION MACs (multiply-accumulates) to process a single 2 Mega Pixel image! This is with the Winograd Transformation; it’s more than 300 Billion without it. And there is a lot of discussion of the large size ... » read more

Inference Acceleration: Follow The Memory
By Geoff Tate - 07 Mar, 2019 - Comments: 0

Much has been written about the computational complexity of inference acceleration: very large matrix multiplies for fully-connected layers and huge numbers of 3x3 convolutions across megapixel images, both of which require many thousands of MACs (multiplier-accumulators) to achieve high throughput for models like ResNet-50 and YOLOv3. The other side of the coin is managing the movement of d... » read more

Use Inference Benchmarks Similar To Your Application
By Geoff Tate - 07 Feb, 2019 - Comments: 0

If an Inference IP supplier or Inference Accelerator Chip supplier offers a benchmark, it is probably ResNet-50. As a result, it might seem logical to use ResNet-50 to compare inference offerings. If you plan to use ResNet-50 it would be; but if your target application model is significantly different from Resnet-50 it could lead you to pick an inference offering that is not best for you. ... » read more

Lies, Damn Lies, And TOPS/Watt
By Geoff Tate - 07 Jan, 2019 - Comments: 1

There are almost a dozen vendors promoting inferencing IP, but none of them gives even a ResNet-50 benchmark. The only information they state typically is TOPS (Tera-Operations/Second) and TOPS/Watt. These two indicators of performance and power efficiency are almost useless by themselves. So what, exactly, does X TOPS really tell you about performance for your application? When a vendor ... » read more

← Older posts Newer posts →

  • Sponsors

About

Navigation

Connect With Us

Copyright ©2013-2024 SMG   |  Terms of Service  |  Privacy Policy