What’s Missing In Deep Learning?

It is impossible today to be unaware of deep learning/machine learning/neural networks — even if what it all entails is not even clear yet.

Someone who is intimately familiar with this area, and has some thoughts on this is Chris Rowen, founder of Tensilica (now part of Cadence), who is now a self-described hat juggler. He is still active Cadence several days a month, working technically on new architectures, working with selected customers in key strategic relationships, and providing strategic advice to some major initiatives at a corporate level.

The next hat he wears is that of Cognite Ventures, that he established as a vehicle to focus his work in early stage investing through seed class investing in startups in the deep learning area.

We had a chance to speak this week by phone, and he said he has a number of ideas about some of the major holes in terms of what’s being offered in this space. He is considering developing some technical ideas that may end up being spun out, worked on or funded in some fashion. “I’m taking a more proactive approach, not just waiting for things to come to me but creating the things that I think are probably going to be the most interesting technically there.”

Rowen’s third hat is strategic advice to Stanford’s SystemX Alliance, where in April he will conduct a few workshops on advanced computing architectures, and next generation design productivity.

His passion for deep learning is infectious.

“Certainly there is this tremendous potential and tremendous enthusiasm around deep learning methods. It is nothing less than a revolution in thinking about how to do computing, so it is, at a minimum, the shiny new hammer that everyone has in hand. They look around for nails to hit with it, but they are also banging away on screws and everything else so that it is this technique or this philosophy, which appears to have very broad potential, but nobody really knows what it will ultimately look like or which problems are ultimately a good fit or a bad fit. But just hanging out at a university, the number of students, and the number of researchers that are plunging into deep learning and neural network is really staggering. You show up at a graduate course with an obscure title about natural language interprettion or vision, and you will find hundreds of students that are signed up for these courses because they see this as, A) the most interesting; and B) the hottest area for increasing their economic value,” he said.

Rowen pointed out that, interestingly, at Stanford computer science has become the number one major, and guesses that neural networks is the number one topic in computer science. “It is all the buzz, and I think it’s not very different in the major technical universities around the world. You have this intellectual curiosity, and you have this entreprenuerial spirit, which is so well honed around the world, but especially in American universities, and most especially in places like Stanford that immediately connects some breakthrough idea with people who are saying, ‘Let’s start a company to exploit it.’ It’s no big surprise that we’ll see hundreds of companies spring up over the course of a few years to try and take advantage of it.”

He explained he’s just been trying to get his arms around it, but said it’s clear that it’s quite hard to do because there are so many companies (http://www.cogniteventures.com/the-cognitive-computing-startup-list/) , and because there’s so much hype around it. “Everybody puts, ‘AI,’ or ‘machine learning,’ or some other phrase in their description of what they are doing even if they’re using it only in a small way. Part of the task is to sort through and figure out which are deeply exploiting it, and for whom it is a strategic element of their technology portfolio, or what they are doing is likely to have lots and lots of interactions. For example, people in robotics are likely to be big users of it so I’ve been pretty generous in including robotics-related companies in my list, whereas there are lots of people doing business intelligence, and predictive marketing, and customer relationship management, which may have some data mining piece to it but which I’m just a little bit less generous in assuming they are serious deep learning companies. I’ve applied this filter of which are likely to be the ones for whom deep learning, neural networks are really critical to their success.”

…and there are still 190 of these companies…

There’s no question that there is a flood of activity, and of course there will be some winners and some losers, Rowen continued. “Companies will evolve as they figure out what they are doing but the number of companies working on it, the number of people working on it, the rate of progress in terms of people coming up with genuinely clever ways to apply these learning based algorithms I think will be very substantial.”

Some of them are even moving along pretty well. “The lowest barrier to entry is probably people who are deploying a cloud-based service for some kind of recognition activity where you can go to one of these recognition-as-a-service sites, and they provide an API so that you can call the service from within your own application, and the service will return information on what’s in the picture, what’s the sentiment of the people in the images, what are some standard characteristics of obvious people or objects in these image streams. That’s something where the amount of effort to deploy one of these services is pretty moderate that people know how to take advantage of them, and where there’s kind of an existing pay-as-you-go business model that people are generally comfortable with thanks to Amazon web services, and the like. There, there’s a big variety of things, and people are doing it,” he observed.

There’s also a good bit of activity around things like monitoring and surveillance where someone can buy smarter and smarter cameras that provide additional information, Rowen reminded. “There’s a very rich set of things in terms of identifying patterns in documents or text where again, via services or installable applications, lots of specialiation — ones for looking automatically at contracts or ones for looking at social media interactions or looking at datasheets and product specifications, or customer service dialogue. All these different specialized forms of text-based interaction where you can extract out information and compare it to standard patterns and use that as a mechanism within some larger application.”

Medical is another area of particular focus for deep learning development.

Still, he admitted a particular fondness for what’s going on in embedded systems, as those will be the things that touch us directly, and which go into the real time interactions that we want to have. “Sure, there are these quasi-real time things like Siri and Alexa, which are using neural networks in a big way but which are not really quite real-time. You pose a question and get an answer back some seconds later, where, if you’re driving a car or you’re interacting with a home device (a television or refrigerator or cell phone), you’re going to want something that is even quickler, and has an even better understanding of context, and who you are. I think there is a lot of progress to be made there because it will change a lot of how human-machine interactions really feel.”

Related Stories
Plugging Holes In Machine Learning (Part 2)
Short- and long-term solutions to make sure machines behave as expected.
Neural Net Computing Explodes
Deep-pocket companies begin customizing this approach for specific applications—and spend huge amounts of money to acquire startups.
Convolutional Neural Networks Power Ahead
Adoption of this machine learning approach grows for image recognition; other applications require power and performance improvements.