A cautionary tale of an early attempt to put AI on an IoT system that detects when elderly people fall.
Who knew falling was so complicated? “I don’t want to work on a fall detection system ever again,” said MbientLab CEO Laura Kassovic in front of an ARMTech Con audience. The audience laughed as she flashed a picture of what could now be a better approach—the Apple Watch Series 4, which had recently been announced.
“The Apple 4 has the fall detection sensor built in. It’s probably a good system,” she said. “But I still need to evaluate it.”
San Francisco-based startup MbientLab makes motion detecting sensors for IoT wearables, usually healthcare and fitness applications. “We’re primarily making sensors for medical companies like Smith and Nephew,” said Kassovic. Their sensors have been used in clinical trials and research, and the data has been submitted to the FDA. “The data we provide from our sensors is good data.”
Learning to fall
A few years ago, MbientLab helped a client design a fall-detection system for nursing homes. The system would consist of the patient-worn devices and hubs placed around the facilities. It had to detect when the person wearing it had fallen and signal someone to come help. The wrist-worn sensor had to be lightweight, comfortable, waterproof (for showers) and attractive enough not to be discarded by patients. All this on a seven-day battery, USB rechargeable.
The client wanted an accelerometer, gyroscope and pressure sensor. MbientLab selected the Bosch BMI 160 (16-bit inertial measurement unit) and the BMP 280 (absolute barometric pressure sensor) and initially used an Arm Cortex M0 (later an M4) with a low-power BLE transceiver from Nordic. The system was built on Intel Edison software. “Back then Edison was all the rage,” said Kassovic. “So, we had to build on the unfortunate Intel Edison, which has now been canned. And then we used Javascript libraries because back then they had the best BLE support.” 
Fig. 1: The fall detection system for nursing homes, designed by MbientLab. (MbientLab/Arm TechCon)
Machine learning
The system had to be trained to recognize from incoming data what was a fall. MbientLab chose to train a recurrent neural network (RNN) and specifically an LTSM (long short-term memory), which is less computationally intensive. “We used a recurrent neural network because it has memory. That means that whenever you train it, it has memory in the sense that current inputs are affected by previous inputs, which is normal. Because If you’re doing a speech recognition algorithm, the previous syllable affects the next syllable. Together they create a word, so you need to have a memory to know what the full word is.”
Teaching the system when a person had fallen was the hard part. “An elderly person doesn’t do what they do in cartoons—slipping on the banana peel. That’s not how they fall,” said Kassovic. “They typically tend to do things like roll out of bed, or just fall really slowly at a kitchen counter because they lose their balance. It’s a very slow fall… that made it very difficult because it wasn’t just one fall, it was maybe five or six different types of falls. We had to get data for each type.”
MbientLab’s client decided to hire a separate data science firm to gather the data to train the system to allow Kassovic’s group to concentrate on the system design and manufacturing. The data science firm’s deliverable would be training data in C code.
“We couldn’t push grandma down the stairs,” said Kassovic, so instead, the client’s data science firm used stunts people to fall like elderly people while wearing the sensors and streaming the collected data up to the cloud via iPhone. This data would be used to train the system.
This is when things started to go wrong. “Never hire a data scientist to do a firmware engineer’s job,” said Kassovic. Embedded systems 101 says think small: your system is constrained. These data scientists, however, were used to crunching numbers on “fancy systems” with a lot of power and memory. The data team used Python to clean the data and then wrote a C program for the inference algorithm that would detect fall incidents. The algorithm they provided Kassovic’s team was huge. It took too long to make an inference and was too power hungry for the wearable system.
“Basically, the first time they ever gave us the code, it didn’t even compile,” said Kassovic.
The next mistakes the data scientists made were basic, but important. “When they trained it on their like 14-billion-core machine, they had the good old floating point that we do not have on the M0. So, they train their entire net so the algorithm that came out of it was floating point, had floating point or used floating point. Couldn’t do that,” said Kassovic. “We had to yell at them. They basically used all of our RAM.” The system crashed all the time, and whenever new data came in, it took over 10ms for the algorithm to run.
Using all the sensors was also a problem. “The biggest problem too is that because it used all the sensors, algorithm would kill the battery on our device in about a day, which wasn’t ideal. They need to just get rid of a few nodes, because this thing is like the biggest algorithm known to man …it was ridiculous,” said Kassovic.
The device also would not work anywhere above sea level. The data team—also based in San Francisco, Calif—at sea level— ran their training data in Pascals. The system would work in San Francisco but not in Denver. MbientLab told the data science team to use relative altitude in meters to train their net and not absolute Pascals, “because that was the silliest thing I’ve ever heard,” said Kassovic.
“We ended up telling them how much RAM and flash they had and that they couldn’t go over, what the runtime had to be. And so, every time we’d retrain the net and they would give us some more code, we forced them to tell us everything about it, like is it going to fit? Is it better? Is the accuracy better than before? And then we would say okay, we’re going to go ahead and release this as a product. This will be the new firmware that goes out to those nursing homes.” The detector is now used in about seven rest homes under a subscription model.
Fig. 2. MbientLab CEO at Arm TechCon in 2018 shows how they gathered data about falls to train an RNN. (Semiengineering.com)
This project happened three years ago. Now tools are much better. “I would say definitely leverage the ARM tools because basically they give you tools to train the net externally and then all you have to do it internally as a firmware level is feed it to coefficients and basically done, which is kind of nice,” said Kassovic. “If you’re not going to use the Apple 4 Watch, use the tools available to you, which makes it much easier.”
Related Stories
