Ways to adapt the Transformer architecture to keyword spotting and an introduction to the Keyword Transformer (KWT).
The Transformer architecture has been successful across many domains, including natural language processing, computer vision and speech recognition. In keyword spotting, self-attention has primarily been used on top of convolutional or recurrent encoders. We investigate a range of ways to adapt the Transformer architecture to keyword spotting and introduce the Keyword Transformer (KWT), a fully self-attentional architecture that exceeds state-of-the-art performance across multiple tasks without any pre-training or additional data. Surprisingly, this simple architecture outperforms more complex models that mix convolutional, recurrent and attentive layers. KWT can be used as a drop-in replacement for these models, setting two new benchmark records on the Google Speech Commands dataset with 98.6% and 97.7% accuracy on the 12 and 35-command tasks respectively.
By Axel Berg1, 2 , Mark O’Connor1, Miguel Tairum Cruz1
1Arm ML Research Lab, UK
2Lund University, Sweden
Click here to read the paper. Click here to read the Arm Community introduction to the paper.
 |
 |
 |
 |
 |
 |
 |
 |
 |
Leave a Reply