Training Recurrent Neural Networks at Scale

Training Recurrent Neural Networks at Scale: One of our projects at Baidu’s Silicon Valley AI Lab is using deep learning to develop state of the art end-to-end speech recognition systems based on recurrent neural networks for multiple languages. The training set for each language is multiple terabytes in size and each model requires in excess of 10 Exaflops to train. Training such models requires scale and techniques that are unusual for deep learning but more common in high performance computing. I will talk about the challenges involved and the software and hardware solutions that we employ.

View the slides for this session