Long-Short Term Memory Models Everywhere!

Oriol Vinyals

Google Brain

Tuesday, September 30, 2014
12:30 p.m., Conference Room 5A

Supervised large deep neural networks achieved state-of-the-art on speech recognition and computer vision. Although very successful, deep neural networks can only be applied to problems whose inputs and outputs can be conveniently encoded with vectors of fixed dimensionality - but cannot easily be applied to problems whose inputs and outputs are sequences.

In this work, we show how to use a large deep Long Short-Term Memory (LSTM) model to solve domain-agnostic supervised sequence to sequence problems with minimal manual engineering.  Our model uses one LSTM to map the input sequence to a vector of a fixed dimensionality and another LSTM to map the vector to the output sequence.  We applied our model to a machine translation task and achieved encouraging results. On the WMT'14 translation task from English to French, a model combination of 6 large LSTMs achieves a BLEU score of 34.0 (where a larger score is better). For comparison, a strong standard statistical MT baseline achieves a BLEU score of 33.3.  When we use our LSTM to rescore the n-best lists produced by the SMT baseline, we achieve a BLEU score of 35.6.

This is joint work with Ilya Sutskever and Quoc Le.

I may also cover other applications of LSTMs at Google, such as acoustic and language modeling, time and interest permitting.

Bio:

Oriol received a double degree from the Polytechnic University of Catalonia (Barcelona, Spain) in Mathematics and Telecommunication Engineering, and a Master in Computer Science from the University of California, San Diego in 2009. In 2013, he finished his PhD at the University of California, Berkeley, under the supervision of Nelson Morgan, and was one of the 2011 Microsoft Research PhD Fellowship recipients. Oriol is a Research Scientist at Google Brain, where he continues working on his areas interests, which include artificial intelligence, with particular emphasis on machine learning, language, speech, and vision.