Impact Stories: ICSI in the Drivers Seat

Welcome to ICSI Impacts, a series featuring some of ICSI's most impactful research from its inception to today. First up is the story of how ICSI and the German Research Center for Artificial Intelligence (DFKI) joined forces to create voice-control technology used in cars on the market today. It’s one of many fruitful collaborations between DFKI and ICSI and highlights the magic that happens when postdocs from around the world team up with our scientists.

If you have a newer vehicle, the odds are good that you routinely talk to your car. If you have a newer Audi or Mercedes, you almost certainly talk to your car using technology initially developed at ICSI in a transatlantic cooperation with the German Research Center for Artificial Intelligence (DFKI).

Spoken dialogs combined with gestures and eye-pointing  provide a seamless, intuitive way to control car functions like music and air conditioning—or even converse with your vehicle about road hazards or restaurant reservations—without taking your hands off the wheel or your eyes off the road. When Wolfgang Wahlster and Nelson Morgan started working toward this goal at ICSI in the 1990s, the idea seemed like science fiction. Fifty-one patents and five spinoff companies later, their dream has become reality.

The team completed their initial work, a project called SmartKom, supported by over €25 million from the German Ministry for Research, in 2003. The project’s key innovations were its combination of speech, gesture, and facial expressions and eye pointing as input modes, the use of life-like characters and synthetic speech as output modes, and a sophisticated dialog system for multi-user interaction. In the years that followed, research and engineering teams generated over 200 scientific papers as they converted the core technology into consumer-ready interactive tools. One such tool, the SemVox user interface, won first prize in the Car HMI Europe Competition when it launched with the 2017 Audi A8, the first car using the technology to hit the market.

Wahlster, who has been recognized with prestigious awards including the German Future Prize and ACM’s ICMI Sustained Accomplishment Award, founded DFKI in 1988. The initial ICSI project and its many spinoff technologies benefitted greatly from ICSI’s close ties with Germany, with several contributors coming to ICSI through the IFI-DAAD program. Andreas Stolcke contributed to the ICSI project as a graduate student and postdoc and now serves as Senior Principal Scientist at Amazon Alexa. As IFI-DAAD fellows at ICSI, Christian Mueller and Michael Feld devised a way to automatically recognize a speaker’s age and gender and also collaborated with the Volkswagen Electronics Research Laboratory to integrate audio from outside the vehicle into a multimodal interaction system—technology that is now rolling off the assembly line.

What made ICSI a good place to pursue this work? 

Wolfgang Wahlster: “ICSI provides an excellent environment for international cooperation between top researchers in many important fields of computer science as one of the world’s leading research labs. Its excellent welcome culture for international visitors combined with the California lifestyle and the Silicon Valley ecosystem make it an ideal place for an exchange with the best postdocs from Germany.”