BMW Debuts a ‘Natural Interaction' Assistant, Allowing Drivers to Use Voice, Gestures or Gaze to Control Vehicle Functions

author: FutureCar Staff    

At the MWC2019 (formerly Mobile World Congress) being held this week in Barcelona, BMW debuted a new in-vehicle control system technology it calls "Natural Interaction", that allows drivers to use their voice, hand gestures or gaze control to interact with their BMW vehicle.

Voice assistants are popular now with the widespread use of Apple's Siri and Amazon Alexa. However, BMW says its new system combines the most advanced voice command technology available with expanded gesture control and gaze recognition to enable multimodal operation for the first time.

BMW's Natural Interaction technology allows the driver to combine voice, gesture and gaze in various ways to interact with their vehicle. The driver decides how they want to interact with the vehicle by using either voice or gestures, based on their own personal preferences or the current situation.

For example, if a driver is having a conversation with a passenger, the system can be set up to only use gesture or gaze, so the voice conversation can continue uninterrupted. Other times when a driver need to keep their eyes on the road, the system can respond to voice controls instead of gestures or gaze for added safety.

Spoken commands are registered and processed using Natural Language Understanding (NLU). NLU is an intelligent learning algorithm, which is constantly being refined. NLU combines and interprets complex speech information, so the vehicle can responds accordingly. Other companies working on NLU include Amazon with Alexa, Apple's Siri, Google's Assistant and Microsoft's Cortana.

"Customers should be able to communicate with their intelligent connected vehicle in a totally natural way," Christoph Grote, Senior Vice President, BMW Group Electronics said in a statement.

"People shouldn't have to think about which operating strategy to use to get what they want. They should always be able to decide freely – and the car should still understand them. BMW Natural Interaction is also an important step for the future of autonomous vehicles, when interior concepts will no longer be geared solely towards the driver's position and occupants will have more freedom."

The system's robust speech recognition technology combined with an optimized sensor suite, including a camera to monitor the driver's gestures. Through precise detection of hand and finger movements a driver can control most interior functions.

The driver can point to a control and ask what to do. This feature can be used to open or close the windows or the sunroof. The system uses a infrared light signal combined with the gesture camera to capture hand and finger movements of occupants in three dimensions.

Can Work Outside the Vehicle Also

With enhanced gesture recognition and the car's high level of connectivity to highly detailed maps, in the future, interactions may no longer be confined to just the vehicle's interior controls.

Passengers may one day be able to interact with their direct surroundings, such as buildings or parking spaces. Even complex queries can be answered quickly and easily by pointing a finger and issuing a voice command. For example a driver might point and say, "What's this building?",  "What is this restaurant called?" or "How much does it cost to park here?" and the system will respond.

The system opens up a whole new way in interact with a BMW vehicle. One possible example presented by BMW is a user being able to buy movie tickets as they drive by a movie theatre. Extending this technology further, a user might be able to stream a movie trailer directly into the vehicle of what's playing nearby.

By connecting digital services, when the driver spots a parking space, they'll be able to find out whether they are allowed to park there and what it costs, and then reserve and pay for it directly via the vehicle's internet-connected infotainment system.

Pointing to a nearby building to get more information

The new system can be combined with BMW's Intelligent Personal Assistant, which learns the routines and habits of each driver with every voice command. As the next logical development, BMW Natural Interaction also offers the possibility of using gestures by simply pointing a finger.

For example, the driver can point a finger at objects in their field of vision and give related voice commands, such as asking for information about opening hours or customer ratings, or reserving a table at a restaurant, transforming the BMW vehicle into a well-informed, helpful passenger.

Attendees at this year's MWC2019 can try out the new features in a mixed-reality installation. A specially-designed spatial concept and virtual-reality goggles are used to create a thoroughly realistic experience that showcases the new possibilities during a virtual ride in the upcoming BMW Vision iNEXT.

BMW has even developed a "training mode"  to try out the pointing gesture features of the system. For the installation at MWC, the pointing gestures are visualized by a dynamic light pulse on objects the driver can interact with by pointing their fingers at the object they want to control.

Pointing to a window highlights it and it can be moved up or down with gestures.

For example, if the driver points to a side window, the window is visually highlighted with a frame of light, by using the voice command "open" the window will open.

Development of driver-vehicle interaction will advance along with BMW's Intelligent Personal Assistant. In the future, the system might include enhanced sensor technology, which will be able to take occupants' emotions into account become even more personalized and tailored to the overall situation.

The first BMW Natural Interaction functions will be available in the upcoming fully-electric BMW iNEXT SUV starting in 2021.

Drivers will also be able to give their in-vehicle assistant a personalized name.

FutureCar Staff