PathPartner Develops a Driver Monitoring System Using 4D mmWave Radar & Camera Fusion

author: Eric Walz   

As the world's automakers develop more vehicles that come with level-2 or higher automated driving features that still require human supervision, there is a growing demand in the auto industry for in-cabin monitoring systems designed to keep an eye on the driver to make sure they are paying attention and are ready to take back control of an autonomous vehicle if needed. 

One solution for the world's automakers is to add a driver monitoring system (DMS) to their vehicles. With demand growing for these types of occupant monitors, a handful of companies are actively working on such systems, including PathPartner. 

PathPartner is a specialized product engineering services company founded in 2006. The company has a global presence, with development centers in Bangalore and Cochin, India, Frankfurt, Germany and Silicon Valley. 

More recently, PathPartner developed an occupant monitoring sensor (OMS) that utilizes 4D radar technology combined with camera fusion. It's designed to supplement a camera-based vehicle occupancy monitoring system. However, PathPartner's DMS combines the strengths of both into a single, low cost unit for automakers.

"Combining both camera and radar offers significant improvement in accuracy under low-light, noisy environments," said Vinay MK, Vice President of R&D of PathPartner. "Our camera sensor with near-infrared illumination enables reliable performance under different lighting conditions."

How Does 4D Radar Work?

4D radar represents the next generation of radar technology. It works very much like the flash of a camera that momentarily illuminates the subject before taking a picture. However, instead of using the light from a camera flash, imaging radar uses pulses of radio wavelengths, according to Anthony Freeman, Manager, Planetary Science Formulation at NASA Jet Propulsion Laboratory in Pasadena, CA.

In a traditional camera, the light from the flash is reflected back through the camera lens to capture a scene in low light conditions. Whereas imaging radar uses an antenna and static random-access memory (SRAM) memory to record its images. The radar only captures the light that was reflected back towards the radar's antenna. 

This reflected light is collectively known as a "point cloud" and each frame shows the dimension, shape, location and movement of vehicle occupants. From these 4D radar images, deep-learning algorithms can be used to determine what they are.

When used as part of a driver or passenger monitoring system, PathPartner's 4D radar technology can be used to detect subtle movement or even a person's breath in a vehicle's interior, such as a sleeping child under a blanket or the presence of pets.

Three Types of Automotive Radar Systems

Automotive radar systems can be classified based on their range. Short-range radar for example, can detect objects up to 30 meters away from the vehicle. Short-range radar is usually placed in the rear corners of a vehicle to support blind spot detection, rear collision warnings, cross-traffic alert and for parking assistance.

Medium range radar detects objects within a 30-80 meter range and long-range radar is for distances greater than 80 meters. Long-range radar is used for vehicle safety systems such as Automatic Emergency Braking (AEB) and ACC (Adaptive Cruise Control).

The third type is millimeter-wave radar, which is the type used in Pathparter's DMS, It's used more frequently in the auto industry due to its smaller wavelength, higher resolution, accuracy and ability to distinguish between two objects. It's a popular choice for automakers due to its small size and robust performance.

An overview of PathPartner's occupant monitoring solution.

Vehicle Occupant Monitoring using Millimeter Wave Radar

FutureCar spoke more in-depth with PathPartner's Vice President of R&D Vinay MK, and he explained a bit more about the company's underlying technology used in its DMS. We also asked him to explain the main differences between camera-based cabin monitors and those that use 4D millimeter wave radar.

"Camera-based units can detect proper usage of seat belts and stature of the occupants, both of which are critical inputs to airbag deployment. Camera-based units can be used to monitor the activities of the driver like smoking, talking on the phone, eating, drinking, mobile texting, distraction, falling off asleep, suddenly falling ill," said Vinay. "They can also detect the presence of certain kinds of objects like empty baby seats, hanging coats, etc."

Vinay said that 4D radar systems perfectly complement camera-based monitoring systems with occupancy detection under no-light conditions. He said that 60GHz radar systems have sufficient resolution to detect the difference between infant and adult breathing rates. He also said that combining both camera and radar data offers significant improvements in accuracy under low-light conditions.

"Detection of infants in baby seats, even when covered under blankets, is possible only by a 4D radar module," said Vinay. "Detection of presence in a parked car is one of the major reasons for using 4D radar in DMS." 

The National Safety Council reported that on an average 39 children under the age of 15 die each year from heat stroke after being left in a vehicle. As a result, the Hot Cars Act was created by Congress in 2019 which requires the U.S. Department of Transportation (DOT) to mandate that all new vehicles have a child presence detection system, such as the type of occupant monitoring system being developed by PathPartner.

To address processing demands for automakers, PathPartner DMS has been ported on NextChip's Apache4, which is a pre-processor for vision systems used for advanced driver assist systems (ADAS). The Apache-4 includes a dedicated sub-system of image processing accelerators and optimized software, which reduces compute requirements by as much as 70%, according to Nextchip.

"The APACHE4 processor offers advanced image processing in a highly integrated manner, said Vinay. "Its advanced algorithms deliver a highly improved detection rate versus conventional vector-based classifiers, making it a highly affordable DMS solution for Euro NCAP mass-market adoption." 

PathPartner attracted much attention after leading chipmaker Qualcomm Inc. chose to include the company's DMS in its concept car on display at CES 2019 in Las Vegas, which is one of the world's biggest technology events.

In January, PathPartner announced the launch of its PT605 system on module (SOM) and smart camera reference design kit, which is based on Qualcomm's QCS605 system on chip (SoC).

PathPartner's DMS is designed around multiple small convolutional neural network (CNN) models combined with adaptive machine learning modules. Due to this hybrid architecture, the company's imaging radar technology can process 60 frames per second from a camera feed with very low footprint using DSPs supplied by Qualcomm, the company says. 

The combination of PathPartner's high-performance system-on-module (SOM) and smart camera reference design kit combined with the company's device engineering services also enables device makers to address the growing demand for imaging radar outside of the auto industry, including for AI-enabled cameras and IoT devices, the company said.

"Our camera sensor with near-infrared illumination enables reliable performance under different lighting conditions. The camera module and processing unit are attached via a high-end automotive-grade FAKRA connector, which offers installation flexibility and low form factor camera installations," said Vinay.

FAKRA (Fachkreis Automobil) is an industry standard for automotive-grade connectors developed in Germany that can operate up to 6 GHz and are highly reliable. These types of connectors are typically used to connect wireless imaging and video devices.

PathPartner's DMS is also supported on hardware supplied by Silicon Valley software company Cadence, which makes products suitable for many automotive applications.

"Cadence is a high-performance DSP and convolutional neural network(CNN) accelerator provider with design wins on automotive processors," said Vinay. "By porting our DMS algorithms and models on Cadence processors we have them ready to try out for automotive OEMs."

The Tensilica Vision DSP imaging products from Cadence that PathPartner is using for its DMS were designed to run the complex algorithms used for imaging and computer vision, including object and face recognition and tracking, low-light image enhancement, digital zoom, and gesture recognition. 

The Tensilica family also offers outstanding performance for running AI powered software, which is another reason it was selected by PathPartner.

PathPartner's robust DMS machine learning algorithms have also been showcased on Intel's flagship Apollo Lake processor, as well as on NXP's IMX8x automotive-grade processor, the company said. 

"This unit is being deployed on trucks and vans in Bangalore, India," Vinay told Automotive News in an interview last year. "We've had encouraging feedback from one Tier 1 for being able to port compute-intensive algorithms on just 1.2GHz quad-core processors."

Until autonomous vehicles can safely operate drive without human intervention, which is still years away, companies like PathPartner are developing interim solutions to make sure the driver is always paying attention whenever a vehicle is operating in autonomous mode, or that a child or pet is not inadvertently left behind in a vehicle.  

According to Market Insights, the long-term outlook indicates that demand for DMS will grow significantly in the auto industry, especially if regulations are passed requiring them in all passenger vehicles, such as the case with driver and passenger airbags and backup cameras, which all passenger vehicles sold in the U.S. must come equipped with. 

However, driver and occupant monitoring systems are not just for self-driving vehicles. Commercial fleet operators can improve the safety of their fleets by monitoring drivers for alertness. The technology can also be used for backseat occupant monitoring needs, to make sure people or pets are not accidently left unattended in a hot vehicle. 

"Fleet customers are definitely initial adopters of our technology. They have a well-defined return on investment and we provide very cost-effective solutions to meet their specific needs," said Vinay. 

PathPartner also has a strategic partnership with a leading Tier-1 supplier that's focused on the development of advanced vision systems in the mobility space. Pathparter declined to name the company at this stage. However, Vinay confirmed that the two companies are jointly developing a camera & radar-based occupancy monitoring & driver monitoring ECU to support the processing requirements of the system.

"We continue to work with large Tier-1s and we will disclose their names at an appropriate time," said Vinay. 

PathPartner's strategic partner will integrate its occupant monitoring systems (OMS) and DMS algorithms on an automotive-grade ECU. 

Using Edge Computing Instead of the Cloud for Running AI Applications

PathPartner is also using edge computing for its occupant sensors instead of sending data to the cloud for processing, which increases latency. For the automotive industry, this approach can offer a higher level of security and lessens the reliance on cellular networks to transmit data, which is not always reliable enough for safety-critical automotive systems. 

"Moving AI workloads from the cloud to devices is rising through the roof due to the growing concerns around data privacy, communication latencies and unreliable connectivity," explained Ramkishor Korada, Co-Founder, and VP of Marketing, Business Dev and Sales at PathPartner. "Device makers are adopting this change and bringing next-wave of innovation in smart IoT devices such as those for video surveillance, dashboard cameras, drones, video conferencing, and so on," he said.

We asked Vinay about some of the challenges of integrating a DMS with an existing vehicle's ADAS, since a driver monitoring system requires a certain level of integration with the rest of a vehicle's controls, including braking and steering systems, as well as AI-powered perception systems, which are designed interpret a driver's actions.

"We recently completed a cloud-based AI project for a fleet management company based out of North America," explained Vinay. "A driver looking extremely left would be considered distracted. However when that's followed by a U-turn we know it's not a distraction. Similarly, 76% of the harsh braking events were just around traffic lights. These events need to be contextualized and classified using AI-enabled software."

In addition to developing an in-cabin occupant monitor, PathPartner provides advanced software integration, validation & testing services to all major automotive Tier 1s. The company recently had a start-of-production design win with a major European Tier-1 for in-cabin occupancy detection. 

Vinay predicts that the initial emphasis in the auto industry will be on the low-cost integrated DMS, which can handle most of the varied conditions. So it's a bit of a trade off with performance vs. cost for automakers. He also predicts that the first driver monitoring systems (DMS) deployed at scale may not even support eye-gaze detection due to costs. 

"The focus is going to be on cost impact to the overall roll-out of DMS slated for 2023." said Vinay. "Start-ups like us who are focusing on very advanced features and complex models will also have to plan for a long gestation period."

Eric Walz

Originally hailing from New Jersey, Eric is a automotive & technology reporter covering the high-tech industry here in Silicon Valley. He has over 15 years of automotive experience and a bachelors degree in computer science. These skills, combined with technical writing and news reporting, allows him to fully understand and identify new and innovative technologies in the auto industry and beyond. He has worked at Uber on self-driving cars and as a technical writer, helping people to understand and work with technology.