Consumer Reports Says Tesla's Autopilot Performs Worse Than a Human Driver

resource from: Consumer Reports   

Tesla's "Navigate on Autopilot" feature is capable of automated highway driving, including automatic lane changes.

Last month, Tesla CEO Elon Musk boasted that Tesla's Autopilot self-driving feature was "vastly ahead of everyone", including Waymo, the self-driving arm of Alphabet that spun out of Google's early self driving car program.

Musk said in a recent interview with MIT researcher Lex Fridman, who recently published a study on "driver functional vigilance" when using Tesla Autopilot, "I think it will become very, very quickly, maybe even towards the end of this year – but I'd say, I'd be shocked if it's not next year at the latest – that having a human intervene will decrease safety."

However, Consumer Reports (CR) recently conducted its own test of Tesla's new "Navigate on Autopilot" update and concluded that it doesn't work as well as Musk claims. In fact, using it could be a potential safety risks for drivers.

Much of the testing by CR was focused on the software update "Navigate on Autopilot" that allows drivers to "grant permission" to their Tesla vehicle to automatically change lanes on the highway.
The feature is activated by the driver, who must first turn it on. Once operating in Navigate by Autopilot mode, a driver can cancel an automated lane change at any time by using the turn-signal stalk, touching the brake pedal or holding the steering wheel in place.

CR determined that Navigate on Autopilot lagged far behind a human driver's capabilities. The consumer watchdog said that the feature cut off cars without leaving enough space and even passed other cars in ways that violate state traffic laws, according to several law enforcement personnel CR interviewed for its report. As a result, the driver often had to prevent the system from making poor decisions.

"The system's role should be to help the driver, but the way this technology is deployed, it's the other way around," says Jake Fisher, Consumer Reports' senior director of auto testing. "It's incredibly nearsighted. It doesn't appear to react to brake lights or turn signals, it can't anticipate what other drivers will do, and as a result, you constantly have to be one step ahead of it."

Although Musk promised that Tesla will have full self-driving technology by the end of next year, Cr's experience suggests it will take longer. CR spoke with experts who said that the automatic lane-change feature demonstrates the technological limits of Tesla's current hardware.

Navigate on Autopilot was Released Last Year

Tesla released the first version of Navigate on Autopilot last year. The system keeps the car centered in a lane and controls its speed, maintaining a set distance from vehicles in front. Tesla's claimed it would make driving "more relaxing, enjoyable and fun."

In the first version of Navigate on Autopilot, Tesla said the software could guide a car through highway interchanges and exits and would only make a lane change if the driver confirmed it by using the turn signal or accepting an onscreen prompt.

However when CR tested this version of the Navigate feature in November 2018, it found that it lagged behind a human driver's abilities in more complex driving scenarios.

On April 3, Tesla announced an updated version of Navigate on Autopilot that gives drivers the option to hand over control for automatic lane changes to the car's computer. It's available as a software update for Tesla owners who purchased the Enhanced Autopilot upgrade or Full Self-Driving Capability.

Once activated by the driver, the vehicle will automatically execute lane changes and continue to do so until drivers change the setting or turns off Autopilot. To turn it off, a driver will have to navigate to a sub-menu on the vehicle's touch screen and re-enable lane-change confirmation.

The first time a driver disables lane-change confirmation, a warning message pops up that reads, in part, "This does not make your vehicle autonomous." However, this pop-up does not appear again when the system is in use.

Autopilot was engaged when this Tesla Model S crashed into the back of a fire truck near Culver City, California in Jan 2018.

A Tesla spokesperson told CR that "Navigate on Autopilot is based on map data, fleet data, and data from the vehicle's sensors. However, it is the driver's responsibility to remain in control of the car at all times, including safely executing lane changes."

CR also pointed out that Autopilot has been engaged in at least three fatal crashes in the U.S., according to the National Transportation Safety Board (NTSB).

The latest fatal crash occurred in March, when the driver of a Tesla Model 3 was killed after the vehicle struck the side of a semi-truck that crossed its path in Florida. NTSB investigators found that the driver activated Autopilot about 10 seconds before the collision. The system should have at least activated emergency braking, but failed to do so.

CR Testers Experienced Agressive Lane Changes

For its tests, CR enabled the feature and drove on several highways across Connecticut. Multiple testers reported that the Tesla often changed lanes in ways that a safe human driver would not, including cutting too closely in front of other cars and even passing on the right.

In addition, Tesla's claims that the vehicle's three rearward-facing cameras can detect fast-approaching objects from the rear better than the average driver can. However, CR testers found the opposite to be true.

"The system has trouble responding to vehicles that approach quickly from behind," Fisher says. "Because of this, the system will often cut off a vehicle that is going a much faster speed since it doesn't seem to sense the oncoming car until it's relatively close."

Fisher added that merging into traffic smoothly is difficult for Autopilot and the vehicle applied that brakes once merge was made.

"It is reluctant to merge in heavy traffic, but when it does, it often immediately applies the brakes to create space behind the follow car—this can be a rude surprise to the vehicle you cut off." Fisher said.

CR reported that testers often had to cancel a pass that had been initiated by Autopilot—usually by applying steering force to move the car back into the travel lane—when they felt that the maneuver would be unsafe.

Overall, CR testers found that the system's lack of situational awareness made driving less pleasant.

Despite the finding by CR, Tesla stands behind the safety of Navigate on Autopilot and said that 500,000 miles have been driven with the lane change confirmation turned off, meaning that the vehicle would perform them automatically.

In a statement to CR, Tesla wrote "Our team consistently reviews data from instances when drivers took over while the feature has been in use, and has found that when used properly both versions of Navigate on Autopilot offer comparable levels of safety. We've also heard overwhelmingly from drivers in our Early Access Program that they like using the feature for road trips and during their daily commutes, and we're excited to release the option to the rest of the Tesla family."

Illegally Passing on the Right

Several CR testers observed Navigate on Autopilot initiate a pass on the right on a two-lane divided highway. A law enforcement official confirmed to CR that this as a "improper pass" in the state of Connecticut and could result in a traffic citation.

Navigate on Autopilot also failed to return to the right-hand travel lane after making a pass, which also could result in a driving infraction, the official confirmed.

A Tesla spokesperson told CR that "it is the driver's responsibility to remain in control of the car at all times, including safely executing lane changes."

Tesla CEO Elon Musk has said that upgrades to Autopilot would be available by 2020. However it would require a robust new processor combined with Tesla's neural nets. Musk said that the new chip is being installed in Tesla Model's produced after late March.

Musk said the new chip is "the best chip in the world." It includes with 6 billion transistors that offers 21 times the performance of the Nvidia chips it was using before.

Musk said that owners who purchased the Full Self-Driving Capability option would get the new chip as part of a no-cost hardware upgrade.

David Friedman, vice president of advocacy at Consumer Reports, warned that as it currently exists, the automatic lane-change function raises serious safety concerns.

"Tesla is showing what not to do on the path toward self-driving cars: release increasingly automated driving systems that aren't vetted properly," he says. "Before selling these systems, automakers should be required to give the public validated evidence of that system's safety—backed by rigorous simulations, track testing, and the use of safety drivers in real-world conditions."

resource from: Consumer Reports

Eric Walz

Originally hailing from New Jersey, Eric is a automotive & technology reporter covering the high-tech industry here in Silicon Valley. He has over 15 years of automotive experience and a bachelors degree in computer science. These skills, combined with technical writing and news reporting, allows him to fully understand and identify new and innovative technologies in the auto industry and beyond. He has worked at Uber on self-driving cars and as a technical writer, helping people to understand and work with technology.