Imagine, just a piece of black tape can cause a sophisticated car to hit 85mph after “seeing” a 35mph-limit sign.
For those who have been eagerly observing the steady move towards autonomous driving, the last several years have been fascinating. While semi-autonomous vehicles have existed for many years, the vision of fleets of fully autonomous vehicles operating as a single connected entity is very much still a thing of the future.
However, the latest technical advances in this area bring us a unique and compelling picture of some of the capabilities we might expect to see “down the road.” For example, nearly every new vehicle produced in 2019 has a model which implements state-of-the art sensors that utilise analytics technologies, such as machine learning or artificial intelligence, and are designed to automate, assist or replace many of the functions humans were formerly minding—ranging from rain-sensors on the windshield to control-wiper blades, to object detection sensors using radar and lidar for collision avoidance, to camera systems capable of recognising objects in range and providing direct driving input to the vehicle.
This broad adoption represents a fascinating development in our industry; it is one of those very rare times when researchers can lead the curve ahead of adversaries in identifying weaknesses in underlying systems. And that is what inspired two researchers from McAfee to embark on. Steve Povolny, Head of Advanced Threat Research, and his team mate Shivangee Trivedi—had a specific goal: identify and illuminate a broad spectrum of threats in today’s complex landscape.
With model hacking, the study of how adversaries could target and evade artificial intelligence, they had an incredible opportunity to influence the awareness, understanding and development of more secure technologies before they are implemented in a way that has real value to the adversary.
With this in mind, the duo decided to focus their efforts on the broadly deployed MobilEye camera system, today utilised across over 40 million vehicles, including Tesla models that implement Hardware Pack 1.
New defensive tool: Model Hacking
MobilEye is one of the leading vendors of Advanced Driver Assist Systems (ADAS) catering to some of the world’s most advanced automotive companies. Tesla, on the other hand, is a name synonymous with ground-breaking innovation, providing the world with the innovative and eco-friendly smart cars.
McAfee Advanced Threat Research (ATR) had been studying what they call “Model Hacking,” also known in the industry as adversarial machine learning. Model Hacking is the concept of exploiting weaknesses universally present in machine learning algorithms to achieve adverse results. This is to identify the upcoming problems in an industry that is evolving technology at a pace that security has not kept up with.
The duo started their journey into the world of model hacking by replicating industry papers on methods of attacking machine learning image classifier systems used in autonomous vehicles, with a focus on causing misclassifications of traffic signs. They were able to reproduce and significantly expand upon previous research focused on stop signs, including both targeted attacks, which aim for a specific misclassification, as well as untargeted attacks, which do not prescribe what an image is misclassified as; just that it is misclassified.
Ultimately, the team was successful in creating extremely efficient digital attacks that could cause misclassifications of a highly robust classifier, built to determine with high precision and accuracy what it is looking at, approaching 100% confidence.
The hacks worked! What next?
They further expanded efforts to create physical stickers that model the same type of perturbations, or digital changes to the original photo, that can trigger weaknesses in the classifier and cause it to misclassify the target image. This set of stickers is specifically created with the right combination of color, size and location on the target sign to cause a robust webcam-based image classifier to think it is looking at an “Added Lane” sign instead of a stop sign.
In reality, modern vehicles do not yet rely on stop signs to enable any kind of autonomous features such as applying the brakes, so the team decided to alter their approach and shift over to speed limit signs. They knew, for example, that the MobilEye camera is used by some vehicles to determine the speed limit, display it on the heads-up display (HUD), and potentially even feed that speed limit to certain features of the car related to autonomous driving. More on that later…
Next, the team repeated the stop sign experiments on traffic signs, using a highly robust classifier, and their trusty high-resolution webcam. And just to show how robust that classifier was, they can make many changes to the sign—block it partially, place the stickers in random locations—and the classifier does an outstanding job of correctly predicting the true sign. While there were many obstacles to achieving the same success, the team was ultimately able to prove both targeted and untargeted attacks, digitally and physically, against speed limit signs.
Switching from White Box to Black
At this point, you might be wondering “what’s so special about tricking a webcam into misclassifying a speed limit sign, outside of just the cool factor?” Not much, really. The team felt the same, and decided it was time to test the “black box theory.”
What this involves, in its most simple form, is attacks leveraging model hacking that are trained and executed against white-box, also known as open source systems, will successfully transfer to black box, or fully-closed and proprietary systems, so long as the features and properties of the attack are similar enough.
For example, if one system is relying on the specific numeric values of the pixels of an image to classify it, the attack should replicate on another camera system that relies on pixel-based features as well.
The last part of their lab-based testing involved simplifying this attack and applying it to a real-world target. They wondered if the MobilEye camera was as robust as the webcam-based classifier they built in the lab? Would it truly require several highly-specific and easily noticeable stickers to cause a misclassification?
The team was able to run repeated tests on a 2016 Model “S” and 2016 Model “X” Tesla using the MobilEye camera (Tesla’s hardware pack 1 with EyeQ3 mobilEye chip). The first test involved simply attempting to recreate the physical sticker test—and, it worked, almost immediately and with a high rate of reproducibility.
In the lab tests, they had developed attacks that were resistant to change in angle, lighting and even reflectivity, knowing this would emulate real-world conditions. While these were not perfect, their results were relatively consistent in getting the MobilEye camera to think it was looking at a different speed limit sign than it was.
The next step in testing was to reduce the number of stickers to determine at which point they failed to cause a misclassification. As they began, they realised that the HUD continued to misclassify the speed limit sign. They continued reducing stickers from four adversarial stickers in the only locations possible to confuse their webcam, all the way down to a single piece of black electrical tape, approximately two inches long, and extending the middle of the “3” on the traffic sign.
Even to a trained eye, this hardly looks suspicious or malicious, and many who saw it did not realise the sign had been altered at all. This tiny piece of sticker was all it took to make the MobilEye camera’s top prediction for the sign to be 85 mph. The finish line was close.
Speed Assist? Or accident assist?
Finally, they began to investigate whether any of the features of the camera sensor might directly affect any of the mechanical, and even more relevant, autonomous features of the car. After extensive study, they came across a forum referencing the fact that a feature known as Tesla Automatic Cruise Control (TACC) could use speed limit signs as input to set the vehicle speed.
This took the researchers down their final documentation-searching rabbit hole; Speed Assist, a feature quietly rolled out by Tesla in 2014. Finally! They can now add these all up to surmise that it might be possible, for Tesla models equipped with Speed Assist (SA) and Tesla Automatic Cruise Control (TACC), to use their simple modification to a traffic sign to cause the car to increase speed on its own!
The ultimate finding here was that the team was able to achieve the original goal. By making a tiny sticker-based modification to their speed limit sign, they were able to cause a targeted misclassification of the MobilEye camera on a Tesla and use it to cause the vehicle to autonomously speed up to 85 mph when reading a 35-mph sign. The vehicle’s speed indeed started to spike and TACC accelerated on its way to 85, but given their safety considerations, they applied the brakes well before the vehicle reached target speed.
It is worth noting that this is seemingly only possible on the first implementation of TACC when the driver double taps the lever, engaging TACC. If the misclassification is successful, the autopilot engages 100% of the time.
Of note is that all these findings were tested against earlier versions (Tesla hardware pack 1, MobilEye version EyeQ3) of the MobilEye camera platform. They did get access to a 2020 vehicle implementing the latest version of the MobilEye camera and were pleased to see it did not appear to be susceptible to this attack vector or misclassification, though testing has been very limited. The researchers are thrilled to see that MobilEye appears to have embraced the community of researchers working to solve this issue and are working to improve the resilience of their product.
Still, it will be quite some time before the latest MobilEye camera platform is widely deployed. The vulnerable version of the camera continues to account for a sizeable installation base among Tesla vehicles. The newest models of Tesla vehicles do not implement MobilEye technology any longer, and do not currently appear to support traffic sign recognition at all.
EPILOG: 18 Months of Research
McAfee ATR follows a responsible disclosure policy, as stated on their website. As such, the team disclosed the findings below to both Tesla and MobilEye 90 days prior to public disclosure. McAfee disclosed the findings to Tesla on September 27, 2019 and MobilEye on Oct 3, 2019. Both vendors indicated interest and were grateful for the research but have not expressed any current plans to address the issue on the existing platform. MobilEye did indicate that the more recent version(s) of the camera system address these use cases.
Moving forward on the long road ahead, the McAfee ATR team feel it is important to close this research report with a reality check. Is there a feasible scenario where an adversary could leverage this type of an attack to cause harm? Yes, but in reality, this work is highly academic at this time.
Still, it represents some of the most important work that they as an industry can focus on to get ahead of the problem. If vendors and researchers can work together to identify and solve these problems in advance, it would truly be an incredible win for us all. Povolny and Trivdei have this parting words of advice:
In order to drive success in this key industry and shift the perception that machine learning systems are secure, we need to accelerate discussions and awareness of the problems and steer the direction and development of next-generation technologies. All puns intended.
