Since July 6th, car manufacturers have been obliged to fit a number of advanced driver assistance systems (ADAS) into new models of vehicles they want to sell in the EU. These include an intelligent cruise control that connects the cruise control to a signal recognition system; a reversing alarm that makes maneuvering easier, or a black box that can help find out what caused an accident. All these measures work thanks to various sensors (radar, lidar, ultrasound, external cameras) that process information about the vehicle’s surroundings.
But data is also collected on what happens behind closed doors. Brussels has added to the new mandatory measures one more that affects drivers directly. Vehicles planning to be registered in the EU from this summer, the necessary procedure before they can be sold on Community territory, must include systems that measure the driver’s level of alertness, sleep and fatigue. When drowsiness is detected, an audible, visual, or sensory alert is sent to the user. These measures, which some brands such as BMW have already introduced voluntarily in their models, will be mandatory for all new cars sold from 2024 (Brussels gives the industry two years to sell vehicles that do not have these devices).
How can the car recognize if the driver is showing signs of fatigue? The solution manufacturers are opting for is to fit eye-tracking or facial recognition systems into cars, although without calling it that. In other words, small cameras placed at strategic points in the cabin to watch what’s happening in the cabin. For example, Bosch advocates the installation of cameras on the steering wheel that analyze the driver’s blinking.
Measuring blink frequency and head tilt is an effective way to infer whether or not you are drowsy. “It is believed that the more times you blink, the more likely you are to get tired. He’s been experimenting with it for years. These systems require some calibration so that the person being measured is at a certain distance, but they are reliable,” explains José Miguel Fernández Dols, professor of social psychology at the Autonomous University of Madrid and an expert in facial expressions and emotions.
Affectiva is the world’s largest supplier of so-called driver monitoring systems. The American company, which was acquired by Swedish SmartEye in 2018, has deals with 14 major automakers, although they can only talk openly about Volvo and BMW for reasons of confidentiality. Although this last manufacturer recently switched suppliers, since 2018 the Series 2 models have featured the SmartEye driver status detection software, which is also present in the Series 4 and X3 through X7. “When the Urban Cruise Control function with traffic light recognition is activated, the system checks whether the driver is facing the road. If so, the vehicle drives off automatically as soon as the traffic light turns green. If not, an audible signal will be activated to warn that you can now drive,” a BMW spokesman said.
“If you want to understand driver behavior and make sure they’re not tired or distracted, it’s important to look them in the face,” says Gabi Zijderveld, Marketing Manager at SmartEye, via video call. The technology company markets systems that focus only on the driver and other more advanced ones that take the rest of the passengers into account. “We use machine learning and computer vision systems with in-car cameras to analyze passenger faces and recognize objects to see if someone is on the phone. We can identify multiple drivers of the car, so everyone is offered the configuration they want, and detect if a child is sitting in the back seats to warn if they stay inside,” the manager describes. Its systems have been trained in more than 40,000 driving hours by around 6,000 different people, resulting in around 500,000 different data entries.
At the moment they only use cameras, says Zijderveld, but don’t rule out installing additional sensors in the future to measure passenger sweating or the audio from the cabin. This is the direction in which the automotive industry is moving, he assures. “The industry has realized that by putting cameras in cars and using artificial intelligence, you can collect a lot of very complex data to detect human behavior that evolves over time. And that can then be monetized.”
The market trend seems clear. The European New Car Rating Program (EuroNCAP), a body backed by several EU governments and the industry’s major manufacturers, which issues reports and tests of the vehicles for sale, speaks of driver monitoring systems as a technology that will become as necessary as seat belts. “These types of systems will be very important to improve both the safety and the driving experience of the BMWs of the future,” they say of the German brand.
Security vs Privacy
It may come as a surprise that municipal institutions advocate the installation of such devices in vehicles. Brussels has been very restrictive on biometric recognition technologies precisely because it handles extremely sensitive personal data. Just as the fingerprint can be intentionally erased or burned in, the human iris is the same from the age of three. The patterns that describe the face are also unchangeable: even age is not able to change the unique disposition of the facial features of each individual. The EU’s approach to biometrics is currently conservative. Legislation states that you must be very careful about collecting and using this data (only if the use is proportionate to the risks involved). The devastating potential of this technology if it falls into the wrong hands was demonstrated in Afghanistan last year: According to international media reports, the Taliban used biometric data from former state officials to pursue and murder them.
But there are nuances. “The so-called EU General Safety Regulation Difference between biometric information and physiological metrics. The first includes technologies that allow people to be personally identified, such as the retina reader or facial recognition; the second has to do with other parameters, such as heart rate measurement, perspiration, blinking, electrodermal activation or respiration,” says Lorena Jaume-Palasí, expert in ethics and legal philosophy of technology and adviser to the Spanish government and the European Parliament on issues related to the ethics of artificial Intelligence.
According to this nomenclature, the researcher points out, biometric technologies could be passed off as something else. “Physiological metrics capture biological aspects and process this data based on statistical measurements. And that is biometrics,” he emphasizes. “This regulation appears to be a strategy to accommodate biometrics in mobility without using that word,” he says. The European regulation 2019/2144, which regulates the innovations that cars must contain from this summer, clarifies that “each of these safety systems must function without using any kind of biometric information about drivers or passengers, including facial recognition”. However, Jaume-Palasí insists that it is difficult to place the scale correctly. “Depending on the artificial intelligence technique used, something is not specified in the regulation, we can talk about facial recognition or not. Since we don’t know what the algorithm is looking at, we can’t say for sure that facial recognition exists,” he concludes.
The use of these technologies entails a number of risks and questions: Where and for how long is user data stored? Who can access it? What happens if the police ask for access to this data for security reasons? What if this data is not 100% reliable and leads to errors?
The articles of the regulation itself state that “driver drowsiness and alertness warning systems (…) shall not continuously record or store data that is not necessary for the purposes for which the data was collected for the closed system”. The data must remain in the same car and cannot be processed by third parties. That’s what Affectiva executives say. “Our systems come into the car and don’t have to be in the cloud. We don’t store any images either: we use the video captured by the cameras in real time to understand what’s happening in the vehicle,” describes Zijderveld.
According to sources from the Spanish Data Protection Agency (AEPD), national authorities are not authorized to carry out prior assessments of this technology as it is a European regulation. “The controllers who want to implement facial recognition technologies must consider whether their treatment could fit into one of the exceptions of the General Data Protection Regulation (RGPD),” the same sources warn. In other words, biometric data may only be processed if the need to do so is justified.
Is it necessary to study the face?
According to a 2015 study, there are simpler methods than biometrics to detect driver fatigue: it is enough to know how to drive. Collecting data about how and how hard you brake, accelerate or turn provides sufficient and reliable information to determine if you are driving with due caution. It would also be a non-intrusive method, says Jaume-Palasí, since it doesn’t require the installation of cameras.
Using facial expression to detect sleepiness can be misleading. “The information on muscle movements is interesting, but of questionable reliability for a face in constant motion and changes in position. And the attribution of emotions is straight up science fiction,” concludes Fernández Dols. He and his team have shown that it is impossible to infer emotions from people’s facial expressions alone.
The possible impact on the safety of vehicle occupants is also unclear. “Controlling distraction does not solve the problem of distraction,” stresses Jaume-Palasí. “How does this improve the safety of the driver and his companions?” asks Professor Fernández Dols. “Or is it part of another big Cambridge Analytica-style experiment?”