At the recent Chaos Communication Congress in Hamburg two IT experts, Felix Domke and Daniel Lange (former Head of IT strategy at BMW) gave a talk on what exactly had happened with Volkswagen’s cheating of emissions tests.
Over the last several months we have seen a large ammount of media stories published on Dieselgate. With that said, few explained how this trickery worked and who is really responsible for its implementation. That’s why it’s interesting to look at the research conducted by independent experts who tried to find out the truth.
Why this trickery had occurred at the first place?
The biggest issue of emissions tests is that they are always performed with some standard model, like the so-called NEDC (New European Driving Cycle). This model consists of a few pretty short acceleration-braking cycles and one long cycle with higher speed, which represent city and highway traffic respectively. In real life nobody drives like this, and definitely nobody drives exactly like this.
But for emissions measurement they use this very model, thus engineers at car companies can do tricks to improve measurement results. Why do they do it? Plain and simple: it’s way cheaper than to do real improvements. If an enterprise could do something in a cheaper way, it definitely would prefer this way to any other as the bottom line is important to company performance.
“Trickery on that tests is very common,” says Lange. “What tricks people are doing to drive down the emissions? For example they blow up the tires by 3 bars more than you could actually use them on the road. The bottom of the tire looks like this, so that means that you only have that very small portion of the tire that still touches the ground, your resistance gets reduced.”
“They put diesel into the oil, because diesel is lighter than the oil, so friction gets reduced. They take off the mirror on a passenger side, because that is not legally required to exist. So resistance gets away with it. They tape close all the openings of the vehicle. Obviously, when the wind goes over it, it goes much smoother once you have everything taped. All of these things are either Ok, or they kind of borderline grey area. And they do this. This is how actually emissions are tested.”
NOx emissions in NEDC and alternative ADAC tests
The results of this trickery are very simple: measured values have pretty much nothing to do with what is going on in the real world. The whole auto industry knows this very well. Perhaps every car manufacturer uses software tweaks, just like Volkswagen did. As a matter of fact, 15 years ago BMW was actually caught on using a similar trick with software of its motorcycle.
But how do they deploy these software cheats? To understand that, we need to examine how cars’ electronic systems work.
What’s inside the box car?
The piece of electronics directly responsible for everything that happens with car’s engine, including exhaust, is Engine Control Unit (or ECU). Car manufacturers don’t develop these devices themselves, but purchase this equipment from companies who specialize in vehicle electronics systems. There are not that many companies, and at least in Germany the market leader is Bosch.
The firmware code for the ECU is also developed by the same manufacturer; since this firmware is really critical for proper operation of car’s engine and car itself, it is reviewed thoroughly.
As Lange puts it, “this thing is simulated and tested to death. Because it’s hugely important. Because if you have this machine here which has like 200 HP, and if you steer it wrong it will blow up, and it will blow up really hard. That’s why this thing is about the best tested piece of software you will ever find.”
Another fact is that car companies are not allowed to change ECU firmware. At the same time, each specific ECU model can be used by a whole list of car manufacturers in a variety of car models and engines. Therefore, to be compatible with various cars, ECU’s firmware has to be flexible. To achieve this flexibility ECU manufacturers use a lot of variables, which can be adjusted by car manufacturer in accordance to specific requirements of this or that particular car/engine model.
For instance, Bosch EDC17C46, which is the model of ECU used by cars involved in the Dieselgate scandal, has more than 20,000 of such variables. If this virtual benchboard could be somehow materialized in a real world, it definitely would be the hugest and the most complex control panel in the whole world.
Quick summary: the tweaks in firmware code can’t be developed by car manufacturer, but by ECU manufacturer, with request from the former. Since every such modification leaves really long paper trail, ECU manufacturers are likely to inform car companies that using these tweaks is illegal. And eventually the final decision is made by the car manufacturer — to use the tweaks the car company needs to play with above mentioned variables. But how exactly do these tweaks work?
The truth is in the code
To investigate that Felix Domke bought a spare ECU on eBay and, along with his own Volkswagen Sharan (affected by VW recall), reverse-engineered it. He exploited a zero-day hardware vulnerability in ECU’s chip to obtain 2 MB firmware out of its flash memory and examined the code. First of all, the research gave him a very good understanding of how complex processes in car electronics are.
For instance, it turned out that what you see on such ordinary thing as tachometer doesn’t represent engine’s RPM directly. The value displayed by tachometer depends on more than 20 other signals processed by 12 KB of dense code (in case you were wondering: that’s a whole lot of code).
“You realize at this point that there is a lot of cheating could go on here without most people noticing,” says Domke. “You don’t really believe that speedometer in your car displays actual speed. It displays something that is related to speed.”
The emission control part of firmware is even more complex and flexible. However, the core idea of how nitrogen oxides emissions reduction is supposed to work is comparatively simple. In order to get rid of nitrogen oxides, you can add to exhaust some substance called urea (branded by VW as AdBlue). With high temperature it turns to ammonia and reacts with NOx with nice and harmful water and nitrogen as resultant substances.
However, if you added too much of urea, the excess of ammonia goes into exhaust. This is not good for you or your car. That’s why in normal situation it is better to add less urea than to add more (the best approach would be to dose the exact amount of urea, but it’s quite hard to do). Unfortunately, this insufficient dosage won’t get you high score at emissions test. That is basically why car companies need to cheat.
In the case of Volkswagen this trick was performed in the following manner: there were two modes in which urea dosage could be operated. One of them was called ‘regular model’ in which the dose of urea was pretty big, and another called ‘alternative model’ in which urea was significantly underdosed. Domke revealed that about 75% of the time, a VW car’s exhaust system was operating in this ‘alternative mode’ with almost all of the remaining time not dosing urea at all.
As analysis of firmware code showed, the criteria of switching to ‘regular mode’ (which still happened from time to time), besides simple conditions for engine and fuel temperature and atmosphere pressure, included one more interesting criterion. To put it simply, ‘regular mode’ was switched on, when a vehicle’s travelled distance over time chart was within one of several pairs of min-max limit curves.
#Dieselgate investigation: At #32C3 hackers explained what exactly had happened with dirty diesel emissions
Funny enough, one pair of these curves was designed in the exact way to very accurately include distance over time chart for a vehicle running above mentioned New European Driving Cycle. In this case the dosage of urea was high and NOx emission was low enough to meet the very strict European requirements. And that is exactly the core idea of VW software trick.
While there are many takeaways from this story, one stands out. The ability to create hacks or tricks that can impact the bottom line need someone with an IT background. Put into a bigger context, digital technology is now more important and the people who know how to leverage them will become more influential and important to companies. On one side of the coin, they can help fool systems to hide a deficiency in a product. However on the other, they can expose tricks and frauds similar to what these two researchers did with Dieselgate.
Tips