Toyota Prius and Camry, drive-by-wire, and our failure to learn from experience...

I see from the morning news that Toyota's adventure into the world of embedded software is going badly. The company's second attempt to find a quick fix for unintended acceleration in its conventionally-powered vehicles is barely underway, and already evidence is emerging that the underlying problem is likely in the engine controller, not in the pedal mechanical assembly. And now we hear from Japan that the Prius, Toyota's golden child, has a problem with its brake-by-wire control system.

One has to recall Audi, which decades ago accidentally introduced drive-by-wire with its advanced cruise control on the Audi 5000. The cars were allegedly subject to spontaneous acceleration. The company blamed the problem on operator error. At the time, I was told that researchers at another European high-end auto company had uncovered a problem in Audi's engine-control firmware and reproduced the acceleration without requiring a driver to mistake the gas pedal for the brake. But in the ensuing liability litigation, all hope was lost of diagnosing the actual problem and documenting it so that the rest of the real-time software community could avoid it.

The reason all this came to mind this morning was actually not the newspapers, but a panel I attended yesterday at DesignCon. The subject was achieving quality closure. But the issue of software sat like an elephant in the corner of the room, awaiting notice. One of the panelists—I believe it was Design Rivers president Camille Kokozaki—pointed out that perhaps the most serious quality problem in IC designs now is not quality closure on the hardware, but the integrity of the firmware and software that will run on the chip. There simply is no systematic approach to ensuring the quality of an integrated hardware/software system.

And this is a tragedy. Thirty years ago, work was well under way on the problem of formally proving software correctness. One company had designed a completely deterministic microprocessor—no interrupts, no indirect addressing—that made it possible to mathematically prove all of the possible trajectories of a code set. And computer scientists such as Edsger Dijkstra were making strides in methodology to create formally proven software. But along came C, UNIX, and the cult of the bemused hobby programmer, and the entire notion of formal correctness vanished under a smokescreen of hacking.

So now, after decades invested in metrics-driven verification, formal verification, and methodology management, we find that our chips don't work as expected because the software is still being "verified" by feeding it test cases until the schedule expires. And we find that our cars run into things for the same reason, and the press of course will blame the problem on "electronics."

And once again, as in Audi's day, it is safe to conclude that whatever accurate diagnostic work actually gets done on the Toyota problems will be wrapped up in a gag order as part of a class-action settlement, so that no one in the industry can benefit from what Toyota engineers did or did not learn from the problem. That way we can repeat the situation with the next generation of software-governed systems, a new set of executives can avoid blame for the tragedies, and a new set of lawyers can make their fortunes off the resulting litigation.

The only parties in this little comedy that have an interest in actually improving the state of the art are the engineers, who won't be consulted, and the victims, who will be silenced by the lawyers. How much better for everyone if it were a principle of civil law that when it is found that damage has been inflicted by a failure, all of the diagnostic information determined by the vendor and by independent parties must be placed in the public domain, and may not be used to assess or assign damages. Such a notion might somewhat restrict the income opportunities of litigators, but it would unquestionably assist the engineering community in learning from our mistakes.
Related entries in: Embedded Systems | SOC (System on a chip) | Verification |