Since the 1960s, one of the key developments in making cars safer has been the idea of a “crumple zone”, a part of the vehicle (usually the front) that is designed to absorb the energy from a collision by controlled deformation, ie crumpling. By doing this, the zone protects the most important part of the car – the cell containing the driver and passengers. (If you doubt the effectiveness of crumple zones, by the way, a video made some years ago by the Fifth Gear TV programme makes interesting viewing.)
As the world moves to vehicles that drive themselves, our roads are likely to become safer because most road accidents are caused by driver error. That, at any rate, is the standard spiel of evangelists for autonomous vehicles. There will, they say, be many fewer collisions and therefore less carnage on our roads. Crumple zones will become less necessary, though autonomous vehicles will continue to need them.
For some time to come, however, the new crumple zones won’t look anything like the ones built into today’s Volkswagens, Toyotas and Fords. In fact, they will be indistinguishable from you and me because they will be humans. This is the argument advanced in Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction, a fascinating academic paper by Madeleine Clare Elish, an anthropologist who works at the Data & Society institute in New York. She had been reading media coverage of self-driving cars over the last few years and noting that the prevailing narrative was, and remains, consistent in one respect: on the (rare) occasions when a self-driving car has been involved in an accident it is invariably the fault in some way of human drivers rather than of the machine itself.
A classic example is the case of an accident in March last year when an Uber self-driving car struck and killed a pedestrian who was wheeling her bike across a highway in Arizona. At the steering wheel of the supposedly autonomous vehicle was a safety driver whose job was to monitor the car’s systems and take over in the event of an emergency. The safety driver may now face criminal charges of vehicular manslaughter. This tragic accident brought to the fore the question that has been on everyone’s mind – or at least on the minds of tech company lawyers – for the last few years: if a driverless car kills someone, who or what is to blame?
This particular case is relatively recent but it’s just the latest example of an older question, namely: what is the responsibility of a human who is obliged to work in a complex, interactive system designed by others? To illustrate this, Dr Elish looks at two celebrated earlier cases: the partial meltdown of the Three Mile Island nuclear reactor in Pennsylvania in March 1979; and the case of Air France Flight 447 in 2009, in which an Airbus jet with 228 people on board crashed into the Atlantic, killing everyone on board.
Both cases involved human operators and pilots struggling to mitigate disasters caused by the malfunctioning of complex and inscrutable technical systems. But in both cases the prevailing tone of media (and some “expert”) commentary was that human error – rather than design failures – provided the main explanation for what happened. “The explicit point,” writes Elish, “is that the autopilot and associated automation are smart enough to outsmart and save the human every time, the same narrative we saw in nuclear power plant design. The idea that the automation and its software could ever fail was never a possibility.”
This mindset prompts Dr Elish to coin the term “moral crumple zone” to describe the role assigned to humans who find themselves in the positions that the Three Mile Island operators, the Air France pilots – and the safety driver in the Uber car – occupied. It describes how responsibility for an action may be wrongly attributed to a human being who had limited control over the behaviour of an automated or autonomous system.
“While the crumple zone in a car is meant to protect the human driver,” she writes, “the moral crumple zone protects the integrity of the technological system, at the expense of the nearest human operator. What is unique about the concept of a moral crumple zone is that it highlights how structural features of a system and the media’s portrayal of accidents may inadvertently take advantage of human operators (and their tendency to become “liability sponges”) to fill the gaps in accountability that may arise in the context of new and complex systems.”
This all aligns nicely with one of the most pernicious aspects of our time: the subliminal tendency to assign more credibility to supposedly “smart” machines than to humans. In the meantime, I can see corporate lawyers at the big tech companies salivating at the idea of a ready supply of liability sponges and wondering why they hadn’t thought of it before.
What I’m reading
Breaking Vlad
Russia is planning an experiment: to cut itself off from the global internet. It might be more difficult than Putin realises, reports the MIT Technology Review.
Beyond the pale
Stanford University has a new institute to look into the challenges of AI and show its commitment to addressing concerns over tech’s lack of diversity. Guess what, Quartz reports: of 121 faculty members associated with the venture, 100 appeared to be white and the majority were male!
News unworthy
The Columbia Journalism Review surveys how mainstream media covered the Christchurch killings. Hint: the UK’s media outlets didn’t cover themselves in glory.
