The father of cybernetics cautioned human beings against the desire to be waited upon by intelligent machines that are equipped to improve their minds over time. “We wish a slave to be intelligent, to be able to assist us in the carrying out of our tasks,” Wiener writes. “However, we also wish him to be subservient.” The obvious problem is that keen intelligence and groveling submission do not go hand in hand.
‘Westworld’ and the Moral Dilemma of Cyborgs
by Brent Staples
http://www.nytimes.com/2016/11/20/opinion/sunday/westworld-and-the-moral-dilemma-of-cyborgs.html
The mathematician Norbert Wiener founded the science of cybernetics in 1948 with a warning that has become a cornerstone of futurist writing. The computer age was still in its infancy when he cautioned that intelligent machines could wreak havoc on civilization — and perhaps even snuff it out — if we failed to handle them with humility and care.
He was not referring to malevolent entities like Skynet, the genocidal artificial intelligence in the “Terminator” films, or the Cylon robots who wage war on their makers in the television series “Battlestar Galactica.” Instead, he posed a scenario in which a relentlessly logical machine brings on catastrophe by obeying instructions containing hidden hazards that its master failed to detect. The machine would complete its work with lightning speed, Wiener said, giving comparatively slow-witted humans no time to appraise the danger and react.
Wiener understood that people who owned intelligent or sentient machines would face moral problems similar to those associated with the practice of owning human beings. The futurist writer Philip K. Dick exploited this problem to great effect in his work, most famously in “Do Androids Dream of Electric Sheep?,” the novel that inspired the 1982 film “Blade Runner.” This movie crystallized what futurists like Dick had long seen as the next phase of enslavement. It depicted synthetic beings called replicants, manufactured for use as sex toys, soldiers, assassins and so on. When the replicants take charge of their lives, a bounty hunter tracks them down and “retires” them — which is to say, he shoots them to death.
The new HBO series “Westworld,” though based on a 1973 sci-fi thriller about a rebellion of robotic slaves at an amusement park, is closely related to “Blade Runner” in both tone and substance. It acknowledges this debt in a number of ways, not least of all by using the word “retire” to describe the park’s decision to lobotomize a cyborg prostitute who fights back when she is assaulted, despite programming that requires her to submit to rape, strangulation, gunshot wounds and temporary death. Under ordinary circumstances, cyborgs who “die” are surgically repaired, subjected to memory erasure and put back into circulation. The next day, hell begins all over again.
“Westworld” has been attacked for sensationalizing violence. But the violence in this series is tame compared with the carnage depicted in any number of video games that show characters being blown to pieces in every frame. Now imagine what gamers decades from now might pay to enter into a world where a quite-nearly-human adversary bleeds, cries and “dies’’ when injured in battle. This is the bloody, morally compromised future that “Westworld” envisions.
The father of cybernetics cautioned human beings against the desire to be waited upon by intelligent machines that are equipped to improve their minds over time. “We wish a slave to be intelligent, to be able to assist us in the carrying out of our tasks,” Wiener writes. “However, we also wish him to be subservient.” The obvious problem is that keen intelligence and groveling submission do not go hand in hand.
The humans who work at “Westworld” get a sense of what that means when a cyborg learns that her bloody nightmares are not dreams but a representation of the world as it is. The resulting conflict is one that Wiener would clearly have viewed with fascination.