Why AIs Will Become Hackers

RSA CONFERENCE 2022 — “Nice to see you all again,” Bruce Schneier told the audience at his keynote for the in-person return of RSA Conference, taking off his trademark cap. “It’s kinda neat. Kinda a little scary.” 

Schneier is a security technologist, researcher, and lecturer at Harvard Kennedy School. He has a long list of publications, including books from as early as 1993 and as recent as 2019’s We Have Root, with a new one launching in January 2023. But he’s best known for his long-running newsletter Crypto-Gram and blog Schneier on Security. And his upcoming book is about hacking.

To Schneier, hacking does not necessarily mean computer systems. “Think about the tax code,” he said. “It’s not computer code, but it’s code. It’s a series of algorithms with inputs and outputs.”

Because the tax code is a system, it can be hacked, Schneier said. “The tax code has vulnerabilities. We call them tax loopholes. The tax code has exploits. We call them tax avoidance strategies. And there’s an entire industry of black-hat hackers — we call them tax accountants and tax attorneys,” he added, to audience laughter.

He defined hacking as “a clever, unintended exploitation of a system, which subverts the rules of the system at the expense of some other part of the system.” He noted that any system can be hacked, from the tax code to professional hockey, where a player — it’s contested just who — started using a curved stick to improve their ability to lift the puck. That player hacked the hockey system.

“Even the best-thought-out sets of rules will be incomplete or inconsistent,” Schneier said. “It’ll have ambiguity. It’ll have things the designers haven’t thought of. And as long as there are people who want to subvert the goals of the system, there will be hacks. 

“What I want to talk about here is what happens when AIs start hacking.”

Rise of the Machines

When AIs start hacking human systems, Schneier said, the impact will be something completely new. 

“It won’t just be a difference in degree but a difference in kind, and it’ll culminate in AI systems hacking other AI systems and us humans being collateral damage,” he said, then paused. “So that’s a bit of hyperbole, probably my…