While Augmented Reality (AR) and Virtual Reality (VR) are envisioned as the next iteration of the internet immersing us in new digital worlds, the associated headset hardware and virtual keyboard interfaces create new opportunities for hackers.

Such are the findings of computer scientists at the University of California, Riverside, which are detailed in two papers to be presented this week at the annual Usenix Security Symposium in Anaheim, a leading national conference on cyber security. 

The emerging metaverse technology, now under intensive development by Facebook’s Mark Zuckerberg and other tech titans, relies on headsets that interpret our bodily motions —  reaches, nods, steps, and blinks — to navigate new worlds of AR and VR to play games, socialize, meet co-workers, and perhaps shop or conduct other forms of business. 

A computer science team at UCR’s Bourns College of Engineering led by professors Jiasi Chen and Nael Abu-Ghazaleh, however, has demonstrated that spyware can watch and record our every motion and then use artificial intelligence to translate those movements into words with 90 percent or better accuracy.

“Basically, we show that if you run multiple applications, and one of them is malicious, it can spy on the other applications,” Abu-Ghazaleh said. “It can spy on the environment around you, for example showing people are around you and how far they are. And it can also expose to the attacker your interactions with the headset.”

For instance, if you take a break from a virtual game to check your Facebook messages by air typing the password on a virtual keyboard generated by the headset, the spyware could capture your password. Similarly, spies could potentially interpret your body movements to gain access to your actions during a virtual meeting in which confidential information is disclosed and discussed.

The two papers to be presented at the cybersecurity conference are co-authored Abu-Ghazaleh and Chen toether with Yicheng Zhang, a UCR computer science doctoral student, and Carter Slocum, a visiting Assistant Professor at Harvey Mudd College who earned his docorate at…