Google's latest AI agents are learning how to navigate the familiar space of gaming.
The tech giant announced new research on Scalable Instructable Multiworld Agents (SIMA) on Wednesday. This agent can follow instructions to perform video game tasks and play games never seen before.
But like Genie, which Google's AI research arm DeepMind discussed in a research paper published on February 23, SIMA is a research project.
“In the future, you might have agents like SIMA playing with you,” said Tim Hurley, a research engineer at DeepMind who co-led the project. “A collaborative agent who can talk and give instructions about various things in the game on the spot.”
DeepMind says one of the reasons it's interested in video games is because they're a great training ground for AI systems. AI companies hope that such research will help them “understand how AI systems can be more useful.”
Since OpenAI released ChatGPT in November 2022, the market has been flooded with generative AI tools from Microsoft, Google, Adobe, Meta, and Anthropic. More recently, generative AI has expanded beyond text to include images, video, music, and, of course, games, as tech companies seek to differentiate their products in the fast-growing sector.
Research goal
According to Hurley, SIMA is trained to do what it's told, but that doesn't necessarily mean it will win.
The researchers' primary question initially was whether the AI agent could transfer skills between games, and how it would behave in games it had never played before.
“Those goals are provided by human users in unrestricted free-form natural language, and then [SIMA] “These video game environments just use a natural interface to the games,” Hurley said, “and the only way agents can observe these games is from a screen in real time.” said.
training
Researchers recorded images and keyboard and mouse inputs of human players and used imitation learning techniques to teach SIMA to play games like No Man's Sky, Eco, Teardown, and Goat Simulator just like humans. I taught you to do it.
They evaluated agents on 600 skills, including navigation (e.g., “turn left”), object interaction (e.g., “climb a ladder”), and menu use (e.g., “open map”), and found that SIMA's performance I found it to be better than the specialist.
“He can leverage concepts shared between games to learn better skills and learn how to better execute those instructions,” said Frederic Besse, research engineer at DeepMind. Masu. “Confirming positive transfer between matches is an important milestone for the study.”
However, SIMA is not perfect.
“All the mistakes we're seeing are related to a more detailed understanding,” Hurley said. “So if you ask an agent to cut down a tree in the Valheim game, the agent will go and cut down the tree, but you can't specify exactly which tree to cut down.”
He is reluctant to call SIMA's imperfections an “illusion.”
“What we see when an agent fails is not an illusion. The agent's behavior often appears intentional, but it is unable to perform the desired behavior,” he added. Ta.
“It’s a great practice area.”
Going forward, DeepMind plans to improve the performance of SIMA, including allowing agents to follow more detailed instructions, ultimately allowing them to “operate in as many environments as possible, achieve a variety of goals, and interact with users.” We would like to develop an AI system that can ” Besse said.
But it's not just human-agent communication in games.
“We believe that games and simulations in general are great training grounds for AI systems,” Besse said.
One reason for this is that games approximate the real world. They offer visual variety, along with a variety of settings, mechanics, and graphical styles. But they also share common themes, such as navigating complex spaces and interacting with objects, characters, and players.
