Google DeepMind's SIMA AI agent can play never-before-seen video games

DeepMind, Google LLC's artificial intelligence research division, today revealed that it is experimenting with a new type of AI agent that can perform multiple types of tasks in 3D video games unlike anything you've ever played before.

The research organization has long been known for its work in gaming, building intelligent AI systems that can play against world champions in Go, Chess, and Stratego. We also built a model where people can learn how to play the game without being taught the rules.

Since then, DeepMind has focused specifically on video games, with its most powerful agents reportedly being at home in a variety of game worlds and able to perform numerous tasks based on instructions from humans.

DeepMind's research team worked with a number of game studios to train the Scalable Instructable Multiworld Agent, known as “SIMA,” on nine different games. Additionally, to power SIMA, we used four research environments, including one built with the 3D game engine Unity. SIMA therefore tasked you with forming sculptures from building blocks as a prelude to allowing you to learn how to adapt to different video game settings using different graphical styles and perspectives, such as first and third person. It was done.

“Each game in SIMA's portfolio opens up a new interactive world with a wide range of skills to learn, from simple navigation and menu use to mining resources, piloting spaceships and crafting helmets.” the researchers wrote in a blog post.

Researchers said SIMA's ability to follow instructions and complete tasks in a video game world could pave the way for more useful AI agents that can operate in real-world environments.

To teach SIMA, they first recorded humans playing video games and recorded the keyboard and mouse inputs used. This information was entered into SIMA. SIMA is based on accurate image language mapping and video models that allow you to view computer games being played on screen and predict what will happen next.

According to the researchers, SIMA can understand various gaming environments and perform almost any task asked of it. What's impressive is that SIMA doesn't require access to the game's source code. This is simply playing the commercial version of the game, which requires only two inputs: on-screen actions and instructions from a human user. Then play the game using the same inputs as humans: keyboard and mouse.

DeepMind's team evaluated SIMA's performance on hundreds of fundamental gaming skills across categories such as navigation, menu-based tasks, and object interaction. He then trained to play one game and used it as a baseline for performance to test his abilities before playing the same title.

According to the researchers, a SIMA agent that was initially trained on all nine games performed much better on one particular game than an agent that was trained on that same game only. This suggests that experience gained from playing other games can be leveraged to improve performance.

They also trained a SIMA agent on eight games and tested it on a ninth game, and it gave almost the same results as an agent they had never encountered before and was only tested on that one game. Ta. “This ability to function in an entirely new environment highlights SIMA's ability to generalize beyond training,” the researchers noted. “While this is a promising initial result, further research is needed before SIMA can perform at the human level in both visible and invisible games.”

The researchers found that SIMA requires a little bit of human guidance to work properly. According to the researchers, the SIMA agents, which received no language training or instructions, did not walk where they were told to go and only performed common actions such as gathering resources. “[It] “behave in an appropriate but aimless manner,” the researchers said of agents without such training or guidance.

DeepMind said the research shows there is potential in the idea of ​​developing “a new wave of generalist, language-driven AI agents.” Research suggests that when AI models are exposed to more training environments, they can become more versatile and generalizable.

Eventually, DeepMind plans to create agents that can perform more sophisticated multi-step tasks based on natural language prompts. So humans may eventually be able to direct agents playing games like Command and Conquer to gather resources, build bases and military forces, and go out to destroy their enemies. not. Currently, such tasks are too complex for SIMA agents.

“Ultimately, our research builds toward more general AI systems and agents that can understand and safely perform a wide range of tasks in ways that are useful to people online and in the real world.” DeepMind said.

Image: DeepMind