Microsoft Unveils Muse: A Generative AI Model Transforming Game Development

Microsoft Research has introduced “Muse,” a generative AI model designed to support game development by generating both game visuals and controller actions. The announcement accompanies a publication in Nature detailing the model’s development and capabilities.

Muse’s Work in Gaming

Muse is part of the World and Human Action Model (WHAM) framework and was created through a collaboration between Microsoft Research’s Game Intelligence and Teachable AI Experiences teams, along with Xbox Game Studios’ Ninja Theory. The model is trained on gameplay data from Bleeding Edge, a 2020 multiplayer game developed by Ninja Theory.

The release marks a significant step toward integrating generative AI into game design. Muse can simulate gameplay sequences by generating visual game environments and player actions based on human guidance. This allows developers to explore new creative ideas and streamline their prototyping process.

To encourage further research and development, Microsoft is open-sourcing Muse’s model weights, and sample data, and providing access to the WHAM Demonstrator — a tool offering a visual interface for interacting with Muse. These resources are available through Azure AI Foundry.

Key contributors to the project emphasize the model’s potential impact. “It’s been amazing to see the variety of ways Microsoft Research has used the Bleeding Edge environment and data to explore novel techniques in a rapidly moving AI industry,” said Gavin Costello, Technical Director at Ninja Theory.

Development

The development of Muse was driven by advances in machine learning and the need to scale model training. Researchers initially used a V100 GPU cluster before transitioning to H100 GPUs, enabling the model to generate higher-resolution visuals (300×180 pixels) and operate across all seven maps in Bleeding Edge.

Evaluation of Muse focused on three primary capabilities: consistency, diversity, and persistence. Consistency ensures gameplay sequences adhere to in-game physics and rules. Diversity measures the model’s ability to produce varied gameplay outcomes from a single prompt.

Persistency allows the model to integrate user modifications, such as introducing new characters into a scene. The multidisciplinary approach was key to Muse’s development.

Researchers worked closely with game developers to align the model’s capabilities with creative needs. “This is why we invited game creators to help us shape this technology from the start,” said Linda Wen, a Design Researcher involved in the project. She added that the team prioritized diversity by involving creators from underrepresented backgrounds.

Muse in Action

A Microsoft internal hackathon led to the creation of the WHAM Demonstrator, enabling hands-on interaction with Muse. This tool allows developers to explore gameplay sequences and test ideas collaboratively.

Senior Researcher Tabish Rashid reflected on the process: “After months of experimentation, it was immensely rewarding to finally see outputs from the model on a different map… and not have to squint so much at smaller images.”

With Muse’s public release, Microsoft aims to inspire developers and researchers to push the boundaries of generative AI in gaming. The open-source model, data, and WHAM Demonstrator are expected to serve as valuable resources in shaping the future of AI-assisted game development.