Generative AI In Game Development (original) (raw)

In this blog post, I provide an overview of how Generative AI (GenAI) technology is impacting different areas of game development. By no means, this blog post is meant to cover every work on GenAI in gaming. I will point to some solutions that I am familiar with but there are many more out there.

Somehow, I have created a kind of taxonomy to differentiate the different areas in game development where GenAI is impacting:

I will also mention how GenAI is being used to manage game communities, probably something that is not well known. I will also briefly mention some of the latest works in GenAI that potentially can move from the academy to game studios.

The AI revolution in gaming

The introduction of Machine Learning and Generative AI in gaming has sparked an authentic revolution in graphics and gaming, comparable to the launch of the first programmable GPU GForce2 by NVIDIA in 2000. Just as GPUs unlocked real-time rendering and complex physics in gaming, GenAI is now enabling automation, efficiency, high quality graphics and unprecedented creative possibilities.

I was lucky to attend SIGGRAPH 2018 when NVIDIA presented DLSS and neural ray-denoising for the first time. Also, I was in the Augmented World Expo when Adobe presented their ML-based solution for animation, showcasing the animation of a wolf running in the forest. We all know how difficult is to animate characters, and animating quadrupeds is even more challenging. That was just an anticipation of all that was about to come in gaming and graphics.

In general, what we can see already today from the use of GenAI in gaming is:

GenAI in game design

GenAI can help us from the very beginning when we are brainstorming for game ideas, by suggesting new features and game mechanics based on latest trends and player preferences.

GenAI can also provide great help in creating the game story and dialogs, and storyboards, by developing storylines, branching the narratives and creating characters and interactions between them.

As an example, I would like to mention the work from Riot Games they shared last year at London AI and Games event. They have got a project called Narrative Driven Generation. They used a LLM to generate from the scratch a Murder Mystery Game, from game settings and actions to story, the game world, characters and dialogs. Riot Games diagramFigure 3. Picture from Riot Games' talk at AI and Games conference in London, 2024.

The most amazing thing here is that any of us can make use of a LLM to start designing a game as detailed as we want.

We can start by brainstorming with ChatGPT our vision, the type of game we would like to develop, for example, a fantasy adventure game, and then refine the initial idea by exploring different worlds and themes.


Figure 4. Pipeline showcasing main steps of brainstorming with a LLM on designing a game.

Next, we can ask for help to develop the story and the characters: “Can you help me outline a storyline for a game where the hero unravels ancient mysteries? “. Get suggestions for protagonists and antagonists and supporting characters, including their motivations and backgrounds.

We can then move to the gameplay mechanics: “_Can you suggest engaging mechanics I can include in an adventure game?_”. We brainstorm with the AI about ideas for levels.

At this point we can create a Game Design Document and move to the technical implementation and prototyping, get coding help by asking for sample code for core features and enquire about game engines that suit our project.

Going through a continuous cycle of iteration and refinement, we can save a lot of time and speed up the design process.

GenAI in game development

Looking at GenAI in game development, it is perhaps where we see more intensive use of GenAI today, during the different stages of the game development pipeline.

GenAI is already helping in automated content creation in games engines. GenAI can generate game assets like levels, maps, characters, and environments based on predefined rules and parameters. This allows designers to quickly generate unique and diverse content without having to create each element manually. For example, AI might generate a new level layout for a platformer game or even create a procedurally generated world in an open-world game.

GenAI is also contributing to generating textures, 3D models, and animations that fit the aesthetic of a game, saving time for artists and designers.

In coding AI is providing great help in programming and scripting. From code completion, error detection and code optimization to generation of full scripts for game engines.

Testing is a crucial part in game development. AI helps in automating playtesting, bug detection, performance monitoring, and user behavior simulations to streamline Quality Assurance processes.

All major game engines are incorporating AI in different stages of development pipeline.

GenAI in major game engines

Different game engines are intensively incorporating GenAI in their game development pipeline. It includes coding, animation, texturing, troubleshooting problems, asset creation like 3D objects, music, and more.

In the Unity Project Muse, we can mention:

Roblox:

Unreal integrates AI through different plugins:

The picture in the slide was taken during a Roblox talk at GDC 2023 where they shared a high-level view of their GenAI roadmap, from texture to 3D assets generation, to the most ambitious full scene generation. So far, they have shown a consistent progress in their roadmap. Roblox GenAIFigure 5. Roblox's GenAI roadmap shared at GDC 2023

There are other independent AI animation tools which are impressive, capable of extracting animation information from any video clip, allowing us to easily build an animation library for our games (see Kinetix and DeepMotion). Others, like SayMotion, generate animations from a text prompt.

GenAI at Game Runtime

Here we can mention several well-known ML-based solutions to improve graphics rendering like NVIDIA DLSS (Deep Learning Super-Sampling) and NVIDIA Optix Ray Denoising technology and Frame Rate Upscaling. AMD FSR 4 (Fidelity Super Resolution) powered by AI has been recently released.

Regarding other applications different from graphics enhancement, the use of GenAI is just starting. We can mention AI-Dungeon’s scripts for crafting custom adventures. Inworld have also developed solutions for AI-driven NPCs.

A new way of interacting with game NPCs is presented in the work “Verbal interaction with NPCs” driven by a language model. It is a project I personally led and was successfully presented last year at SIGGRAPH in Denver, demo included (see Poster and Appy Hour). For the first time, thanks to language models we have a new, intuitive and natural way of interfacing in games. Example of an NPC in a mobile gameFigure 6. Screenshot from demo Verbal Interaction with NPC on Mobile Games, presented at SIGGRAPH 2024.

AI can transform static, predictable character behaviour into dynamic, learning entities. From sophisticated decision-making and adaptive strategies to realistic navigation and engaging dialogue, these technologies create a more immersive and challenging gaming experience. As AI continues to evolve, we can expect even more lifelike and responsive NPCs that push the boundaries of interactive storytelling and gameplay design.

I would like also to mention in this section the latest Arm technology KleidiAI to enable the inference of LLMs with great performance on mobile devices powered by Arm CPUs. When integrated within the game engine, this allows to use LLMs at runtime for different use-cases. For example, verbal interaction, game assistance, runtime strategy, and more.

KleidiAI is an open-source library that provides highly optimized and performant functions, known as micro-kernels, for AI workloads tailored for Arm CPUs.

GenAI in Game Community Management

The application of AI in game communities is not well-known. Game communities are vital for game developers and players. So, GenAI has come also to improve and optimize the management of game communities and improve safety and content moderation.

An example is Roblox’s AI moderation tool Advance Group Wall Filter, that helps detect harmful content and facilitate community interactions.

Their AI-Powered Real-Time Chat Translation tool helps overcoming language barriers between users of different nationalities.

Discord use AI bots to optimize player engagement.

AI also assists in connecting creators with developers for asset exchanges, optimizing this way asset marketplace in game communities.

Recent Developments in GenAI with Potential Impact on Gaming

Looking at recent developments in GenAI, there are many works with potential impact on gaming. From my personal experience attending game and graphics related conferences I have witnessed a substantial increment of papers on the use of GenAI for 3D assets generation, avatars, high quality faces, environments, including realistic cities, editing of generated images and geometry, video overlying, and many more topics relevant for game development.

Things like hair and cloth simulation which are very difficult to render, are getting better. GenAI is impacting animation and lighting, which require great experience to get correctly.

We see also emerging new rendering paradigms like Gaussian Splatting and Neural Radiance Fields finding their ways within game developers, avid for new rendering technologies.

I would like to highlight here a kind of emerging trend. The time from papers to practical tools for game development is shortening. We can see how after short time, original papers presented in conferences see their implementations shared in Hugging Face or in the authors’ code repository. Everybody is hungry to test new models and tolls.

Conclusions and future outlook

Finally, I would like to share a few thoughts as conclusions which are summarized in the picture below.

GenAI diagramFirst, the fact that AI has triggered an authentic revolution in gaming, enabling automation of repetitive tasks like animation, texturing, 3D modeling. This is speeding up developments and helping to reduce costs.

Second, the use of GenAI opens many new opportunities in gaming at runtime, more realistic and smart characters, infinite worlds capable of auto-evolving, new type of interaction with NPCs based on speech, and many other new features that can be implemented at runtime, where the real fun is. Ultimately, it is the player and the gaming experience, all game development is for.

Finally, I would like to highlight the fact that we see the time from papers to implementation shortening each year. So, it is very difficult to make predictions, but I do not see far the moment when we will be able to generate whole complex scenes with all details, 3D assets, characters, animations, visual effects, lighting, everything that usually takes weeks and months of work, all just from a prompt description.

It will be interesting to see the use of new reasoning models in gaming, where the strategy is important. Also, AI can be very useful in the Metaverse for building avatars and virtual environments.

You can get mor detailed information on Arm KleidiAI technology here. There is also a complete section of Learning Paths devoted to AI here, that provides practical examples on how to use KleidiAI to accelerate GenAI workload and run LLMs on different hardware backends.