In Search of the 'Right' Sound: Why Games Don't Generate Audio Like Graphics

Originally published on LinkedIn on December 16, 2024 (original article). Republished here unchanged for easy access.

In my previous article, I considered the question, ‘When is the “wrong” sound the “right” choice?’ The answer we arrived at was simple: when the wrong sound tells a better story than the right one. This realization opens up deeper questions: Could we create the ‘right’ sound? How would we do that? Why don’t all games use the ‘right’ sound all the time? And can the ‘right’ sound tell a better story than the ‘wrong’ one?

The previous article: When is the Wrong Sound the Right Choice?

OK. Maybe these questions are a bit too ambitious for one article, but I think we can do it. I’ll admit upfront that my use of “right” and “wrong” is deliberately provocative - but let’s define what we really mean by these terms.

Defining Terms

For these explorations, ‘right’ sound represents acoustic authenticity—the unaffected, physically accurate sound that would emerge if the digital event were to occur in the physical world. Imagine, for instance, a wooden chair collapsing: the ‘right’ sound would capture every nuanced creak, stress, and final splintering moment exactly as physics would dictate. The ‘wrong’ sound, by contrast, might be a pre-recorded sample that emphasizes certain aspects while minimizing others, or even a completely synthetic sound designed for dramatic effect rather than accuracy.

The Current State of Game Audio

Anyone who still thinks there is a one-to-one relationship between a game event and a wave file just doesn’t understand game audio – Brian Schmidt

Game audio today almost exclusively relies upon pre-rendered audio files—carefully curated and triggered by in-game events. When a door creaks open, developers deploy a meticulously selected creak. When a character’s foot contacts the ground, a precisely designed “footstep” audio file plays, calibrated to match the character’s weight, footwear, and surface texture. Advanced techniques like layering, dynamic filtering, and parameter randomization help maintain an often convincing veneer of realism, but fundamentally, this approach remains an intricate orchestration of pre-rendered audio snapshots.

Take this implementation demo by Joshua Oosterhuis as a fantastic example. For the tech heads among you, Oosterhuis provides a full walkthrough of the system he created using Unreal Engine & Wwise that tracks the physics-based interactions of objects and interpolates that data to control the playback of a curated list of pre-made audio files.

Like Oosterhuis’s implementation, most modern games use complex control systems that manage vast libraries of audio files, each carefully matched to specific environmental conditions. This approach goes by several names—system-based, data-driven, or procedural audio—reflecting how the sense of realism comes not from how the sounds are created, but from the sophisticated systems controlling when and how they’re played back.

The power of this approach is beautifully demonstrated in Playdead’s game Inside, where the audio team created an intricate breath control system that responds dynamically to the player’s actions. In the video below (starting at 9:03), you can hear how the system adjusts the protagonist’s breathing patterns based on their movement, stress levels, and environmental conditions, creating an incredibly immersive experience. For those interested in the technical details, the audio team’s technical postmortems provide fascinating insights into how they achieved this level of sophistication

These sophisticated audio systems continue to evolve, as detailed in Ronny Mraz’s comprehensive overview of procedural audio in games like No Man’s Sky, Just Cause 4, and Grand Theft Auto 5.

Ronny Mraz, How procedural audio brings sounds to life in video games, Splice.

However, pursuing ever more complex sample-based approaches leads us to an intriguing paradox, one that recalls Jorge Luis Borges’ thought-provoking short story “On Exactitude in Science.” Just as Borges’ cartographers futilely attempt to create a map mirroring the world point-for-point, audio teams find themselves chasing an ultimately unmanageable goal: pre-recording and categorizing sounds for every conceivable in-game scenario—a library that would need to be as vast as the possibilities of play itself. Yet while audio designers grapple with this seemingly insurmountable challenge, another field of game development has already found a different path entirely.

A thousand pictures for a single word

The visual side of game development has already solved a remarkably similar challenge. Modern game engines don’t rely on vast libraries of pre-captured images—they don’t need to store every possible camera angle or lighting condition. Instead, they dynamically simulate entire visual environments, modeling objects, materials, and lighting conditions to render scenes in real time. The engine calculates exactly how light should bounce, how shadows should fall, and how surfaces should reflect or absorb light, all happening instantaneously as the player moves through the world.

The latest Unreal Engine 5 tech demo demonstrates just how far this technology has come:

This stark contrast with audio development raises an intriguing possibility: What if we could apply the same computational approach to sound? Instead of relying on vast libraries of samples, could we create a system that generates sound dynamically—one that’s acoustically accurate, contextually responsive, and computationally derived?

The answer, surprisingly, is that we already can. We have the technology to generate physically accurate sounds in real time. Yet despite this capability, most games still rely primarily on sample-based audio. The first steps towards integrating computational audio into the sonic side of games is already happening through acoustic modeling—technology that simulates how sound waves interact with virtual environments just as they would in the physical world. These early implementations offer a glimpse of what might be possible if we approached all game audio with the same computational mindset we apply to graphics.

Acoustic Modeling

Today’s games typically handle audio spatialization through a computationally efficient but simplified approach. This basic method relies on just two key data points: the locations of the sound source and the listener within the virtual space. Using this information, the audio system makes two primary adjustments—it modifies the sound’s volume based on distance and distributes the audio between left and right speakers based on the listener’s orientation.

This rudimentary approach to spatializing audio more often than not succeeds in creating convincing sonic environments that give players all the crucial information they need—whether an enemy is approaching from behind, to the left or right, or getting closer.

Recommended by LinkedIn

For a more thorough investigation of sound localization and spatialization techniques see Chapter 2 of my dissertation “Compositional Possibilities of New Interactive and Immersive Digital Formats”

However, this approach overlooks a crucial aspect of how sound behaves in the real world: the environment itself. While a player can tell where a sound is coming from, they receive no information about the space those sounds exist within.

This is where acoustic modeling enters the picture. At its core, acoustic modeling simulates how sound waves bounce off surfaces and interact with the environment. Just as real sound waves reflect off walls, get absorbed by soft materials, and create reverberations in large spaces, acoustic modeling calculates these behaviors in real-time within the game world. When a gunshot echoes through a concrete tunnel, or footsteps sound different moving from a carpeted room to a tiled bathroom, that’s acoustic modeling at work. The system analyzes the virtual materials, geometry, and space to determine how each sound should be transformed as it travels from its source to the player’s ears.

The growing importance of realistic acoustic modeling has become particularly evident with the surge in virtual, augmented, and mixed reality experiences driven by companies like Meta and Apple. These new platforms have sparked the development of numerous tools and approaches aimed at creating more authentic spatial audio experiences.

For a deeper dive into practical implementation, Robert Bantin’s GDC 2024 talk Finding Space for Sound demonstrates how recent games have developed systems for modeling environment-specific acoustics.

But if acoustic modeling can give us the ‘right’ sound for virtual spaces, could we apply similar computational approaches to generate the ‘right’ sound for every virtual object and interaction?

Physical Modeling

Having seen how acoustic modeling can simulate the way sounds interact with virtual environments, we can take this computational approach one step further. Physical modeling for audio aims to generate the sounds themselves through mathematical equations and physics-based simulations, eliminating the need for pre-recorded samples entirely.

Consider a door in a game. With physical modeling, instead of playing back a pre-recorded creaking sound, the system would calculate the sound waves in real time based on the door’s material properties, hinge friction, weight, and how the player interacts with it. Like a real door, no two openings would sound exactly alike—a gentle push produces a different sound than flinging it open. Just as modern game engines calculate how light bounces off surfaces to create dynamic visuals, physical modeling calculates how objects vibrate and interact to create dynamic sound.

The video below demonstrates the remarkable potential of physical modeling. While this is a technical demonstration from researchers rather than a game implementation, it shows just how interactive and realistic physically modeled sounds can be:

This approach might seem cutting-edge for games, but physical modeling has actually existed since the 1980s, particularly in music technology. Digital instruments like Pianoteq use physical modeling to recreate remarkably realistic pianos, capturing nuances of expression that sample-based approaches struggle to achieve. Here’s an explanation of how this technology brings unprecedented accuracy to digital instruments:

Yet despite these capabilities, physical modeling remains largely on the fringes of both musical and gaming applications. While experimental VR experiences and academic projects have shown promising results with simple interactions—like balls rolling or floorboards creaking—the technology hasn’t been widely adopted by mainstream games. What’s holding physical modeling back from broader adoption in interactive media?

The Barriers to Adoption

The obstacles preventing widespread adoption of physical modeling in games aren’t primarily technical—the technology exists and works. Instead, the challenges fall into three main categories: computational cost, implementation complexity, and creative control.

Most significantly, physical modeling requires substantial processing power. While a single physically modeled door might not tax a system, imagine modeling every sound source in a complex game environment—each footstep, every breaking object, all environmental sounds. Each instance requires real-time physics calculations, potentially competing with other game systems for computational resources. In an industry where developers already struggle to balance visual fidelity with performance targets, dedicating significant, potentially unpredictable CPU power to sound generation is a hard sell.

Implementation presents another hurdle. Unlike traditional sample-based audio, where sound designers can directly shape and control the exact sound players will hear, physical modeling requires a deep understanding of acoustics and physics. Sound designers must think in terms of material properties, physical behaviors, and mathematical models rather than waveforms and mixing techniques. This represents not just a technical challenge, but a fundamental shift in how audio content is created and integrated into games.

Perhaps most crucially, physical modeling can feel creatively restrictive. While it excels at creating realistic sounds, it can be challenging to achieve stylized or exaggerated effects—the kind of larger-than-life sounds that make games more exciting and engaging. Remember our earlier discussion about when the ‘wrong’ sound is actually right? Physical modeling, by its nature, pushes toward acoustic accuracy, which isn’t always what serves the game best.

Consider a punch in a fighting game. The realistic sound of fist meeting flesh might be underwhelming compared to the exaggerated impact players expect. Or think about a sci-fi weapon—how do you physically model something that doesn’t exist in the real world? While workarounds exist, they often negate the supposed benefits of using physical modeling in the first place.

This explains why we most commonly see physical modeling used in specific, contained scenarios—like the piano simulations mentioned earlier—where the benefits of realistic behavior outweigh the costs and limitations. For broader game audio, a hybrid approach often makes more sense: using physical modeling where it truly adds value while relying on traditional techniques where they work better.

Looking Forward

At the start of this exploration, we asked when the ‘wrong’ sound might be the right choice, which led us to examine how games could potentially generate the ‘right’ sound through computational approaches. What we’ve discovered is a fascinating tension between technological capability and practical reality. While we have the tools to create physically accurate, computationally generated sound—through both acoustic and physical modeling—the practical and creative constraints of game development often push us back toward traditional approaches.

This isn’t necessarily a shortcoming. Just as game visuals often prioritize artistic vision over photorealism, game audio must balance acoustic accuracy with emotional impact. The ‘right’ sound, it turns out, isn’t always the most physically accurate one. Sometimes it’s the sound that best serves the game’s narrative, enhances player feedback, or creates a more compelling experience.

As processing power increases and tools become more sophisticated, we may see greater adoption of computational audio approaches. But rather than replacing traditional sound design techniques, these new technologies will probably become another tool in the sound designer’s arsenal—used when they serve the game’s needs best, and set aside when other approaches work better.

After all, the goal isn’t to perfectly simulate reality—it’s to create memorable and meaningful player experiences. Whether that’s achieved through physical modeling, pre-recorded samples, or a combination of both depends not on what’s technically possible, but on what best serves the game.