creating interactive music systems
i.e. my favorite thing about game scoring (and one of the many, many reasons why i like it way more than film scoring)
Opening this with a potentially controversial opinion! Film scores have more in common with concert music than they do with game scores.
In film scores and concert music alike— note that I’m mainly referring to orchestral compositions when I say concert music— the viewer, or audience, has no input on how the music progresses. There’s no agency there, measure 4 will always come after measure 3, no matter what happens.
A movie is never going to change based on the audience’s actions, unless we’re talking about something like Black Mirror: Bandersnatch (2018)… But that is not the point!
My point here is that the approach to composing music for games is very different than film or concert music due to the nature of the medium. Games have a player (or players!) that is actively influencing the pace at which the experience progresses. In a film, you know exactly how long it will take for a viewer to get to the next scene— the opposite is true for games. You never know exactly how long a puzzle will take, or how much time the player’s going to spend fighting a certain boss. And that’s what makes game scoring so uniquely challenging (and fun!).
please take all of this with a grain of salt, by the way! i’m just one (1) guy with 0 real qualifications besides “i went to school for music haha” :,-)
what is an “interactive score”?
What do we mean when we say that a score is interactive? That term can be used more broadly to refer to anything from video game music to audio installations, but the rest of this post is about video game music so…
An interactive game score (also referred to as an adaptive score or adaptive music!) is a system that can adapt and change to reflect what the player is currently doing. This can mean a lot of things, which we’ll get into in a moment!
Say, hypothetically, you’re playing an RPG and you enter a combat state. You’re going to hear a change in the music, for sure. Maybe there’ll be a flashy stinger, a riff or a drumroll or something, then a much more high-energy track will start playing. Pretty standard, right?
Both the tune that was playing before (likely an area theme of some sort) and the combat theme are most likely looping tracks, meaning that once they reach the end, they’ll repeat from the beginning (or from a different pre-defined point, but that’s neither here nor there).
Here’s an example of a short looping track that I wrote for a game jam two years ago!
The piece repeats back to the beginning at around 53 seconds. It’s a little hard to catch if you’re not paying attention! At least, I’d like to think so…
Looping tracks are probably the first thing that comes to mind when discussing game audio, and despite how simple it might seem on the surface, writing a piece of music that’s interesting enough to not annoy the listener after the 10th listen-through that stays somewhat static enough to prevent an awkward loop is difficult! And the fact that we as game composers write looping tracks most of the time is enough to set it apart from linear scoring in a fundamental way.
designing systems
The level of interactivity in the audio system for your games will 100% depend on the project. Sometimes you’ll have a project where the team just wants the music to crossfade as you travel between areas. Other times you’ll have a project that wants every little thing the player does to trigger a change in the music.
Say you’re tasked with writing a game score where the music needs to sound like it was tailored specifically to fit the player’s experience, in the sense that your very non-linear game score sounds linear. No hard cuts, no obvious crossfades, none of that. It needs to flow seamlessly. How do you approach that? You’ve gotta start thinking about your score as a system.
That’s an example of a more involved music system, a simplified version of it might look something like this:
horizontal resequencing, branching, and transitions
In the most basic terms, horizontal resequencing just means that you’re stopping one piece of music to start another one. The easiest (and least elegant) way to do that is to either hard cut or fade out a track, but there are some more involved approaches that we can take, such as…
Branching: This technique consists of short (ideally one or two measures) musical phrases that’ll play through to the end before moving onto the next thing. This is probably my favorite technique because it’s inherently musical! You can guarantee that your musical ideas will complete before moving on! Yay!
There are some unique challenges that come with branching, though! If you need a change to be super quick, this might not be the right technique to use.
Bridge transition: Changing between two tunes with another really short chunk of music that’ll create a smooth transition!
That’s why it’s called a bridge, you’re connecting two separate pieces of music.
There are also several ways you can approach this depending on the situation! I have an example of a project I’m working on that uses two different bridge transition techniques. One is having custom transitions for every single branch you compose, another is having a single transition that’ll be used no matter what.
Crossfading: This is exactly what it sounds like. Sometimes it’s an acceptable approach to take, depending on the context (like fading out a title screen theme to get into the actual game), but it is the least musical way to go about things.
vertical layering
Also known as layering. This technique is all about stems, which might be a more familiar term for folks that have only worked with linear scores. Layering entails breaking up your music into multiple mixes that will be muted and unmuted as the player triggers in-game events.
Just like the previous techniques, there are several possible approaches that you as the composer can take to writing a layered track and it will ultimately depend on what the game is calling for.
One thing that I do think is really important to mention about writing layered compositions is that you probably shouldn’t compose layers that only play in one section of your piece. If the player triggers the layer at a part where the mix is silent, they’re not going to get that musical feedback and it’ll defeat the point of even having that layer there, if that makes any sense.
General rule of thumb: always assume that the player is going to trigger a change at the worst possible moment for you as the composer. Anticipate these types of problems and try your best to prevent any potential weird situations.
this is barely even scratching the surface of interactivity in game audio!
There’s so, so, so much more that can be discussed here, especially when it comes to the actual implementation of these techniques in-engine (or, better yet, in audio middleware)! But for the sake of brevity, I will be moving onto examples of these things. If anyone wants to hear about my shenanigans in Wwise, I’d be happy to write about that in the future…
Another thing that I didn’t touch on at all is algorithmically/procedurally generated scores, NOT TO BE CONFUSED WITH GENERATIVE AI, like what the audio teams behind Spore (2008) and Ape Out (2019) created. If I’m being real, I just don’t know as much about this particular approach to adaptive music! I would have to do a bit more research myself before getting really into it here.
real-world examples!
Now that we’ve gone through some of the basics of adaptive music, I’d like to show you a few examples of this stuff in actual published games!!
Monkey Island 2: LeChuck’s Revenge (1991)
If you’re at all interested in adaptive video game music, odds are you’re already familiar with Monkey Island 2. This was one of the earliest examples of what we now consider to be adaptive scoring and possibly one of my favorite examples to bring up because of how well executed it is.
There’s a fascinating story behind the development of this game’s music system, iMUSE (short for interactive music streaming engine)! If you’re curious to learn more about it, here’s an interview with Michael Land, the composer and one of the two people responsible for creating this system.
The Legend of Zelda: Skyward Sword (2011)
Skyward Sword’s Bazaar has a base version of the theme that you’ll hear immediately upon entering and unique variations on the theme for each of the shopkeepers the player can interact with. Systems like this go such a long way in emphasizing these characters’ personalities! And it’s a lot easier to implement stuff like this in the present day, compared to what it was like when Monkey Island 2 released.
The Legend of Zelda: Tears of the Kingdom (2023)
This is probably one of my favorite examples of layering in recent games, just because of the narrative element to it. When you find a new member of the Stable Trotters and reunite them with the rest of the group, you’ll be rewarded with a new layer for their tune! Could not stop smiling after completing each of these sidequests.
Fire Emblem Fates: Conquest (2015)
The worse the story, the better the music… Anyway! There are two versions of the level themes in this game, one with very sparse instrumentation that’s played when you’re looking at the map and a fully orchestrated version that comes in when you zoom in on the action.
Pikmin !!!! the whole series !!!!!!
The music in this series is generally a really good example of all of this stuff. If you’re interested in a deep-dive of the boss music from Pikmin 3 Deluxe (2020) in particular, I’d highly recommend giving Scruffy’s video on it a watch.
The Pikmin series has such wonderful music. Criminally underrated series. Go play Pikmin 4 right now ooooh you want to play pikmin 4 soooo bad oooh
awesome now here’s something i’ve been working on :-)
This is a snippet of the audio system for a fantasy-horror point & click adventure game I’m currently working on titled To Learn To Shudder, which has a primarily orchestral score that is intended to sound completely seamless.
Most of the changes in the music are triggered by…
Location: This includes changes in whether the player is indoors or outdoors, time of day, different parts of a building (i.e. a common area vs a specific character’s room), etc.
How? Branching & a LOT of transition segments
Interactions with other characters: Whether or not the player is engaged in a conversation, how that conversation is going based on the player’s dialogue choices, etc.
How? Layering, mainly!
Progression: Finding a new item, discovering a new piece of information, how far along in a mission they are, etc.
How? Branching and/or layering, depending on the context!
This is a little tech demo featuring some of the tunes you’ll hear in chapters 1 & 2, particularly in segments involving the Church of Xemilon!
Warning: the tracks are all completely unmixed at the moment, I’m just focused on the systems themselves for now...
Here are some screenshots from my project, just to show how all of this is functioning!
And if you’re wondering what a project with a ridiculous amount of transition segments looks like… here’s what I have so far.
yeah!
One of the biggest draws of writing music for games to me is the element of interactivity. It almost feels like magic, putting stuff like this together! And this is possibly the biggest thing that sets game scores apart from film scores, in my opinion.
I leave you with a little mockup of my game’s UI. As a treat (?). Thank you for reading my game audio ramble of the day
