top of page

Music in Games

Music has entertained for thousands of years, evolving alongside human creativity and technology to enhance storytelling and emotional engagement. Matching music to forms of entertainment like theatre and film is a familiar concept and, within modern contexts, shapes our expectations of how music interacts with visual media, often creating paradigms that define how we emotionally connect to on-screen narratives (Stevens & Raybould, 2014). Video games present an opportunity to push the boundaries of music to be reactive to what’s happening on the screen. As the player is an active participant, making a curated musical experience becomes increasingly challenging proportional to their agency. The player’s freedom to explore areas in desired orders and lengths of time (Lazzaro, 2019), for example, makes transitioning and looping music tracks to retain both interest and cohesion key challenges in music design.  

 

The greater soundscape must be considered in this context; music reacting to every action on the screen may overwhelm the player, who needs to be able to hear and react to key audio cues from the world around them, especially given we react to audio cues faster than visual ones (Blauert, 1997). 

 

The blanket term ‘immersion’ is often used to define one of music’s primary purposes in a game-specific context and make the player connect with their on-screen representation, be it emotionally at a pivotal story moment; physically when they suffer an injury; cognitively as they solve a puzzle; or any other way. These states’ representations in music usually stem from established paradigms, such as the use of string orchestration for dramatic story beats. Immersion must, however, be distinguished from presence, which is described as “a sense of ‘being there’” (Nordahl & Nilsson, 2014 p214) and grounds the player in a game’s reality, especially within the contexts of VR and XR. Games aiming to invoke presence therefore limit non-diegetic elements and integrate music into the game through creative means such as radio stations accessible through in-game means (Rockstar Games, 2013). 

 

Another purpose of music in a game is to guide a player, such as when the music changes to indicate an enemy is vulnerable. In this context, the music doesn’t accompany the action but instead directs the player towards it or pre-empts it. This approach can establish ‘call and response’ patterns in the music, creating resolutions to expected musical patterns that enhance enjoyment (Stevens & Raybould, 2014). Implementing these patterns in adaptive soundtracks can usually be challenging, as a player may defeat an enemy halfway through a ‘call’ section. as maintaining both musical coherence and gameplay responsiveness can pose challenges. Constraining gameplay to facilitate a desired musical effect can therefore be a strategic and effective design choice. 

Figure 1: Gameplay from Super Mario Odyssey (Nintendo, 2017) showcasing music changing based on an enemy's state, guiding the player to take initiative (ProsafiaGaming, 2020).

Music can also be considered in games for its effect on gameplay. While exciting music often feels like it contributes to a player’s success and performance in a game, studies have shown this to not always be the case (Yamada, 2002). Besides achieving high scores, the perception of performing well is key to player engagement and immersion (Lipscomb & Zehnder, 2004). This perception also contributes greatly to player enjoyment by reinforcing the impression of "being suitably challenged and having the skills to master those challenges" (Stevens & Raybould, 2014) and achieving “flow” (Csikszentmihalyi, 1989). 

References

Blauert, J. (1997) Spatial Hearing : The Psychophysics of Human Sound Localization. Cambridge: The Mit Press. 

Csikszentmihalyi, M. & Csikszentmihalyi, I. S. (1989) Optimal Experience: Psychological Studies of Flow in Consciousness. Man, 24 (4) December, p. 690. 

Grand Theft Auto V (2013) Rockstar Games. 

Lazzaro, N. (2019) The 4 Keys 2 Fun | Nicole Lazzaro’s Blog [Online]. Nicolelazzaro.com. Available from: <https://www.nicolelazzaro.com/the4-keys-to-fun/> [Accessed 1 October 2024]. 

Lipscomb, S. D. & Zehnder, S. M. (2004) Immersion in the Virtual Environment: The Effect of a Musical Score on the Video Gaming Experience. Journal of PHYSIOLOGICAL ANTHROPOLOGY and Applied Human Science [Online], 23 (6), pp. 337–343. Available from: <https://www.jstage.jst.go.jp/article/jpa/23/6/23_6_337/_article/-char/ja/>. 

Nordahl, R. & Nilsson, N. C. (2014) ' The Sound of Being There: Presence and Interactive Audio in Immersive Virtual Reality ', in Collins, K. (ed.), The Oxford Handbook of Interactive Audio, pp. 213-233. 

ProsafiaGaming (2020) Super Mario Odyssey - All Bosses + Cutscenes (No Damage) [Online]. YouTube. Available from: <https://www.youtube.com/watch?v=AdWppEO_mtw> [Accessed 19 November 2024]. 

Stevens, R. & Raybould, D. (2014) ' Designing a Game for Music: Integrated Design Approaches for Ludic Music and Interactivity ', in Collins, K. (ed.), The Oxford Handbook of Interactive Audio, pp. 213-233. 

Super Mario Odyssey (2017) Nintendo. 

Yamada, M. (2002) The Effect of Music on the Performance and Impression in a Racing Video Game. Proc of the 7th International Conference on Music Perception & Cognition. Causal Productions, Adelaide, pp. 340–343. 

Technology Shaping Music

The limitations of technology has been a constant in the choices of what goes into a video game’s music. These very limitations defined how early eras of video game music were created and propelled their creativity (Cockfield, 2019).

 

Initial restraints meant that the most pressing issues of music programming were managing storage space and arranging the circuit board of the cabinet itself to implement and create the sounds required. This meant that game music was typically reduced to introductory/celebratory bleeps in the soundtrack and most storage was dedicated to sound effects (Collins, 2019, pp.9-12).

 

​

​

​

​

​

​

​

​

​

​

​

​

​

​

 

 

Figure 1: The soundtrack to Space Invaders (1978) is merely four notes long.

 

Three consoles, the NES, MegaDrive and the SNES, used consecutively more powerful sound chips in accordance with their release times. Copetti (2019) Sums up the details of each:

 

  • The NES’s Ricoh 2Ao3 sound chip, a Programmable Sound Generator, had 5 sound channels: 2 Pulse Waves suited for melody and chords, 1 triangle wave suitable for bass lines, a noise channel for percussion and a DMC (Delta Modulation Channel) for sampling. 

 

  • The MegaDrive also had a PSG chip, the Texas Instruments SN76489, but more importantly it contained the Frequency Modulator Synthesiser: The Yamaha YM2612, which has six FM channels, which work by modulating a wave signal to effect a second sound wave. It also contains a modestly powerful sampler.

 

  • The SNES contained the Digital Signal Processor chip: the SPC-700, allowing 8 different channels and programmable volume, panning, envelopes, delay, FM synthesis, frequencies, reverbs and pitch bends.

 

​

​

​

​

​

​

​

​

​

 

Figure 2: In order, the Ricoh 2Ao3 chip, the Yamaha YM2612 chip and the SNES SPC-700 chip.

 

In Super Mario Bros., Schartmann (2015) points out that musical theory was cleverly used to overcome the limitations of the 4 basic sound channels utilised, such as:

 

  • The use of jazz chords such as the fragmented D9 chord beginning the ‘Overworld’ theme and  the ‘Starman’ theme being composed of Dm7 and Cmaj7 chords, alongside the dissonant intervals of the ‘Castle’ theme.

 

  • The use of rhythm, such as the syncopation in the ‘Overworld’ theme and the 3/4 Waltz in the ‘Underwater’ composition.

 

  • Variety and contrast of each piece in their reflection of what is happening on screen, such as the contrast between the melodic, lively ‘Overworld’ theme and the chromatic, sparse and bare instrumentation of the ‘Underworld’ theme.

​

 

​

​

​

​

​

​

​

​

​

​

​

​

​

​

 

​Figure 3: Demonstration of various musical techniques in Super Mario Bros. (1985)

​

Super Mario World’s soundtrack simply utilises the same melody line through most of levels as a conscious choice to support a sense of consistency throughout the games longer run-time. The SPC-700 chip was used for a greater variety of instrumentation to better convey the mood and atmosphere throughout each level (Iwata).

​

 

​

​

​

​

​

​

​

​

​

​

​​​Figure 4: Three examples of the music utilised in Mario World (1990).

 

One thing about the development of Mario’s soundtracks is, apart from more sophisticated hardware allowing greater variation in musical selection, is that the music itself is always composed as small, melodic loops. Although this initial choice was shaped by storage restrictions, this pervades the use of music in Mario games now as an intentional design choice to reflect the level and help progress the gameplay. (Laroche, 2012).

 

The release of the MegaDrive was considered to be a massive step over the NES and its relative hardware restraints. Sega took great leaps to emphasise the power of its Sound Chip - particularly emphasising its ability to mimic particular instruments, particularly utilised to replicate Progressive Rock instrumentation. (Collins, 2019, p. 43).

​

 

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​Figure 5: The Phantasy Star 2 soundtrack (1989).

 

Through the Toy Story port (1991) to the SNES and MegaDrive, we can see the difference that technology makes to the approach to sound design for these two different competing consoles - soundtracks with a more ‘real’ instrumentation seem to flourish on the SNES, while MegaDrive ports fares better with more ‘electronic’ style tracks.

​

 

 

​

​

​

​

 

 

 

 

 

 

 





 

Figure 6: instances of the Toy Story (1995) 'Main Menu' and ‘Really Inside the Claw Machine’ music over each port.

 

Not only does the expanse of technology inform how game music can be utilised, the restrictions from platform to platform can also influence how music informs a gameplay and thereby the game itself. (Cerrati, 2006).

Ricoh2A03 better.jpg

References:

 

Cerrati, M. (2006) Video Game Music: Where It Came From, How It Is Being Used Today, and Where It Is Heading Tomorrow. Vanderbilt Journal of Entertainment and Technology Law, 8(2), pp. 293–334. 

 

Cockfield, C. (2019) Creative Limitation And The Super Nintendo Sound Chips. Available at: <https://hackaday.com/2019/07/31/creative-limitation-and-the-super-nintendo-sound-chips/> [Accessed: 19 November 2024].

 

Collins, K. (2008) Game sound : an introduction to the history, theory, and practice of video game music and sound design. Cambridge, Mass.: MIT Press.

 

Copetti, R. (2019) Mega Drive/Genesis Architecture: New techniques of composition. Available at: <https://www.copetti.org/writings/consoles/mega-drive-genesis/> [Accessed: 19 November 2024].

 

Copetti, R. (2019) NES Architecture: More than a 6502 Machine. Available at: <https://www.copetti.org/writings/consoles/nes/> [Accessed 19 November 2024].

 

Copetti, R. (2019) Super Nintendo (SNES) Architecture: Old hardware with mind-blowing features. Available at: <https://www.copetti.org/writings/consoles/super-nintendo/> [Accessed 19 November 2024].

 

Iwata, S. Iwata Asks | 5. Music Commentary by Koji Kondo (2). Available at: <https://www.nintendo.com/en-gb/Iwata-Asks/Iwata-Asks-Super-Mario-All-Stars/Vol-1-Super-Mario-History-Soundtrack-CD/5-Music-Commentary-by-Koji-Kondo-2-/5-Music-Commentary-by-Koji-Kondo-2--220040.html> [Accessed: 19 November 2024].

 

Laroche, G. (2012) Analyzing Musical Mario-Media: Variations in the Music of Super Mario Video Games. Montreal: McGill University. Available at: <https://docslib.org/doc/11853048/analyzing-musical-mario-media-variations-in-the-music-of-super-mario-video-games> [Accessed: 19 November 2024].

 

Phantasy Star II (1989) Sega. Tokyo.

 

Phillips, W. (2014) A composer’s guide to game music. First MIT Press paperback edition. Cambridge, Massachusetts: The MIT Press.

 

Schartmann, A. (2015) Koji Kondo’s Super Mario Bros. Soundtrack. London: Bloomsbury.

 

Space Invaders (1978) Taito. Tokyo.

 

Super Mario Bros. (1985) Nintendo. Kyoto.

 

Super Mario World (1990) Nintendo. Kyoto.

 

Toy Story (1995) Disney Interactive. Glendale.

MIDI in games

MIDI

As the progression of adaptive music has been discussed earlier, I want to talk about one of the ongoing technologies that has lingered around in games for at least around 40 years.
 

MIDI
 

“Musical Instrument Digital Interface: a standard means of sending digitally encoded information about music between electronic devices, as between synthesizers and computers (Dictionary.com, 2024)”
 

EARLY USES


The idea of being able to react to every action of the screen has been around for many years. One digital approach being through MIDI and similar file types. The concept of using music files that are based from informational data which get rendered in real time for adaptive music in games is not new.
 

The use of data-based music files was not confined to just MIDI either.

The Commodore Amiga which was made in 1985 included a tracker program alternative to MIDI which used MOD files which also stored the data in the notes such as “volume, effects and instrument” (Collins, K., 2013. p58)
 

iMuse developed by Lucas Art’s in the 1991 which sought to create adaptive music through decision points and markers within tracks but noted that “technological progress … in this area has been relatively limited. This was around the time of their game releases such as The Secret of Monkey Island (1991) and Day of the Tentacle (1993) (Collins, K., 2013. p51).


MEMORY
 

It has advantages over rendered media in MIDI form in that it is quick to produce, as it is generally a pre render stage of composition already; and as it contains the data in the notes, those parameters are accessible for adaptive purposes.

The downside of MIDI files is that for adaptive music they require real time rendering within the games to get the benefits of the adaptive inputs which is expensive for processing.
 

“MIDI file contains only the instructions for the synthesizer, so the file size is hundreds of times smaller than WAV.” (AbyssMedia, 2024.)
 

In terms of loaded memory this is beneficial for the audio budget, however it’s the processing where the data costs will arise. By being processed real-time, with a high CPU usage, the overloading could create quality issues such as “gaps, clicks or other audio issues” Ableton. (n.d.).

​

CAN IT BE DONE

Middleware such as Wwise, along with direct implementation in Metasounds already has MIDI enabled inputs which makes integration (Audiokinetic, 2024.) . 

 

User plugins are also available such as from Unreal Marketplace and Sweejtech (Unreal Engine, 2024.) (Sweej Tech. (n.d.).

​​

 floppy disc

[Figure 1: Image of MIDI nodes in Unreal Metasounds. (Brian Michael Fuller, 2024.)]

[Figure 2: Image of MIDI nodes in Wwise. (Audiokinetic, n.d.)]

COSTS IN GAME PRODUCTION
 

2 main issues with utilising MIDI as the musical base are in memory management and financial aspects.
In memory management, the cpu processing required for the realtime buffered effects and transitioning effects to make use of the adaptive qualities of MIDI could be heavy within the audio budget.​

[Figure 3: Image of pricing for Premium Plugins in Wwise. (Audiokinetic, n.d.)]

Financially, the amount of DSP’s in game engines is limited, and it is likely that the use of them in production would be a paid extra on top of the engine development costs. This could be prohibitive depending on the budget, especially if it was a per game share amount rather than fixed rate.
 

Companies may decide to respond by implementing their own DSP’s in order to circumvent the licensing fees for further musical DSP’s required, however this has a staff and time expenditure as well.

~~~~~~~~~~~~~~~~~~~~~~~

References:

​

​AbyssMedia, 2024. MIDI vs WAV. Available at: https://www.abyssmedia.com/midirenderer/midi-vs-wav.shtml#:~:text=Size%20of%20file&text=Without%20the%20use%20of%20compression,synthesis%2C%20so%2Dcalled%20soundfonts [Accessed 18 November 2024].
 

Audiokinetic. (n.d.). Wwise Pricing for Games. [online] Available at: https://www.audiokinetic.com/en/wwise/pricing/for-games/ [Accessed 18 November 2024].
 

Audiokinetic, 2024. Wwise 201: Lesson 4 - Working with MIDI. Available at: https://www.audiokinetic.com/en/courses/wwise201/?source=wwise201&id=lesson_4_working_with_midi [Accessed 18 November 2024].


Audiokinetic. (n.d.). Wwise 201: Importing MIDI Files. [online] Available at: https://www.audiokinetic.com/en/courses/wwise201/?source=wwise201&id=importing_midi_file#read [Accessed 18 November 2024].


Brian Michael Fuller
(2024). UE5.4 HARMONIX - Part 1: MIDI Deep Dive Part 1. [online] Available at: https://www.youtube.com/watch?v=1dyXc5BKWxU [Accessed 18 November 2024].


Collins, K., 2013. Game Sound: An Introduction to the History, Theory, and Practice of Video Game Music and Sound Design. Pp. 51-58.

Crawley, C. (n.d.). Audio File Size Calculator. [online] Available at: https://www.colincrawley.com/audio-file-size-calculator/ [Accessed 18
November 2024].


Dictionary.com, 2024. MIDI. [online] Available at: https://www.dictionary.com/browse/midi [Accessed 18 November 2024].


Epic Games. (2024). Unreal Engine 5.5 Release Notes. [online] Available at: https://dev.epicgames.com/documentation/en-us/unreal-engine/unreal-engine-5.5-release-notes [Accessed 18 November 2024].


Unreal Engine, 2024. MIDI Engine for Metasounds. Available at: https://www.unrealengine.com/marketplace/en-US/product/midi-engine-metasounds?sessionInvalidated=true [Accessed 18 November 2024].


Sweej Tech. (n.d.). MetaSound MIDI. [online] Available at: https://sweej.tech/technologies/metasound-midi [Accessed 18 November 2024].

​

© 2024 Leeds Beckett MSc

Sounds still brighter, more ringing and space on a layer above the lead instrument And more forward

bottom of page