Hello and welcome to the video: Designing Sound and Music for Digital Storytelling—Creation and Production with Digital and AI-Based Applications as Part of Project-Based Humanities Courses.

My name is Katrin and I’m from the Digital Humanities Department at the University Jena, Germany. Digital storytelling has emerged as an important topic in the curriculum for students of the DH.

But story creators face some hurdles due to copyright regulations, especially when it comes to the part of the sound of their stories because existing sounds cannot simply be accessed and reused in own media products.

AI-based and digital music production programs offer an opportunity to create sound by yourself and to use it individually—even if you are not a professional in the field of music or music production.

In the following I´m going to explain what digital storytelling is and I’m giving some examples. Furthermore, I point out possible topics for a teaching/learning scenario with reference to humanities. I’m going to explain what the Digital Audio Workstation (short: DAW) is and why you should use it and how you can use it. Finally, how to schedule this topic in the humanities course will be thematized. For this, also, additional material is provided.

What is Digital Storytelling? Digital storytelling is a form of storytelling that uses digital media and technology to create, share, and present stories. It's a creative and interactive way of telling stories and can take various forms, including video, audio, animation, interactive websites, podcasts, blogs, and social media.

Digital stories can be both fictional and non-fictional and presented in different genres. Digital storytelling is used in various fields including education, journalism, arts, and culture.

An important aspect of digital storytelling is the possibility of audience participation and engagement. Digital stories can often be interactive and engage the audience, whether through the ability to make decisions, influence the course of the story, or to interact with the story. This creates a special experience.

It therefore also provides opportunities to tell and share stories about cultural heritage, identity, traditions, and other aspects of culture.

Within the cultural context:

• It [digital storytelling] can preserve cultural heritage;

• It can foster cultural diversity and inclusion;

• It can culturally educate;

• It can be used for cultural marketing and tourism; and

• [It can be used for cultural activism because] it enables activists to tell stories and spread their messages.

Digital storytelling therefore plays an important role in culture, transforming the way stories about cultural heritage are told, shared, and preserved.

Here are some Examples of Digital Storytelling in famous cultural institutions: The NMAAHC (National Museum of African American History and Culture) in Washington D.C. uses digital media, including video, audio, interactive displays, and other multimedia formats, to tell stories about African Americans and their contributions to American history and culture.

These stories are presented in an engaging and immersive manner to engage the visitor in the narratives of the diverse aspects of African American history.

On their website you can explore many facets of digital stories. Here you can see some stories with different topics and different kinds of media. For example, an embedded video and links to more resources or full virtual tours to exhibitions that implement stories, images, videos, or timelines.

Another famous institution that uses digital Storytelling is the Rijksmuseum in Amsterdam. The Rijksmuseum was established in 1800. It houses an impressive collection of Dutch art and history. Currently it’s hosting an exhibition of Johannes Vermeer, who is known for his detailed and captivating depictions of everyday life in the 17th century. This exhibition can also be experienced virtually (Rijksmuseum).

Vermeer is one of the best-known Dutch masters of the Golden Age and is often referred to as the "master of light".

Here you can see famous paintings of him as the Girl with a Pearl Earring or The Milkmaid.

You can also experience a tour to the ancient Delft where he was born and lived.

Here, in particular, one can see how music and sound can further enhance the digital story and make it more immersive.

You hear strings that convey a special atmosphere [music playing] or birds in the background creating an open and welcoming atmosphere [sound of running water and bird song].

Digital storytelling is not only relevant for working in a cultural institution, but also for working in organizations, companies, or as a freelance artist. The topic can also be linked to such relevance for DH courses and thus incorporated into teaching.

So how can you proceed if you want to integrate or convey the topic in your teaching?

Here you see two possible topics where storytelling can take place.

In the first students explore some of the existing examples of digital storytelling that relate to a cultural theme and use music.

Then the students think for themselves: they start designing their own virtual story on a topic. In the second example, the students should draft an own virtual exhibition experience. For that, virtual exhibitions or exhibits can be explored. Google Arts & Culture can be used for this. Furthermore, the students can use existing free sound files from platforms or create their own sounds that they think fit into the exhibition they have designed.

Digital storytelling is versatile. However, if you wish to focus a little more on the music component, it is important to note that not all participants in a course will have [experience] in this area.

Here digital tools can be used to create and edit [sound] without necessarily being a music producer. DAWs and AI-based chord progressors work well for this.

A DAW is a digital platform for recording, editing, and publishing audio and can consist of (proprietary) software and hardware elements (Rouse). Music or sound is either recorded directly into the program or created directly in the software with program-internal functions or plugins. Sound can be edited with virtual mixers and filters, and managed and shared with organizational tools (Rouse). Some of you may be familiar with the popular audio free software Audacity. But it does not meet the definition of a DAW in the narrower sense, because creating or transforming chords, keys, notes or playing and sampling virtual instruments, as well as accessing a sound library from which loops can be embedded, is not a function here, which is given.

DAWs focus on the aspect of composing, arranging, mixing and mastering music with and in a digital infrastructure.

In doing so, they digitize music production and make it accessible to non-professional musicians (Bell) and it becomes possible to use them outside of the music industry, for example in teaching contexts (Walzer).

However, no widespread use of DAWs in education is currently evident (Uyub). Studies examining the use of a DAW in teaching usually refer to commercial software (Stickland et al.). Concepts for implementing DAWs in a teaching-learning setting for students with an explicitly non-musical practical or theoretical background are not very common.

So how can a DAW be used if you don't come directly from the field of music theory or practice?

For this purpose, the focus is on the DAW GarageBand. Many of the DAWs on the market are purchase software [e.g., Ableton, Logic, Cubase] or require registration [e.g., Bandlab]. For educational use, this can be a hindering factor. Apple's GarageBand software is (mostly) pre-[installed] on the iPad, iPhone, and Mac. The advantage of GarageBand lies precisely in its intuitive usability, which also enables non-professional musicians or music theorists to use the program to learn how to produce and edit sound or music (Siemon; The GarageBand Guide).

[Now,] I’m going to introduce GarageBand on the Mac to you and how you can use so-called chord progressors for generating melodies that can be further edited in GarageBand.

When you click on the GarageBand icon, the program will open. Here you can see that you can create a new project and make settings under it (for example, regarding the tempo or the key). But we can adjust all of this later directly in the project too. If you already have project files, you can either open them directly by clicking on them, or open the last ones here. The suffix of the GarageBand files is .band.

When you open an empty project, this window appears by default, where you have to choose what format the first track in the project should have. You can choose between software instruments, audio tracks, or so-called drum machines. If you want to record your voice using a microphone or connect an electric guitar or bass via an audio-interface, for example, then choose the blue option. If you want to start with a beat, then the yellow one.

We'll start with the Software Instrument, which by default appears as the Keyboard when we select it. We go to “Create” [which creates] and we [arrive] at the project canvas. Here we can now continue working directly and explore a lot.

Now you can see that a music keyboard will automatically appear, which is belonging with the software instrument that we have selected. But we can also click away that first because you can open it again and again by going to Window and show musical typing. Show keyboard opens a simpler representation of the keyboard. In theory, you can start playing here.

But first, let's take a look at the menu itself.

Here in the middle is the main menu and the most important settings for the entire project. Here you can also edit the key or the tempo again.

Different views can also be set here. Next to it you will see various buttons like on a music player. In order to try them out, we first need sound material in the project. I'm now recording the notes of the C major chord I just played [metronome sound starts]. This is easy to do with the red record button [presenter plays notes of the C major chord].

We can now play it back or repeat it as often as we like by clicking on it here and dragging it into a so-called loop shape. If we want to listen to a section over and over again, for example in projects with several different parts, we can choose the Cycle option, which will then repeat the section highlighted in yellow over and over again [the metronome and chord notes play in a loop].

Let's look at the menu here on the left. Up here we still see the instrument library. This allows us to change the instruments in our project. Warning: this only works with MIDI files and not with audio tracks, which I will discuss later. For example, we can now change the recorded keyboard track to another keyboard instrument or say a string instrument [A string instrument and a metronome play in C major].

The question mark opens a help menu, which can be very informative, especially if you are new to the program.

Clicking on the controller opens the so-called Smart Controls. They can be used to change highs and lows as well as the sound of the respective track [metronome and notes].

With the scissors we can now make changes in our track. For example, I wasn't always quite in time when playing, which I can now manually adjust here in the so-called piano roll [music and metronome].

Speaking of beat: If I want the recorded music to be faster, I can change that afterwards. The same applies to the key.

Here on the right, we have another metronome that we can use as an aid, but you can also turn it off. The overall volume can be changed here [music plays without the metronome].

Here you can make notes and this symbol on the far right takes you to what is probably the most important function in GarageBand—the Sample Library of Loops.

You can help yourself from this library and simply browse through what you like. Listen to a bit of what goes with the track being played.

We'll take a beat and maybe this second piano here.

The key of the sample from the library automatically adapts to the key set above.

Nevertheless, from time to time there can of course be tones that don't quite fit.

And here we come to an important point—the difference between MIDIs and audio files—so-called .wavs.

While a WAV file mainly contains audio waveform data, which is the shape and form of how the magnitude of a vibration changes over time, the MIDI file consists of musical instructions to be interpreted. This also allows us to manipulate the MIDI files directly in the program.

For the WAV tracks, only the settings regarding the audio output can be changed, but not the individual sound properties themselves. You cannot change the instrument in the instrument library here either. But you could theoretically export the audio files and some services offer a conversion into MIDI. Often that doesn't sound very good.

A lot of things are possible with this. For example, if you want to create background noise for your digital story, which you can use to add a narration using a voice-over, you can also use the Toy Box sample pack here. Here we have some nice atmospherical sounds [nature sounds] or just sound from the outside [old fashioned car horn].

You can try it out a bit and you don't necessarily have to have theoretical musical knowledge here, so be creative and start!

If you still want to create your own more complex melodies faster, you can use AI-based applications for chord generation in addition to GarageBand. Sophisticated programs also exist here, but they are often expensive or, if used freely, prohibit further commercial processing. These are for example Aiva, Amper Music, or Soundful. These programs then also take over the entire music production.

But if you want to get a little active yourself, so-called chord progressors are very helpful.

Chord progressions are the basis of harmony in Western musical tradition and the basis of popular styles of music or traditional music and genres such as blues and jazz. In these genres, chord progressions are the defining feature on which melody and rhythm are built.

These are often learned in music theory classes because they build on scales and harmonies.

These theoretical aspects are already programmed into chord progressors and thus flow into your own music creation.

Let's look at [OneMotion], which is an example of a free [online chord progressor].

Here you can individually set the desired key, select the instrument, and determine the volume or the style of the tone sequence. If you later want to export it as a file and then import it into GarageBand, it is important that the same key and tempo settings are available here. So, let's see what sounds good [presenter plays keyboard—arrives at a C Em Am G progression]. This sounds good, so let’s take it!

The tone sequences can be dragged and dropped down here. Entire sequences can be created in this way. And these can then also be exported here as MIDI or WAV.

In the downloads you will find then the respective melody: the tracks can now simply be dragged into the GarageBand project, so finally, we are going to try this with another project.

I have created some content for my own digital story and now want to add some music and sound. I’m going to take this guitar because I think it fits very well. And now I want to add some chords and notes and I will use the chord progressor. Make sure to set the same settings in key and beats per minute.

[Presenter demonstration without commentary]

So, this is [what my final track sounds like]: [guitar playing with drumbeats].

And this is my final story, and this is what it sounds and looks like:

[A video plays with a voice over and the same music as the previous clip plays in the background. The voiceover says: “They say ‘Not all those who wander are lost’, but have you ever thought about going on a trip, out of your comfort zone? To embark on a journey, discover new cultures, make friends all over the world, and enjoy the beauty of nature? This is the story of my road trip. And what’s yours?”].

Finally, you might be wondering how to fit all of this into a storytelling course. I have created a [course] schedule for this, which you can find in the related materials. So, feel free to try it and let me know if you have any questions or comments.

Have fun trying it out and thank you very much.