GPT-3 as a muse: generating lyrics

Introduction

Ever since I started playing guitar some 10 years ago I dreamt of writing a song of my own someday that I can be proud of. Sometimes a nice melody or chord progression pops into my head, but somehow the lyrics that I write always feel a bit uninspired.

Then I joined a team for the AI Song Contest in 2021, and we wrote a complete song with the help of AI. We had a process of turning sci-fi stories into lyrics using GPT-3 prompting. This resulted in pretty bizarre, but very interesting lyrics. However I felt this method lacked a feeling of control. I wanted to be able to influence the style of the music, the mood and the topic the song was written about. So I continued the lyrics generation project.

In this blogpost I will explain different techniques I used to generate lyrics with the help of GPT-3.

• Generating lyrics using prompts
• Generating lyrics using a finetuned GPT-3 model
• Creating variations for existing lyrics
• Conclusion.

Generating lyrics using prompts

GPT-3 is created to finish a text given a certain input. These text inputs are also called prompts. The easiest way to use this model is to write a prompt that triggers the model into writing the text you want. For example, if you want the model to write a good joke, you could prompt it with the following text: "A very funny joke:". Prompting works better when you give some examples of what you want first, so GPT-3 can use the pattern in your examples to figure out what you want to achieve.

I first prompted the model without giving it any examples, to write a song about following your dreams. Bold text is the prompt, and normal text is the output that GPT-3 generated.

The result is pretty much what I asked for, but it is very basic. There are no metaphors, and the subject is almost copied literally into the lyrics. But the cool thing is that GPT-3 understood that a song should consist of different verses.

To generate more lyrical results I tried adding some examples to the prompt, in the hope that this would make the lyrics a bit more poetic. This allows GPT-3 to mimic the structure of the examples in the generated text. The prompt with examples is structured as follows

description: example description 1
lyrics: example lyrics 1 
<---> 
description: example description 2
lyrics: example lyrics 2
<---> 
description: example description 3
lyrics: example lyrics 3
<---> 
description: description for the new song to be generated
lyrics:

In between the examples I added a divider, to make it easier for the model to understand the structure I am looking for.

To make it easier to prepare the examples we can also use GPT-3 to generate descriptions of lyrics, by prompting it with the lyrics and asking it to describe them. An example:

The GPT-3 API has a temperature slider that controls how much randomness to allow. Temperature 0 means no randomness is allowed, and only the most probable characters are selected. A temperature of 1.0 is the highest and means a lot of randomness is allowed. This creates more interesting twists in the text, but also makes the texts more chaotic. The default value for temperature is 0.7. I found that increasing this value to 0.8 meant that the generated lyrics were a bit more interesting and poetic.

The following lyrics are created using the same prompt, but with three examples and a temperature value of 0.8 instead of 0.7. The examples are the songs

• Walk away by Franz Ferdinand
• Guaranteed by Eddie Vedder
• Universally Speaking by the Red Hot Chilli Peppers

For the next lyrics I added two more example songs and adjusted the temperature to 0.9, to make the lyrics even more imaginative.

The example songs added are:

• Sunsets for somebody else by Jack Johnson
• The hardest part by Coldplay

It seems that GPT-3 understood the concept of verses now, and it even starts to rhyme a little. See the first verse where "dream" rhymes with "mean", and "stone" rhymes with "own".

There maximum length of prompt and output in the latest GPT-3 model is 4000 characters, and this prompt with output uses up a little over 2000 characters. So I will add two more examples to see if there is more improvement.

The songs added are:

• Stop this train by John Mayer
• Dream catch me by Newton Faulkner

Interestingly if you prompt GPT-3 to answer questions and add "Let's think step by step" the accuracy of the answers increases by a lot.

Here's the leaderboard of prompts to add to GPT-3.
Can you guys come up with anything better? https://t.co/bGnBroMdR5 pic.twitter.com/a3JCdUjlYu

— Aran Komatsuzaki (@arankomatsuzaki) May 25, 2022

We can do something similar by adding tags like [Verse], [Verse variation], [Chorus] and [Bridge] to the examples. This makes the model "think" of the structure of the song, because it has to add a tag before each part. An added benefit is that the tags give you more control over the structure of the generated song. If you want to add another verse you can simply add a [Verse variation] tag, and see what the generator fills in.

An example of a song generated with added tags.

Generating lyrics using a finetuned GPT-3 model

Another way to introduce more of the desired lyrical structure and poetic quality into the generated texts is to fine-tune a model using the OpenAI API.

An important decision is the structure of the data you want to fine-tune the model on. I wanted to have control over the output of the generated songs by providing a description, name and bio of an artist, a mood and a song title. Using this method you have a lot of control over the generated lyrics. In addition to control over the subject of the song, you can decide to generate songs in the style of a given artist, and decide if it should be a happy or a sad song.

In the documentation they recommend that you should provide at least 500 examples for a good quality fine-tuned model. It is a lot of work to write song descriptions and bio's for at least 500 songs, so I created a python script to generate some of the data for me. As input I would provide just the artist name, the song title and the lyrics. The python script would call the OpenAI api to append the json with an artist bio, mood and a description of the provided lyrics. This is done using just the GPT-3 davinci model and a simple prompt.

The prompt for generating the lyrical description is as follows:

{lyrics} 

How would you describe the meaning of the above lyrics? Describe some of the metaphors. 

The prompt for generating the artist bio's:

Write a short bio for {artist}. What is the spirit of their music, their genre, and how would you describe their personality?

The prompt for describing the mood of the lyrics:

{lyrics}

What is the mood of the above lyrics?

The mood is

An example of the output generated by this script. The "bio", "description" and "mood" fields are filled using GPT-3.

Using descriptions like this for 500 songs I fine-tuned a GPT-3 Davinci model. The Davinci model is the largest and most capable of the GPT-3 models available. The larger the model, the more emergent capabilities appear. For example, the smallest model can do things like parsing text, simple classification, address correction, coming up with keywords. Capabilities like summarizing text, finding out cause and effect, and creativity only appear in the larger models. Fine-tuning a model like that for 2 epochs took just a few minutes and cost around $20.

This is a subset of my fine-tuning file, containing 4 out of the 500 songs I used for the model. It should be in json lines format, which is one json per line. Note that \n\n###\n\n is used to indicate this is where the completion starts, and <!> is used to indicate the end of the lyrics. It can be used as a stop condition when generating lyrics.

{"prompt": "artist: Black Keys\nbio: The Black Keys are an American rock band formed in Akron, Ohio, in 2001. The group consists of Dan Auerbach (guitar, vocals) and Patrick Carney (drums). The band's raw blues-rock sound draws heavily from Auerbach's guitar playing and Carney's minimalist drumming. Although they have been compared to other garage rock revivalists such as the White Stripes and the Strokes, the Black Keys have a more soulful and vintage sound.\ndescription: The lyrics are about someone offering to be a constant source of support and guidance for another person.\nmood: The mood is hopeful and reassuring\n\n###\n\n", "completion": " [Everlasting light]\n\nLet me be your everlasting light\nThe sun when there is none\nI'm a shepherd for you\nAnd I'll guide you through\nLet me be your everlasting light\n\nLet me be your everlasting light\n(Shoo-shoo-shoo-wa)\nI'll hold and never scold\n(Shoo-shoo-shoo-wa)\nIn me you can confide\nWhen no one's by your side\nLet me be your everlasting light\n(Shoo-shoo-shoo-wa)\n\nOh baby, can't you see\nIt's shinin' just for you\nLoneliness is over\nDark days are through\nThey're through\n(Shoo-shoo-shoo-shoo-shoo-shoo-shoo-wa)\n\nLet me be your everlasting light\n(Shoo-shoo-shoo-wa)\nA train goin' 'way from pain\n(Shoo-shoo-shoo-wa)\nLove is the coal\nThat makes this train roll\nLet me be your everlasting light\nYeah\n\nLet me be your everlasting light\nLet me be your everlasting light\nLet me be your everlasting light\nOhh...<!>"}
{"prompt": "artist: Arctic Monkeys\nbio: Arctic Monkeys is an English rock band formed in Sheffield in 2002. The group consists of Alex Turner, Matt Helders, and Nick O'Malley. The band's debut album, Whatever People Say I Am, That's What I'm Not, was released in 2006 and became the fastest-selling debut album by a band in UK chart history. The band's second album, Favourite Worst Nightmare, was released in 2007. The band's third album, Humbug, was released in 2009. The band's fourth album, Suck It and See, was released in 2011. The band's fifth album, AM, was released in 2013. The band's sixth album, Tranquility Base Hotel & Casino, was released in 2018.\ndescription: The lyrics of the song seem to be criticizing the pretentiousness of some people in the indie music scene. The singer is talking about how some people try to act like they are from San Francisco or New York City, when they are actually from much smaller towns. They also seem to be making fun of how some people in the scene try to act like they know everything about music, when they really don't.\nmood: The mood is one of disdain and mockery\n\n###\n\n", "completion": " [Fake tales of san francisco]\n\nFake Tales of San Francisco\nEcho through the room\nMore point to a wedding disco\nWithout a bride or groom\n\nThere's a super cool band yeah\nWith their trilbies and their glasses of white wine\nAnd all the weekend rock stars in the toilets\nPracticing their lines\n\nI don't want to hear you\n(Kick me out, kick me out)\nI don't want to hear you no\n(Kick me out, kick me out)\nI don't want to hear you\n(Kick me out, kick me out)\nI don't want to hear you\nI don't want to hear your\n\nFake Tales of San Francisco\nEcho through the air\nAnd there's a few bored faces at the back\nAll wishing they weren't there\n\nAnd as the microphone squeaks\nA young girl's telephone beeps\nYeah she's dashing for the exit\nOh, she's running to the streets outside\n\"Oh you've saved me,\" she screams down the line\n\"The band were fucking wank\nAnd I'm not having a nice time\"\n\nI don't want to hear you\n(Kick me out, kick me out)\nI don't want to hear you no\n(Kick me out, kick me out)\nYeah but his bird said it's amazing, though\nSo all that's left\nIs the proof that love's not only blind but deaf\n\nHe talks of San Francisco, he's from Hunter's Bar\nI don't quite know the distance\nBut I'm sure that's far\nYeah, I'm sure it's pretty far\n\nYeah, I'd love to tell you all my problem\nYou're not from New York City, you're from Rotherham\nSo get off the bandwagon, and put down the handbook\nYeah, yeah, yeah, yeah, yeah\n\nGet off the bandwagon and put down the handbook\nGet off the bandwagon and put down the handbook\nGet off the bandwagon and put down the handbook\nGet off the bandwagon and put down the handbook, yeah<!>"}
{"prompt": "artist: Coldplay\nbio: Coldplay is a British rock band formed in 1996 by lead vocalist and pianist Chris Martin and lead guitarist Jonny Buckland at University College London. After they formed under the name Pectoralz, Guy Berryman joined the group as a bassist and they changed their name to Starfish. Will Champion joined as a drummer and backing vocalist, completing the lineup. Coldplay achieved worldwide fame with the release of their single \"Yellow,\" followed by their debut album, Parachutes (2000), which was nominated for the Mercury Prize.\ndescription: The lyrics seem to be about wanting a simple, down-to-earth love, without any pretence or artifice. The speaker is tired of reading about mythical heroes with superhuman gifts, and instead wants something (and someone) more real and relatable.\nmood: The mood is romantic and longing\n\n###\n\n", "completion": " [Something like this]\n\nI've been reading books of old\nThe legends and the myths\nAchilles and his gold\nHercules and his gifts\nSpiderman's control\nAnd Batman with his fists\nAnd clearly I don't see myself upon that list\n\nBut she said, \"Where'd you wanna go?\nHow much you wanna risk?\nI'm not looking for somebody\nWith some superhuman gifts\nSome superhero\nSome fairytale bliss\nJust something I can turn to\nSomebody I can kiss\nI want something just like this\"\n\nDoo-doo-doo, doo-doo-doo\nDoo-doo-doo, doo-doo\nDoo-doo-doo, doo-doo-doo\nOh, I want something just like this\nDoo-doo-doo, doo-doo-doo\nDoo-doo-doo, doo-doo\nDoo-doo-doo, doo-doo-doo\n\nOh, I want something just like this\nI want something just like this\n\nI've been reading books of old\nThe legends and the myths\nThe testaments they told\nThe moon and its eclipse\nAnd Superman unrolls\nA suit before he lifts\nBut I'm not the kind of person that it fits\n\nShe said, \"Where'd you wanna go?\nHow much you wanna risk?\nI'm not looking for somebody\nWith some superhuman gifts\nSome superhero\nSome fairytale bliss\nJust something I can turn to\nSomebody I can miss\nI want something just like this\nI want something just like this\"\n\nOh, I want something just like this\nDoo-doo-doo, doo-doo-doo\nDoo-doo-doo, doo-doo\nDoo-doo-doo, doo-doo-doo\nOh, I want something just like this\nDoo-doo-doo, doo-doo-doo\nDoo-doo-doo, doo-doo\nDoo-doo-doo, doo-doo-doo\n\nWhere'd you wanna go?\nHow much you wanna risk?\nI'm not looking for somebody\nWith some superhuman gifts\nSome superhero\nSome fairytale bliss\nJust something I can turn to\nSomebody I can kiss\nI want something just like this\n\nOh, I want something just like this\nOh, I want something just like this\nOh, I want something just like this<!>"}
{"prompt": "artist: Counting crows\nbio: Counting Crows is an American rock band from Berkeley, California, formed in 1991. The band consists of Adam Duritz (lead vocals, piano), David Bryson (guitar), Charlie Gillingham (keyboards, piano, accordion), Dan Vickrey (guitar), David Immerglück (guitar, banjo, mandolin), Matt Malley (bass) and Jim Bogios (drums). Counting Crows gained popularity following the release of their debut album, August and Everything After, in 1993. Their hit single, \"Mr. Jones\", peaked at number five on the Billboard Hot 100. The band's influences include Van Morrison, R.E.M., Bob Dylan, and The Band.\n\nThe spirit of Counting Crows' music is one of melancholy and introspection, with Duritz's lyrics often focusing on relationships and heartbreak. The band's musical style has been described as a mix of alternative rock, pop, folk, and country. Counting Crows' personality has been described as laid-back and easy-going. The band is known for their live performances, which often feature extended jams and impromptu cover songs.\ndescription: The lyrics appear to be about a man who is planning to take a trip to Spain with a girl. He is leaving his old life behind him, and is looking forward to starting fresh in a new place. The lyrics could also be interpreted as being about a relationship that is coming to an end. The man is ready to move on, and is hoping to find someone new to start a new chapter with.\nmood: The mood is carefree and happy\n\n###\n\n", "completion": " [Holiday in spain]\n\nGot no place to go\nBut there's a girl waiting for me down in Mexico\nShe's got a bottle of tequila\nA bottle of gin\nAnd if I bring a little music I can fit right in\nWe've got airplane rides\nWe've got California drowning out the window side\nWe've got big black cars\nAnd we've got stories how we slept with all the movie stars\nI may take a holiday in Spain\nLeave my wings behind me\nDrink my worries down the drain\nAnd fly away to somewhere new\nHop on my choo-choo\nI'll be your engine driver in a bunny suit\nIf you dress me up in pink and white\nWe may be just a little fuzzy 'bout it later tonight\nShe's my angel\nShe's a little better than the one that used to be with me\n'Cause she liked to scream at me\nMan, it's a miracle that she's not living up in a tree\nI may take a holiday in Spain\nLeave my wings behind me\nDrive this little girl insane\nAnd fly away to someone new\nEverybody's gone\nThey left the television screaming that the radio's on\nSomeone stole my shoes\nBut there's a couple of bananas and a bottle of booze\n\nOh well, Happy New Year's, baby\nWe could probably fix it if we clean it up all day\nOr we could simply pack our bags\nAnd catch a plane to Barcelona 'cause this city's a drag\nI may take a holiday in Spain\nLeave my wings behind me\nFlush my worries down the drain\nAnd fly away to somewhere new\nTake a holiday in Spain\nLeave my wings behind me\nDrive this little girl insane\nFly away to someone new\nFly away to someone new\nFly away to someone new<!>"}

Let's see if the fine-tuned model is better at generating lyrics. Let's try generating lyrics in the style of a few different artists.

Because the model kept generating existing song title's for the artists, I created an extra prompt to generate these. The title generating prompt is very simple:

Come up with an original title for a song about following your dreams by {artist}.

Below are some songs generated by prompting the fine-tuned model with different artists. I cherry picked the songs by generating up to 5 songs per prompt, and choosing my favourite.

"Chasing After Dreams", in the style of Jack Johnson.

I liked these lyrics a lot, so I added some chords and recorded a simple version of the song.

"The ghost of you", in the style of the Red Hot Chili Peppers.

"Find your way", in the style of John Mayer

"Dreams of the remaining", in the style of Queens of the stoneage

"Clouds on the horizon", in the style of Franz Ferdinand

"Chasing the sun", in the style of Muse.

"A whirlwind of dreams", in the style of Coldplay

To see if the model can also generate songs in a completely different style. Here is "Don't you give up on me", in the style of Taylor Swift.

Generating variations on existing lyrics

Sometimes you have generated a good verse, but GPT-3 decides that repeating the same verse over and over is a good idea. We can use prompts to solve this issue and introduce variation in the lyrics.

To do this we have to give some examples of a verse, and an alternative wording in the same structure. For example:

lyrics: 

On the corner of Main Street
Just trying to keep it in line
You say you wanna move on and
You say I'm falling behind
Can you read my mind?
Can you read my mind?

variation on the above lyrics:

I never really gave up on
Breaking out of this two-star town
I got the green light, I got a little fight
I'm gonna turn this thing around
Can you read my mind?
Can you read my mind?

For this experiment I added seven such examples to the prompt in the format

lyrics:

{example lyrics 1}

variation on the above lyrics:

{example variation 1} 

---

lyrics:

{example lyrics 2 - 7}

variation on the above lyrics:

{example variation 2 - 7}

--- 

lyrics:

{lyrics you want to generate a variation on}

variation on the above lyrics:

If I take the first verse from the generated Franz Ferdinand song and put it in there it outputs the following variation.

Inserting this new variation gives a whole new spin to the song

Without retraining the fine-tuned lyrics model we can steer a song in new directions by adding tags to the already created output. For the next example I thought the lyrics generated for the previously generated song "Chasing after dreams" could use a twist to keep the song interesting until the end. A bridge would be a good way to achieve this twist. To get the model to write a bridge I added tags for verse, chorus and pre-chorus and ended the prompt with a bridge tag. Amazingly it did add a bridge, even though the model was not trained to work with these tags! I think this is a great example of the flexibility of these models.

Conclusion

Although it still takes some of tweaking to get the results you want, this fine-tuned version of GPT-3 is very useful for providing inspiration for lyrics. It often comes up with ideas that I would never have thought of on my own, and it's a great way to get the creative juices flowing. The process of generating new lyrics and trying different styles is a lot of fun, and I found myself trying many different variations. Prompt programming is very flexible so it is easy to try out new ideas.

However one should be careful not to plagiarize other lyrics or writings. for example the prompt with 5 examples used some lyrics that are taken from an inspirational quote from Liam Payne.

Dreams are like stars
You may never touch them
But if you follow them
They will lead you to your destiny

I was also surprised at how useful GPT-3 is out of the box for preparing data. You can ask it to extract useful features from text, and use these features as data to finetune a new model.

I can definitely see tools like this being used by writers or musicians. Maybe not to generate entire songs with one click of a button, but more as a writing companion.