The music of Dream: reimagining music for interactive and immersive technologies

“[A]t the heart of it there’s a sense of a communal experience – that’s the glorious thing about what an orchestra can do.”

Can you describe the music we hear in Dream?

Dream has a mixture of three pieces of music, recorded on Friday 13 March 2020, our last recording before lockdown. One is an excerpt from a work called Gemini by our Principal Conductor Esa-Pekka Salonen. There’s an excerpt from a terrifying but amazing bit of contemporary music called Ärr, by Jesper Nordin, the other key composer we’ve worked with, and there’s also a beautiful lush bit of Ravel.

Gestrument, a tool which Jesper created, gives parts of Dream an interactive musical layer. It tracks performers and allows them to generate music from their movements in real time, that always plays in perfect sync with the pre-recorded orchestral score, so it allows the performer to add a musical signature to some of their gestures.

What impact does the music have on the experience? 

There’s just something really human and communal about 100 musicians playing together, some big overwhelming beautiful music that makes you feel immersed. At moments it’s frightening, and there are moments of wonder, but at the heart of it there’s a sense of a communal experience – that’s the glorious thing about what an orchestra can do.

And there’s the feeling that what you’re seeing is live. The way the performers move and interact with the sound world means that every show is slightly different, the score shifts and changes in response to how they behave and speak.

“The concept behind all of the Philharmonia’s immersive experiences, whether it’s a big walk-through projection-based installation or one of the four VR films that we’ve made, is that you get to step inside the orchestra.”

Could this type of interactivity be used in other settings? 

It has really exciting applications in the future for audiences. Imagine if you were at a concert or in an experience where your hand movements, your gestures or your fingers on the screen interacted with the music either to play along or to add to it.

Gestrument is a wonderfully democratising tool because you don’t have to know how to play an instrument. You can feel like you’re improvising musically, even if you don’t have any musical expertise. It could make music performance and interactivity accessible to children, to people who aren’t musically trained, or people who don’t have the physical ability to play certain instruments.

What do you want to come out of the project?

What I’d really love is to try and make a legacy of the project, to share what we’ve learnt and build a road-map of how these kind of cross-disciplinary projects can work.

Personally, I can’t wait to watch it with my kids. We’re going to watch it on our TV together. I hope that people watch it and feel some kind of communal sense of reconnection.

What were the challenges of the project for you?

Our big creative and technological challenge was reimagining composition for orchestras, for an interactive medium, where audiences or performers have agency and can interact and change the plot.

It is also a challenge to work in such a huge consortium – we’re such different organisations. That’s been amazing but we’ve had to work really hard to stay together and to work in lockstep. It takes us longer to do things because we have to work around each other and have different ways of working. And of course, the pandemic. Last year, we were in mid-production in March, and then everything stopped. It’s changed the show completely. But I think the response is a really positive one.

You’ve worked on a lot of immersive projects in the past – what key things are you bringing to the project from that experience?

The concept behind all of the Philharmonia’s immersive experiences, whether it’s a big walk-through projection-based installation or one of the four VR films that we’ve made, is that you get to step inside the orchestra. When you’re in it, you realise that it’s a huge communal human endeavour and it’s very visceral and physical and personal.

We brought that expertise in spatial audio, immersive experiences and recording beautiful music of the highest quality. The big challenge we set ourselves was how to make that interactive, whether for the audience or the performers.