Philharmonia and partners announce interactive online show, Dream

The Royal Shakespeare Company (RSC), in collaboration with Manchester International Festival (MIF), Marshmallow Laser Feast (MLF) and Philharmonia Orchestra are to stage a world premiere in live, interactive performance using motion capture and an interactive score.

Dream, inspired by Shakespeare’s A Midsummer Night’s Dream, invites audiences to participate in a new type of live performance which is a shared experience between remote audience members and the actors. For digital audiences the latest technology in gaming and theatre are combined alongside an interactive symphonic score responding to the movement of the actors.

The live, real-time  performance is set in a virtual midsummer forest where audiences are able to directly influence the live performance. Under the shadow of gathering clouds at dusk, lit by the glimmer of fireflies, Puck acts as the guide. Audiences are invited to explore the forest from the canopy of the trees to the roots, meet the sprites, and take an extraordinary journey into the eye of a cataclysmic storm. Together with Puck they must regrow the forest before the dawn.  Audiences can choose to buy a £10 ticket to take part or to view the performance for free. The ten Dream performances are scheduled so that audiences across the world can join the event. Dream was due to open in Spring 2020 as an in person and online live performance, and has been recreated during the pandemic for online audiences.

The project is the culmination of Audience of the Future – a major piece of research exploring how audiences might experience the intersection of different artforms and technologies in live performance. The consortium, led by the RSC, involves 15 specialist organisations in immersive technology including leading British arts companies, global technology giants and top British universities. Working collectively to apply their knowledge and expertise in performance, music, video production, gaming and the research sector they have created a new type of live performance for audiences to experience.

Music is integral to the experience of Dream. The Philharmonia, pioneers in combining digital technology with classical music to create immersive audio experiences, have designed an interactive symphonic score for Dream. It will be manipulated in key points in the play in real-time by the performers, who will create interactive music with their movements. The result will be a living, dynamic soundtrack that adapts and interacts live with the narrative and the pre-recorded orchestral tracks – a total reimagining of composition for interactive audiences.

The living score is made possible by Swedish composer Jesper Nordin’s ground-breaking music tool Gestrument, which allows an interactive musical layer soundtrack with recordings captured in spatial audio. Using Gestrument performers generate music from their movement, improvising live to add layers of music to the pre-recorded score.

The installation will feature core classical repertoire and two new modern contemporary orchestral works – excerpts from Gemini, one of the the latest compositions by Esa-Pekka Salonen, the Philharmonia’s Principal Conductor and Artistic Advisor, and Ärr, composed by Nordin. The music was recorded by the Orchestra on Friday 13^th March 2020, its last full-scale orchestral recording (100 players) before the pandemic struck, conducted by Salonen.

The production is performed with seven actors in a specially created 7x7metre motion capture volume created at the Guildhall in Portsmouth, supported by a team from the University of Portsmouth. The performance space includes an LED backdrop which displays the unreal world allowing performers to see their place and act within the virtual environment. Vicon motion capture cameras and state of the art facial rigging capture the movements of the performers. This in turn drives the virtual avatars of each of the characters in real-time through a traditional performance lighting desk into Epic Games’ Unreal Engine. The live performance is mixed with pre-recorded animation sequences.

A bespoke web-player has been created for Dream to enable the effective distribution of real-time content from individual audience members to the Unreal Engine server and back to the audience. This new software allows the level of dynamic, real-time interaction working with a mass volume of users (up to 2000 per performance) in a live environment.

Esa-Pekka Salonen, Principal Conductor, Composer and Creative Advisor said: “Immersive technologies are going to change the way we compose, perform and experience music, and I believe we need to reimagine the symphony orchestra for this new landscape. This is primarily an R&D project – to reimagine immersive technology for live performance – where the creative process itself is one of the most valuable aspects of the whole project. Dream has brought us together with some of the world’s leading theatre practitioners, VR world-builders and game designers to reimagine storytelling. The collaboration has moved from the stage to the real-time games engine. As a composer, it has allowed us to reimagine composition for a new landscape where the audience has agency, and the performers can become part of a living, dynamic score that is integral to the live performance.” 

Luke Ritchie, the Philharmonia’s Executive Producer on the project, said:  “Under Esa-Pekka’s leadership, the Philharmonia has been using digital technology to create immersive musical experiences for over a decade, and Dream represents the next step in this mission, with performers and audiences both interacting in a virtual space. It’s been a delight to work with two leading composers to reimagine composition for this new creative landscape.

The Philharmonia’s R&D work with Jesper Nordin and Gestrument has implications for future orchestral experiences whether they be online, in-game, as a location-based experience or a live performance in the concert hall. In Dream the actors will use Gestrument to perform music; in future we want audiences to be playing this role as well, removing traditional barriers between performers and participants, and opening up a new world for the orchestra.”

 Gregory Doran, RSC Artistic Director said: “What’s brilliant about Dream is the innovation at play.  An audience member sitting at home influencing the live performance from wherever they are – that’s exciting.  It’s not a replacement to being in the space with the performers but it opens up new opportunities. By bringing together specialists in on-stage live performance with that of gaming and music you see how much they have in common. For instance, the RSC’s deep understanding of scripted drama combined with Marshmallow Laser Feast’s innovation in creative tech brings thrilling results.

The story is king, whether you are a gamer, or an audience member.  Stories haven’t changed, but the way we engage audiences with them has.  Shakespeare was our greatest storyteller and it’s brilliant that we get the opportunity to use one of his plays to discover what could be possible for live performance.”

The Philharmonia has expanded its programme of immersive musical experiences with four VR films since 2015, allowing audiences to step inside the Orchestra. Its first large-scale walk-through installations, RE-RITE (2009, based on Stravinsky’s The Rite of Spring) and Universe of Sound: The Planets (2012) were succeeded by the audio-led VR experience the VR Sound Stage, which premiered at SXSW and Southbank Centre in 2018.

Artistic leadership:

Gregory Doran – Artistic Director, RSC

Robin McNicholas – Founder and Creative Director, Marshmallow Feast

Esa-Pekka Salonen – Principal Conductor & Artistic Advisor, Philharmonia Orchestra

Cast and creatives:

Robin McNicholas – Director

Pippa Hill – Script Creation

Robin Mc Nicholas & Pippa Hill – Narrative

Esa-Pekka Salonen – Music Director & Composer

Jesper Nordin – Composer, Interactivity Designer and Creative Advisor, Music

Sarah Perry – Movement Director

Maggie Bain (Cobweb), Phoebe Hyder (Understudy Puck and Mustardseed), Durassie Kiangangu (Moth), Jamie Morgan (Peaseblossom), Loren O’Dair (Mustardseed), EM Williams (Puck), Edmund Wood (rehearsal assistant, Understudy Moth, Cobweb & Peaseblossom).

Dream is one of four Audience of the Future initiatives, supported by the government Industrial Strategy Challenge Fund which is delivered by UK Research and Innovation.