EPD Students Create Dream Machine, a Virtual Performance Experience

What do you do when you’ve been planning a live interactive theatrical concert all semester, only to have the entire world moved online and all live events cancelled due to the COVID-19 pandemic? If you’re Berklee’s Electronic Production and Design (EPD) Department, you quickly pivot and find innovative ways to use technology to create a virtual performance experience.

The project initially got rolling last fall with Grammy-nominated visiting artist Nona Hendryx, who has been collaborating with EPD for the past several years on projects ranging from a dance piece with producer Hank Shocklee to an interactive performance with alumnus and Living Colour drummer Will Calhoun ‘86. Hendryx worked with faculty member Lori Landay’s Digital Narrative Theory and Practice classes to develop an interactive script, not only for the multimedia element, but also for the staging of the event.

In the spring semester, Akito van Troyer, assistant professor of EPD, taught the department’s Special Projects Seminar. Van Troyer, who had previously worked at the MIT Media Lab, was charged with developing a technology for a live interactive performance that would also incorporate the use of virtual reality (VR) into a performance environment. 

The project coalesced around the Mi.Mu performance gloves developed by artist and producer Imogen Heap, who visited Berklee both in Boston and at its campus in Valencia, Spain, where she received an honorary doctorate in 2019. “Mi.Mu is the gold standard for controlling visual and sound using gloves,” said Michael Bierylo, chair of the Electronic Production and Design Department. “We got two pairs of the gloves in February and were planning to use them in a theatrical piece with music, with students doing the sound design and visuals—a lot of which would be controlled in real time with the Mi.Mu gloves.” 

When the shift to remote learning happened in March, the project quickly evolved into a narrative video game called Dream Machine. With the help of consultants, and under the supervision of van Troyer and Landay, students developed software for an online multiplayer environment in Unity, a game engine platform. “We had always thought there would be a game component, but that the performance would happen in some representation of the game environment,” said Landay. “Then the game environment became the whole environment and the questions became: What are the game mechanics? What are the performance mechanics? And who is the player?”

“For students and musicians to be doing this, coding in an undergraduate program that’s not a computer science program, was pretty remarkable."

—Michael Bierylo, chair of the EPD Department 

After a six-week sprint to develop Dream Machine—during which Hendryx often said, “we were building the airplane as we were flying”—50 people played the game simultaneously, navigating through the game world as avatars as part of the performance experience hosted on Twitch.tv in late May. “For students and musicians to be doing this, coding in an undergraduate program that’s not a computer science program, was pretty remarkable,” said Bierylo. 

The EPD Department also collaborated on the project with accomplished Dutch multimedia artist Chagall, who performs frequently using the Mi.Mu gloves. Chagall had spent a week on campus as a visiting artist working with students just before the shutdown began. A return visit was planned for later in the semester, but when that became impossible, she motion-captured herself performing some of the songs she would have done in person, and the students created videos to her performance. 

“I do use a lot of technology; there are the gloves, sensors on my body, and visuals on the screen,” said Chagall. “The reason is because I feel there is a need for electronic music to be performed in a way that allows the audience to connect emotionally with what’s going on inside the computer and with the music itself. So, for me, bringing people together in live music is really important, and that’s exactly the thing we’re all not allowed to do right now.”

Bierylo sees the Dream Machine project as a prototype going forward. “This is a way that we can work as artists and distribute our work in a global way,” he said. “We can have it be entertaining, we can have it be interactive, we can have it be all of the things that we originally thought of for a live performance. This was a living laboratory for one kind of future for performance.” 

Watch a video representation of Dream Machine featuring Chagall: