New York City’s Times Square is already a bustling and chaotic location when experienced in everyday reality. In augmented reality, the center of Manhattan will become otherworldy, thanks to an artistic endeavor that relies in part on Microsoft HoloLens.
In collaboration with New York-based arts organizations Times Square Arts, No Longer Empty and Queens Museum, Microsoft has helped develop a mixed reality work called “Unmoored” to serve as a companion piece to “Wake,” a large-scale sculpture by the artist Mel Chin.
The 60-foot-long “Wake” — created to evoke the hull of a shipwreck crossed with the skeletal remains of a marine mammal — blends with the digital world of “Unmoored” in HoloLens to make it look as if the skyscrapers and blinking neon of Times Square are being viewed from below rising waters. “Wake” also features a 21-foot-tall animatronic female sculpture, derived from opera star Jenny Lind, the “Swedish Nightingale,” whose figurehead was once mounted on the USS Nightingale, a 19th century clipper ship.
The exhibition opened Wednesday afternoon in New York. The HoloLens component will be available to visitors for just a few days, ending on Friday. But a mobile app showing the augmented experience on iOS and Android, will be available to Times Square visitors until Sept. 5.
In the HoloLens version of “Unmoored,” users will watch as “Wake” transforms from a shipwreck into a full-scale model of the Nightingale, floating down Broadway. Other animated boats appear, arriving in Times Square from adjacent streets and eventually bunching together to create a gridlocked flotilla. Sea life also begins to appear.
Through the app, users get a similar experience, but must rely on a map to locate one of six nodes on the plaza adjacent to “Wake” in order to align their mobile device’s camera view. “Wake” transforms, more boats arrive and bump into each other and sea life appears to draw the user’s attention downward. The jarring view is meant to challenge our assumptions about what type of future awaits us, especially in the wake of climate change and rising sea levels.
The Times Square undertaking is part of Chin’s “All Over the Place,” a multi-location exhibition which expresses themes prevalent in the 67-year-old artist’s decades of work.
Microsoft’s collaboration with Chin was over a year in the making, and teams from both sides first met to understand the vision and key challenges.
“We were immediately drawn to his idea of virtually sinking Times Square under 30 feet of water and knew that mixed reality — specifically Microsoft HoloLens — could be a compelling solution,” Ryan Gaspar, director of Microsoft Brand Partnerships, told GeekWire. “After showing Mel what HoloLens was capable of he knew that his vision could become a reality and ‘Unmoored’ was born. We then spent the next 12 months working hand in hand with Mel’s team to bring ‘Unmoored’ to life.”
While Microsoft is no stranger to partnering with artists and cultural institutions, creating a public work of art in Times Square proved to be an incredible undertaking with its own unique challenges.
“From a technical standpoint, developing a consumer-facing HoloLens experience to work in a heavily trafficked, outdoor, and well-lit — both natural sunlight and artificial LED screens — setting was difficult given that wasn’t ever a primary scenario meant for the technology,” Gaspar said.
Gaspar said “Unmoored” highlights the potential of HoloLens and mixed reality. But the opening sequence might steal the show.
“The entire experience is breathtaking since it’s not every day you get to see a nautical traffic jam 30 feet above your head in Times Square,” Gaspar said. “That said, I believe the most compelling aspect is right at the beginning when the physical ‘Wake’ structure comes to life before your eyes and begins to rise. It’s a magical moment where the physical and virtual worlds collide to create a new reality.”