What would it be like to see pictures of moonwalkers, comic-book characters and painted portraits get up and walk right out of their frames? It’s an eerie thought – but Photo Wake-Up, a software application developed by computer scientists at the University of Washington and Facebook, gives you an idea how it would look.
And someday, the app could come to an augmented-reality headset near you.
The project has been in the works for months. It won a share of the spotlight last November at UW’s annual Madrona Prize competition, and made another media splash a month later when the team put out a preprint paper.
Next week, the researchers will be presenting their results at the Conference on Computer Vision and Pattern Recognition in Long Beach, Calif.
The idea of bringing two-dimensional pictures into something like live action has been the stuff of movie fantasies for decades, going back to Gene Kelly’s dance with a cartoon mouse in “Anchors Aweigh” nearly 75 years ago. More recently, the Harry Potter movies and “The Ring” horror-film series have taken advantage of the trope.
Computer scientists have been working on the issue as well, and not just for entertainment purposes.
“This is a very hard fundamental problem in computer vision,” study co-author Ira Kemelmacher-Shlizerman, an associate professor at UW’s Paul G. Allen School of Computer Science & Engineering, said in a news release.
“The big challenge here is that the input is only from a single camera position, so part of the person is invisible,” said Kemelmacher-Shlizerman, who did some of the work on the project as a research scientist at Facebook. “Our work combines technical advancement on an open problem in the field with artistic creative visualization.”
Another author of the study, UW computer science professor Brian Curless, said some teams have tried to construct 3-D representations of photo subjects by combining multiple 2-D viewpoints. “But you still couldn’t bring someone to life and have them run out of a scene, and you couldn’t bring AR into it,” Curless said.
The UW team solved the problem by identifying the body components of a figure in an image, matching those components to a virtual 3-D template, creating a full-body model of the person, draping colors and textures over the model and then projecting it back into the 2-D image.
“It’s very hard to manipulate in 3-D precisely,” said co-author Chung-Yi Weng, a UW doctoral student. “Maybe you can do it roughly, but any error will be obvious when you animate the character. So we have to find a way to handle things perfectly, and it’s easier to do this in 2D.”
The computer algorithm fills in the data gaps in the image – for example, by creating a reasonable facsimile of the person’s back as well as other parts of the body that were obscured in the original picture. Photo Wake-Up also borrows image data from other parts of the picture to fill in the blank spaces when virtual figures walk out of their frames.
To demonstrate how it works, the team took a picture showing the Golden State Warriors’ Stephen Curry at a basketball game, and animated it to look as if he was running right out of the picture:
Similar liberties were taken with other sports pictures, a photo of an Apollo astronaut on the moon, a Banksy mural and a Beatles album cover. The subjects didn’t even have to be realistic to be “woke.” Cartoon characters and abstract-art figures painted by Pablo Picasso and Henri Matisse worked just as well, and the effect was even spookier.
The research team reported that their technique worked much better than other algorithms for turning 2-D still pictures into 3-D animations. There are several potential applications: Video gamers and illustrators could easily turn their artwork into animated characters. A museum could create an augmented-reality experience that lets patrons have tea with a virtual Mona Lisa. And kids could click a mouse to turn their digitized drawings into custom-made cartoons.
“Photo Wake-Up is a new way to interact with photos,” Weng said. “It can’t do everything yet, but this is just the beginning.”
The algorithm doesn’t work so well if the subjects aren’t facing forward, and if the people in the picture have their legs crossed, it won’t work at all. But maybe we should be grateful that the program isn’t perfect: If Photo Wake-Up gets too good, that could add to the “deep fake” problem we’re already facing with altered videos.
