Bill Zhou wanted to be able to help his mom fix the WiFi router when she called. The only problem? He was at school in Berkeley, Calif., and she was not.
“So I try to send her links online or send her videos or try to do a phone call with her, but it’s not really clear,” Zhou said. “And sometimes I wish I could just teleport my presence back home just for five minutes, show her what’s going on, and then teleport back to Berkeley to do whatever I’m doing.”
That personal desire was part of the inspiration for Pengram, an augmented reality tool for remotely assisting and collaborating on projects such as fixing equipment or assembling furniture. The Pengram team, made up of University of California, Berkeley graduate students Zhou, Vedant Saran, and Will Huang, will be one of 49 teams competing in the world finals of Microsoft’s Imagine Cup student competition starting Monday in the Seattle region. Imagine Cup brings together high school and college students who are “innovating and addressing some of humanity’s biggest problems.”
Pengram uses both augmented and virtual reality to “holographically teleport” an expert to assist on a task involving a physical object in another location. The expert, wearing a virtual reality device, can work in the virtual world on a virtual model of the object that needs fixing, such an engine. Whatever the expert does to that virtual engine will be reflected on the other person’s side, except in augmented reality, with an avatar representing the expert demonstrating on the physical object.
“The operator will be able to see the expert as if he was actually there,” Zhou said.
Saran said the platform uses Microsoft Azure to deliver the content via the HoloLens device. Pengram allows users to watch experts in real-time or in previously recorded videos.
Though personal use was one part of the team’s vision for Pengram, inspiration also came from what the team noticed businesses needed. Companies worked with much more complicated machines, like wind turbines or locomotive engines. Zhou said that often, to repair the machines, they would have to fly out an expert because field technicians wouldn’t know how to fix them.
“So what they’re looking for is actually a remote assistance solution where the expert can teleport their presence to the field to assist their technicians anywhere in the world,” Zhou said.
Pengram has worked with companies like smartphone maker HTC to explore the possible uses for the platform. Zhou explained that any company could use Pengram’s capabilities in a unique way suited to their needs. HTC, which supports trade schools in China, finds pre-recorded assistance helpful in training students.
Flashes of Pengram’s capability can be seen in Microsoft’s own Holoportation project, which Microsoft revealed in 2016. Holoportation, like Pengram, uses the HoloLens as a tool to holographically transport 3D models into a physical space in real time, as if all participants were in the same space. In another demonstration, Microsoft showed how someone using a tablet in one location could annotate the real world for someone using a HoloLens in another, such as a plumber showing a homeowner how to fix a sink.
The Pengram team, who have known each other for three years and met through the VR@Berkeley club when they were undergraduates, began the project at a Cal Hacks hackathon a year-and-a-half ago. Pengram won the Microsoft Imagine Cup U.S. Finals to advance to the world finals.
On the other side of the world in Pakistan, Iqra Irfan, Areeba Kamil, and Sami Ullah are developing a wearable belt that monitors fetal health. The team, named Fe Amaan, consists of three undergraduates in their last year at the National University of Sciences and Technology. They wanted to tackle Pakistan’s miscarriage and stillbirth problem, which they described as one of their home country’s biggest issues.
“One of the major issues we found in the healthcare facilities in our country is that there is not enough access to facilities for expecting women,” Kamil said. “And the women who have to suffer the most are women in rural areas, and then they become the target of stillbirths. Later on we also realized that this problem is not just confined to Pakistan, but it’s also a worldwide issue.”
Fe Amaan works as a remote fetal monitoring device, helping ease the consequences of a lack of access to medical facilities. The belt and corresponding Internet of Things sensor device, which sits on the mother’s abdomen, can monitor fetal movements and heart rate. It sends the data to a mobile app, which analyzes it and generate alerts if it detects any anomalies. The device uses Microsoft’s Azure cloud platform to host its applications and to predict the state of the fetus in advance, based on the data gathered.
The hope is that precautionary measures can then be taken before it’s too late.
“We believe it’s the right of every woman to have good medical facilities and we want to make sure it’s our aim to eliminate the risk of having a stillbirth,” Irfan said.
The three have were friends prior to the project, and decided to work on Fe Amaan as part of their senior projects for university. Fe Amaan has gone through clinical trials, which the team cites as the most difficult part of the process. The team participated in the Pakistan national finals and won the Middle East and Africa finals to advance to the world finals.
The Microsoft Imagine Cup World Finals will take place in Seattle next week, from July 23 to 25. The annual student technology and innovation competition requires participants to submit their software, instructions, and give live presentations on the team, the project, the target market, and how the team plans to bring the project to market.
Forty-nine teams, including Pengram and Fe Amaan, will compete on the world stage after winning national and regional competitions throughout the year. The winning team will get $100,000 and a mentoring session with Microsoft CEO Satya Nadella.
This year’s Imagine Cup, the 16th annual competition, includes awards for projects in artificial intelligence, big data, and mixed reality. The judges include Microsoft executive vice president of business development Peggy Johnson, coding community Glitch CEO Anil Dash, and software package management company Bitnami co-founder and COO Erica Bresica. Snowboarder and Olympic gold medalist Chloe Kim will also be a special invited guest at the competition.
