Caroline Stefani has a cool job: She works at the Benaroya Research Institute at Virginia Mason in Seattle, looking for new ways to treat diseases like multiple sclerosis and Type 1 diabetes. To do that, Stefani spends a lot of time taking pictures of tiny, tiny cells through a microscope, then trying to imagine those images in three dimensions.
It’s not easy. In fact, it turns out a lot of scientists have a hard time wrapping their minds around the tiny, three dimensional structures that they work with every day, like cells and proteins.
Luckily, Stefani’s office is right next door to Tom Skillman, who leads Benaroya’s research technology efforts. Talking with Stefani over drinks one evening, Skillman had an idea: using data from microscopes, he could build a full 3D model of the cells she studies — in virtual reality.
A year later, the lab that Stefani works in has a fully-operational VR program that lets her and other researchers see their work like never before. It’s just one of many ways that virtual and augmented reality are making waves in the world of medicine and medical research.
Adam Lacy-Hulbert, who runs the lab Stefani works in, says the program gets to the heart of a scientist’s work. Although it can seem flashy, medical research mostly means trying the same thing over and over — and over — again.
“A lot of it is looking, noticing, and trying again,” Lacy-Hulbert said. His lab studies autoimmune diseases, where a person’s immune system attacks parts of their body. He said the scientists spend a lot of time trying to understand exactly what’s going on in a single cell or an interaction between cells.
“That’s what’s been so cool about it — it’s almost immediate,” he said about the VR program.
Skillman, who comes from a technology background, said it has been incredible to see the scientists at Benaroya use the program.
“I wish we could capture one of the times when one of these scientists look at it for the first time,” he said. “They’re just giddy with excitement because they’re seeing things they never saw before.”
Here’s how the program, called Confocal VR, works: It starts with images taken in a process called confocal microscopy. Those images look like the one below, a 2-dimensional slice of the cell taken on different planes. It’s similar to an MRI that you would get in a hospital.
These images are of a dying cell, in green, being eaten by an immune cell. The immune cell’s nucleus is blue and its digestive tract is red.
Although the images help us understand roughly where the cells are and how they are interacting, it’s hard to get a sense of depth or see precisely what’s going on. Even a 3D compilation of the images is challenging to understand through a computer monitor.
Those images are then transferred to Confocal VR, which Skillman built using Unity, the popular video game making software. The process is fast: Lacy-Hulbert said it basically works in the amount of time it takes for him to walk from the microscope to the VR headset.
“You have a quick look at your sample in the microscope, and then almost the first time you actually interact with it is in VR,” he said.
Skillman and Stefani worked together on the program, alongside another researcher in the lab, Mridu Acharya. They used what the tech world calls rapid prototyping.
“I would build something, give it to Caroline. She would use it, give me feedback. I would go off, adjust it, and we just iterated as fast as we could,” Skillman said.
The end product is rather stunning: Not only does the program show scientists cells in new ways, it’s also incredibly intuitive to use.
Personally, I’ve never used a confocal microscope before, nor have I studied the immune system in any depth. When Stefani explained the details of the 2D image, I had trouble following her. But as soon as I put on the lab’s Vive headset, I instantly understood the details she had described. I started noticing more things and asking the researchers questions about different parts of the cells.
You can see a demo of what I saw in the video below:
The program also lets scientists control and manipulate the images to glean more insights. Confocal’s VR space includes a control panel with sliders to adjust things like brightness, contrast and lighting, also very intuitive to use.
The intuitive nature of VR is one of its strengths, particularly in the medical field, where people are constantly trying to understand highly complex material.
The technology is also gaining traction in hospitals and other medical settings. One example of that integration is Seattle-based startup Pear Med. The company is developing a mixed reality program that can create 3D, interactive models from a patient’s scans, effectively showing a patient and their doctor a 3D version of their organs and other pieces of anatomy.
Pear Med CMO Dr. Stephen Seslar said the goal of the tool is to “build a better mental model of the patient’s anatomy through better visualization and a 3D interaction with the relevant tools,” in a medical setting, like an operating room.
The goal is both to give patients a better understanding of their health and help doctors prepare for complicated operations.
“We envision a day very soon when our technology will be used to help visualize a patient’s anatomy in real time during the clinical procedure,” Seslar said.
On the other end of the spectrum, Benaroya is also developing a VR program that lets scientists interact with protein structures in VR. These highly complicated molecules are responsible for most of a cell’s functions.
The program, called AltPDB, lets multiple people collaborate in a VR “room,” and the party is open to the public: Anyone with a VR headset and a free AltSpace account can try it from their own living room. There are also plans to share Confocal VR with nonprofits organizations.
Whether tiny proteins, small cells or whole organs, mixed reality helps us see things in a new way.
Lacy-Hulbert said every scientist he knows who has tried Confocal VR has been blown away. Often times, they see things they had never noticed before, even though they have been studying these kinds of cells for decades. The image of an immune cell eating a dead cell is a great example.
You can see in the 2D image that part of the cell’s digestive tract seems to go through its nucleus. Scientists could see from those images that something strange was happening, but they weren’t sure what it was.
In VR, “you can grab it, and turn it and get it at exactly at the right angle and stretch it and go, ‘yes, those things are right in there,” Lacy-Hulbert said.
“That’s important because previously people had interpreted that in a different way,” he said. Scientists thought certain proteins had been actually invading the cell’s nucleus. “What we can now tell is that they are two different structures, but one is pushing one out of the way.”
In some cases, that detail might be a small change. But in others, it could mean the success or failure of a new treatment for allergies, a life-saving drug for MS or any number of other medical advances.
