There are lots of tasks in biomedicine that are more accurately and efficiently accomplished by a computer than a person — take the detection of breast cancer from a mammogram, as just one example.
But there are also plenty of health- and disease-related issues that no one has been enticed to solve with an artificial intelligence algorithm. Or there’s a bunch of models tackling the same problem and it’s unclear which one works best.
Biomedical challenges can help.
These contests allow everyone from high school programmers to pharmaceutical companies to craft the best machine learning- or AI-powered model to solve a medical riddle. Some have prizes while others offer bragging rights and the satisfaction of helping advance science.
Now there’s a platform that corrals and promotes these contests. Sage Bionetworks, a Seattle nonprofit, this month launched OpenChallenges, a free website where people offering and participating in biomedical challenges can find new events and match-ups.
There are currently 11 active and upcoming contests on the site, and 270 completed events. They include predicting bipolar disorder by sifting through snippets of genetic sequences, recognizing abdominal trauma in CT scans, detecting parasites in cells, and predicting the odor of a molecule based on its physical properties — among more esoteric tasks.
“The really cool thing is that challenges are an interface between two communities that don’t speak with each other that much and really should be working together a lot,” said Luca Foschini, president and CEO of Sage Bionetworks. Those communities are the scientists working on health issues, and the AI-engineers who build the algorithms.
Here’s how the challenges work:
- An organization such as a research nonprofit, universities, and pharmaceutical and other companies creates a contest and finds a sponsor to pay for the match-up and sometimes offer a prize.
- Participants build an algorithm, training it off of public or proprietary datasets.
- Participants submit their algorithm in a “black box” to ensure the privacy of both the proprietary datasets and the model.
- Organizers take a dataset procured from a health system or other third party and test the algorithm. The dataset has been checked by humans or “ground proofed” — in the case of breast cancer, for example, researchers have verified by biopsies which images contain a cancer — so the correct answers are known.
- The winning algorithm sets a benchmark by which to measure other approaches. (There are also benchmark data sets that are the standard for testing algorithms.)
In addition to creating OpenChallenges, Sage Bionetworks also hosts its own contests. The nonprofit is eager to create a standardized process for the logistics of setting up the challenges, which can be labor intensive and expensive.
It can also be deceptively difficult to successfully evaluate biomedical models.
“Up until now, machine learning has mostly been about accuracy: How well can you can you match the ground truth?” Foschini said. “People are starting to realize that’s not the only thing that matters. Safety matters. Equity matters.”
An algorithm might, for example, perform well on patients in general, but poorly when looking at subsets such as people of a certain race.
A new difficulty that’s emerging is the use of time- and computing-intensive large language models for the biomedical analyses. While these models might produce great results, the solutions also need to be feasible and practical for research and commercial use — and for the contests themselves. The cost of that kind of computing adds up fast.
“That’s what keeps me up at night,” Foschini said. “How are we going to move into the era of large AI models that are so costly to just run once to evaluate [it]?
