Baseball players are judged by their batting averages, RBIs, ERAs and other statistics voluminous enough to fill a scorecard. But how do you rate researchers?
The classic measure is citations: that is, who’s quoting whom in their research papers. But just as in baseball, the statistics are becoming more nuanced. Now Semantic Scholar, a scientific search engine developed at Seattle’s Allen Institute for Artificial Intelligence, is introducing a whole new set of stats.
If you’re in the “publish-or-perish” game, get ready to find out how you score in acceleration and velocity. Get ready to find out who influences your work, and whom you influence, all with the click of a mouse.
“We give you the tools to slice and dice to figure out what you want,” said Oren Etzioni, CEO of the Allen Institute for AI, a.k.a. AI2.
Etzioni and his colleagues have been working on Semantic Scholar for months, but the beta version that was rolled out today is the most ambitious reboot yet. Today, Google Scholar sets the standard for scanning and organizing the wide sweep of academic research: If you know how to construct a search, you can zero in on the studies that are most relevant to what you’re writing about.
Other measures have cropped up: For example, there’s a statistic known as the h-index that measures the productivity and impact of given researchers or journals.
Etzioni hopes that Semantic Scholar’s scales will come to be seen as more user-friendly. Velocity, for example, measures how many citations a given paper has picked up over the past three years.
Acceleration keeps track of how quickly those citations are coming. “It might not be a recent paper, but it’s getting hot,” Etzioni explained. “People are discovering it.”
Authors get profile pages that graphically show who has had the biggest influence on their work, and who they’re influencing the most. Influence can be a tricky thing to measure, Etzioni said, and the algorithms have to be tweaked.
The results vary, based on the tweaking. For example, if you go by raw citations, the top researcher in computer science is Berkeley’s Scott Shenker. But Semantic Scholar suggests that a colleague of his at Berkeley, Michael Jordan, is No. 1 in influence.
The fact that the guy has the same name as one of the world’s most famous basketball players isn’t lost on Etzioni. “Some people refer to him as the ‘Michael Jordan of machine learning,'” he joked.
The issue isn’t merely academic, so to speak.
“When there are hiring decisions and promotion decisions to be made, people are hungry for data,” Etzioni said. “We’re starting to offer the community better data about somebody’s impact.”
The algorithms used to tease out connections in the academic world can also be applied to wider fields: For example, Google Scholar’s analytical tools are related to the PageRank algorithms that Google uses to weigh search-engine results. The new approaches pioneered by Semantic Scholar could someday help determine what’s catching fire on the Web or in social media.
For now, Semantic Scholar focuses only on computer science research, but Etzioni said the spectrum will be widened to include neuroscience in November, and other fields in years to come. He said the project fits in with the Allen Institute’s aim of putting artificial intelligence at the service of the scientific community.
“It’s really part of our mission of ‘AI for the common good,'” Etzioni said.