The Allen Institute for Artificial Intelligence says it has built a machine that can solve geometry questions from the SAT about as well as an average American 11th-grade student.
It’s a bigger feat for a computer than you might imagine, and represents a major breakthrough in AI research by the Allen Institute, founded by Microsoft co-founder Paul Allen.
The system, called GeoS, gets about 49 percent of questions right, which extrapolates to about a 500, out of a possible 800, on the SAT’s math test. That’s pretty solid amongst its robot peers and was the average human score in 2015.
You can test your own math skills head-to-head against GeoS online here. The demo underscores how even basic concepts required for the test become more challenging when they’re not being interpreted by a human brain.
Take this question for instance:
In the figure to the left, triangle ABC is inscribed in the circle with center O and diameter AC. If AB=AO, what is the degree measure of angle ABO?
GeoS first has to be able to understand the natural language used in the text.
Then it uses computer vision to understand the diagram so it can compare the two in order to figure out what the test is asking. Once it understands the question, the computer can finally do the math and compare its answer to the multiple choice options.
Along the way, concepts that are automatic to a person taking the test have to be translated into formulas that are meaningful to a computer. For instance, “what is the degree measure of angle ABO?” becomes “Is(MeasureOf(ABO), What).”
“Much of what we understand from text and graphics is not explicitly stated, and requires far more knowledge than we appreciate,” said Allen Institute CEO Oren Etzioni, in a statement announcing the results. “Creating a system to be able to successfully take these tests is challenging, and we are proud to achieve these unprecedented results.”
The research has been published online in a paper titled, “Solving Geometry Problems: Combining Text and Diagram Interpretation.” It was a joint effort between the Allen Institute and the University of Washington Computer Science and Engineering department.