Now that the dust has settled, we can ask how could the smart and savvy team at Zillow Group, armed with cutting-edge AI methods and mountains of data, lose half a billion dollars buying homes in the second half of 2021. This post identifies two Achilles’ heels that led to this debacle: the “inside information” advantage of homeowners coupled with unanticipated changes in home prices over time.
It’s reasonable to ask whether statistical/AI-based price prediction can ever work. The answer is a resounding “yes” based on the success of Wall Street “quant funds” and on their extensive hiring of computer scientists, statisticians, and more recently deep-learning experts.
Farecast, the startup I founded in 2013, was able to predict airfare fluctuations with 75% accuracy before it was acquired by Microsoft in 2018.
Finally, Zillow’s home-price “Zestimate” has a reported average accuracy of roughly 96% nationally, and closer to 99% on homes listed for sale.
The Zillow team knew the accuracy of their pricing algorithm in different regions of the country, under different market conditions; they had the opportunity to carefully back-test its performance on historical house-buying transaction data.
So, what went wrong with Zillow’s home-buying business, also known as “iBuying”?
Simulated success on historical data does not imply success in buying from homeowners for a simple reason: many homeowners have a better sense of the value of their own home due to variables outside of Zillow’s model (e.g., the house has a strong odor or other idiosyncratic characteristics that repel would-be buyers).
An analogy to the stock market can help illustrate this point. It is widely acknowledged that company insiders (e.g., its CFO and CEO) have an information advantage relative to outsiders. As a result, they are only able to trade their company’s shares during restricted periods and a record of their transactions is closely watched by other investors.
When it comes to a person’s home, the homeowner is the ultimate “insider” whereas Zillow is limited to objective factors such as number of bedrooms, the price of similar transactions, etc.
When Zillow’s offer is too low, homeowners are likely to reject it, but when Zillow’s offer is too high, they are likely to accept it — capitalizing on Zillow’s errors. As a result, even when historical backtesting of a pricing algorithm indicates a profit, the results in the real marketplace could be quite different.
The technical term for this situation is “adversarial machine learning.” When a machine-learned model is trained and tested on a historical distribution of data, but used on a different distribution, errors can result, particularly when that different distribution is defined by homeowners (who are Zillow’s adversaries in this case).
Of course, homeowners are wrong at times, but they (and their real estate agents) know much more about the house, the neighborhood, and the reactions of potential buyers, which gives them an information advantage over Zillow’s algorithm.
If Zillow has an information disadvantage, why didn’t its disadvantage show up during Zillow’s early iBuying transactions?
Before buying tens of thousands of houses, Zillow tested its approach over three years achieving profits that emboldened Zillow to rapidly scale up its iBuying business. The answer is that home prices were moving up rapidly during that period. As a result, even if Zillow’s purchase was modestly over-priced, Zillow was able to make a profit when the house was sold some weeks later. However, when the market cooled down in 2021, Zillow’s disadvantage was starkly revealed.
In other words, the proverbial tide went out, and we found out that Zillow was … exposed.
“Fast decisions will become increasingly popular over time, but the Zillow debacle is an important reminder that McDonald’s is no substitute for fine dining or a home-cooked meal.”
When a real estate market cools down, many homeowners hold on to their home, giving the market a chance to recover. However, this wasn’t an option for Zillow due to the capital intensity of the iBuying business and the potential for increased losses if home prices declined further.
Zillow’s pricing errors can occur both when buying and when selling a home, resulting in pressure on profits. Moreover, transaction costs due to agent commissions, mortgages, and remodeling costs make it harder for Zillow to achieve iBuying profits. Of course, Zillow can’t share specific information about its transactions publicly, so these observations cannot be confirmed directly.
Much has been made of Zillow’s losses, but we need to remember that the homeowners transacting with Zillow gained hundreds of millions of dollars. In this case, human decision makers outsmarted AI.
Thinking broadly, the speed and scope of AI algorithms, armed with tons of data, has birthed a “fast-decision industry” where loan decisions, credit card approvals, product prices, and even parole decisions are increasingly automated. This fast-decision industry can be likened to the fast-food industry. Fast decisions will become increasingly popular over time, but the Zillow debacle is an important reminder that McDonald’s is no substitute for fine dining or a home-cooked meal.
Editor’s note: This post was written by Oren Etzioni, CEO of the Allen Institute for Intelligence. Etzioni was on the Zillow Technical Advisory Board when the company launched. Nothing in the post was based on any internal company information or knowledge.
