Problems nag Redfin’s Scouting Report, CEO says ‘we screwed up’

Redfin CEO Glenn Kelman apologized today for the poorly conceived launch of Scouting Report, a new online service designed to provide vital statistics on more than one million real estate agents across the country. As we reported last Friday, Redfin suspended the service in Washington, D.C. after the company’s MLS data provider raised questions about Scouting Reports.

Now, in a blog post today, Kelman says that they are disabling the service in Sacramento, Atlanta and parts of the San Francisco Bay Area.

Kelman lays out a series of problems with the service, apologizing for what he classified as “big mistakes.” He writes:

“Surfacing millions of records that have never seen the light of day before is always a bit like opening Pandora’s Box. But it’s worth it in the end if you can get it right. That’s what we’re trying very, very hard to do, working late in the night and through the weekends. Again, we are sorry for all the ways we screwed up in presenting data from dozens of feeds, based on thousands of rules.”

Glenn Kelman (Randy Stewart photo)

Kelman has always been quick to admit mistakes, but the series of blunders around Scouting Report appears to be taking a toll on the online brokerage service.

Some of the problems were of a technical nature, and Redfin is taking steps to correct those errors. But the company also miscalculated whether it could access the data from its MLS providers.

The Scouting Report service is used to show statistics for real estate agents such as the number of homes sold; the average number of price drops for each home sold; the median sales price; the range of prices; how often the agent represented buyers and sellers and more.

One agent lambasted Redfin in the comments of the blog post, dubbing the roll out of Scouting Report a “disaster.”

“I’d have been ok with my stats being published if this had been accurate from the beginning, but it’s far from it, and you’ve burned a lot of bridges with the agents about whom you’ve published inaccurate information,” the agent wrote.

  • http://blog.redfin.com GlennKelman

    John! I meant it when I said we made some big mistakes but let’s not go too far my friend: I wasn’t suggesting that Scouting Report was, at launch, “poorly conceived.” We have been reflective about the design, but we humbly submit that from our perspective, the primary problem was in the execution, not the conception; it shipped with some bugs, one serious, all of which we believe we have immediately fixed.

    And I certainly don’t think the feature is, as you claim, “taking its toll” on Redfin’s overall business. Had you interviewed me about this, I’d have told you the feature has had no impact yet on our revenues or profits, and it has increased traffic. Consumers like it.

    Maybe you were worried about our reputation among agents with whom we cooperate and compete. I know what you mean here, and read your post feelingly: bugs hurt and our peers matter. But do you think bugs are the reason some agents like the feature, and some don’t? It is safe to say that it isn’t our critics’ main issue; the “concept” of transparency about agent performance is.

    You have been covering Redfin long enough to know that some agents at other brokerages have been vocally opposed to almost every Redfin innovation, including most recently Agent Insights, in which we publish our agents’ observations about homes we’ve toured. But we now have competing agents asking us to post Agent Insights on their listings.I hope this will happen here: many agents will also appreciate Scouting Report because it shows just how good they are, oftentimes better than Redfin agents; many already appreciate it, sometimes publicly in blogs. And of course consumers are thrilled.Our obligation to other agents is to be fair, professional, truthful and humble — which is a challenge when you’re dealing with one million agents — but beyond that Redfin as a business serves consumers, and only consumers.What surprises me about your post is that you highlight the tenor of the situation as dire, without actually identifying the problem: that for hours we miscalculated how long it took agents to sell listings. This is a serious problem, and one for which we apologized to the affected agents, but apparently not serious enough to show up in the story. For the purposes of this post, the news is that I apologized, not why.Other problems have surfaced but they have affected less than 1% of the overall agents in the system and none distorted agents’ performance. You reported that we are taking steps to correct those problems but actually we have already shipped bug fixes.We’re not so arrogant as to assume that we’ve fixed every bug, but we have at least shipped fixes to the known issues. And we are still listening to the community of agents, and discussing whether everything we have done has been fair; your post didn’t address this issue but I have worried about how we can deliver innovations to consumers while still being humble members of our community. This is hard. But what I like about Redfin is that we try to do hard things.Beyond that, your characterization of our own “blunders” seems to be conflating bugs in our software with a larger issue beyond our control: the data licensing rules of the Multiple Listing Services from which we draw data. When an MLS determines that Scouting Report needs further review, without citing any specific data-licensing violations, is that a Redfin blunder? Is it an MLS blunder?It’s probably nobody’s blunder. Many MLSs have affirmed that the feature is within the rules, and even the ones who have asked for further review may yet approve the feature. This for us is a good if imperfect result so far.It is a fact of our business that many features are available only in some markets; this is true for example of the pictures of past sales, which are not available in Atlanta, Sacramento and many other markets that we serve. It is true of Agent Insights which is not available in full form in many markets, including Seattle. It has been true of Scouting Report since it launched, with a blog post explaining which markets would allow it. We knew the Scouting Report was risky when we launched it or we wouldn’t have designed it to be easily disabled market by market. Given the risks from our data providers, would you have preferred that we not release it at all? That certainly isn’t our point of view.This blog can embrace a startup ethos of taking risks and acknowledging mistakes, or it can blow up those mistakes into “blunders” and from there into an outright “disaster.” If you take the first approach, you don’t have to be a startup cheerleader — Redfin deserves criticism in this case — but we do want you to understand how small companies approach shipping software: we take risks with disruptive features, we do our best to test beforehand, we jump all over bugs, we tell the world exactly how and why we screwed up, but we keep moving forward.If you want to take the second approach of reporting this like a breaking scandal for a larger venture-backed company in Seattle forced by the mob to concede an apology, you have to interview the subject of your story. What we would have said in an interview is that we have made mistakes but Scouting Report isn’t, as you suggest, a disaster. It is, for consumers, a delight. If you lose sight of that, you’ve really lost your way.

    • johnhcook

      Glenn. I’ve always appreciated your willingness to share the good and bad at Redfin. I don’t know many startup CEOs who do that, and you should be applauded for it. 

      I do not claim in the post that the product/concept was poorly conceived, but the launch of it was. (Something that reader Sam DeBord also notes below). That’s the tenor of the post, echoing what you wrote in your blog post. 

      In fact, in my last couple posts on Scouting Report, I talked at length about how it works and I quoted you at length about the promise of the service. It may be the best thing since sliced bread in the real estate industry, but there are problems in terms of the data collection and access to that data which you openly address in your blog post. If the data is inaccurate, unavailable or incomplete, consumers won’t trust it or use it. 

      I could have rehashed the technical problems — which I say above you have fixed or are in the process of fixing — but I believe you addressed that in great detail in your blog post. I didn’t think it was fair to go through that laundry list, and I thought you provided a good analysis of the data collection issues. That’s why I provided a link to the post, and included a quote from you from it. 

      In terms of using the phrase “taking its toll,” I did not mean that the mistakes were impacting Redfin’s bottom line. But my belief (and you can fault me for sharing it) is that Redfin has worked really hard over the past few years to reverse an impression among agents that the company is at odds with the rest of the industry. You know better than I that Redfin must work closely with other agents, and relationships do matter. Based on some of the feedback I’ve been reading — including comments on your own blog — I wonder if you’ve taken a few steps back. 

      The early problems you’ve encountered also impact consumers’ decisions and whether they think the data is reliable enough to use.    

      That’s not to say Scouting Report is doomed or a blunder that will torpedo the company, but mistakes were made in the launch of the product. You freely admit that. And I thought my post conveyed it. 

      One last thing: The startup ethos of “failing fast” does not give entrepreneurs a free pass. And I don’t think you are suggesting here that I should not have covered the mistakes you made, simply that more detail should have been provided in what went wrong. 

      Fair enough. As you know, it is a balance trying to pack all of the facts into a tight, reader-friendly story, but I’ll try to do a better job of providing more of those details next time. 

      Thanks again for the comment. 

  • http://SeattleHome.com Sam DeBord, SeattleHome.com

    Glenn has been very open to the criticism, and that’s admirable.  It doesn’t change the way the service was launched, however.  If it had been tested more internally instead of in the public eye, and the data providers were consulted on proper usage of their data, this might have gone well.  It’s disappointing, in that the tool may have been groundbreaking and useful for those that chose to use it in a responsible way.

    For now, MLS boards in multiple cities are shutting down Redfin’s data feed for violating agreements and agents are waking up to shocking misrepresentations of their work in Google search results.  This is not how good relationships are formed in the industry.

    • http://twitter.com/ShaunWolfe ShaunWolfe

      Sam– that MLS boards have determined that Redfin has violated their agreements is breaking news, then (if true).  Hopefully John will be interviewing you soon to post a blog to get the scoop on this.

      • http://SeattleHome.com Sam DeBord, SeattleHome.com

        Shaun, some MLSs have confirmed that Scouting Reports are within the rules, some have put a temporary hold for review, and others have confirmed that they are violating current rules.  Glenn has pointed each of them out in the above-referenced post today.  Each MLS sets its own rules.  It’s a boondoggle to navigate them on a national scale, but a necessity if you want to play on the big stage.

        • http://twitter.com/ShaunWolfe ShaunWolfe

          I have re-read the comments and cannot find where anybody has said that Redfin launched in a region and the MLS board determined it was a violation. All I can see is “further review, without citing any specific data-licensing violations”.   If you know of places where Redfin has launched this and it has then been determined to be a violation, I do think that’s news and worth John following up on.

          • http://SeattleHome.com Sam DeBord, SeattleHome.com

            Sacramento was the first (Redfin agreed that “Scouting Reports violates data-sharing rules”).  Southern California, Denver and Portland noted a violation for public display without registration.  Redfin is working on that. 

          • http://twitter.com/ShaunWolfe ShaunWolfe

            Thanks.

  • http://twitter.com/cgarre Chandra S Garre

    Whatever .. i love redfin, it is possibly the best real estate site .. it is addictive ! I like the fact that updates are sort of real-time, well done .. i will always be on redfin !

  • Henry

    May I just say that we really should just get rid of agents altogether.  It’s not difficult to sell or buy a house if the information is readily available.  Unfortunately, the cartel known as the NAR likes to make the process of buying or selling a house as clouded as possible just so the 6 percenters can charge us all through the nose.  An antiquated WW2 business model that needs to go the way of the dinosaur.  What is the solution to some person who cannot spell properly most of the time (most agents) picking my pocket for a unjustifiably huge sum of money, just for pushing a few pieces of paper around?  Most people find the homes they want to look at themselves and are then forced to call the agent.  Cut ‘em out altogether.