Google this morning sent a message to publishers who use its DoubleClick service to serve ads, explaining the reasons for a 90-minute outage on Wednesday that affected an estimated 55,000 sites, including the Wall Street Journal, Forbes.com, BBC.com and GeekWire, among many others.
In short, the outage was caused by a degradation in performance of an unidentified “internal service,” along with a misconfiguration that thwarted the safeguards that would normally prevent the DoubleClick for Publishers (DFP) ad servers from failing when that happens.
For all the jokes about Google inadvertently ridding the Internet of ads this week, the outage showed just how many web publishers rely on the company’s ad-serving system.
Here’s the full text of the message.
Dear Publisher,
As you are aware, our DoubleClick for Publishers (DFP) ad server experienced a global outage on Wednesday, November 12 from 5:45am to 7:31am PST. This was the first DFP outage of this significance for many years and we take it very seriously. Our team’s first priority was restoring service and making sure the immediate issue would not recur. We are writing now to explain what happened and what we are doing to protect against future incidents.
The details:
• The DFP ad server relies on an internal service that began degrading in performance. This caused a cascading failure on DFP ad servers, leading to the outage.
• We designed our systems to gracefully handle performance degradation from dependent services. However, due to a misconfiguration, we were unable to prevent the outage.
• To restore ad serving and prevent cascading failures, we restarted the services by provisioning additional resources.
• We reproduced the failure in a test by degrading the availability of the internal service, proving the misconfiguration caused the cascading failures. We have since rolled out a fix to the configuration globally.
• We are conducting a complete review of all our processes and production configurations to prevent this from happening again.As always, our goal is to provide a best-in-class product and the unmatched global scale and reliability you’ve come to expect from Google and DoubleClick. We value your partnership deeply and apologize for the disruption this incident caused to your business.
Best,
Neal Mohan – VP, Display & Video Advertising
Scott Silver – VP, Engineering
