[Love sports and technology? Sign up for our sports tech newsletter to receive this rundown from GeekWire’s Taylor Soper via email every week.]
TAYLOR’S TAKE ON THE WEEK IN SPORTS TECH: Each Sunday for the past four months, millions of fantasy football users fired up their Yahoo apps and browsers, setting lineups and checking their team scores.
It takes a ton of technology to ensure that Yahoo’s tech platforms can withstand huge spikes in traffic and provide constant live scoring updates. For this week’s sports tech newsletter, GeekWire caught up with EJ Campbell, a senior director of engineering for Yahoo Fantasy Sports, to learn more about how he and his colleagues use the latest cloud-based tech tools every week during the NFL season.
“We put an incredible amount of effort into building resilient system,” he said.
Beyond updating scores and player data — more than 100,000 personalized notifications per second during peak times — Yahoo also provides fairly in-depth data-fueled research tools for users and automated matchup recaps.
Some of the data Yahoo shared also reflects a larger trend of more people using smartphones vs. desktop:
You can notice the spike on Sept. 17, a Sunday, when Yahoo sees its most usage during the season. Read on for an edited Q&A with Campbell, who offered a behind-the-scenes look at the technology powering Yahoo’s fantasy sports platform:
GeekWire: Overall, from a tech perspective, what are the keys to keeping all of Yahoo’s fantasy football services running on a Sunday?
EJ Campbell: “We put an incredible amount of effort into building a resilient system. With Kubernetes, nodes are continually monitored via extensive health checks and readiness probes. If a node is detected as ‘bad,’ it is quickly shut down, with traffic moved automatically to another system. For non-Kubernetes systems, we have an automatic health checking system, ‘Halo,’ that runs alongside our systems that attempts to automatically repair a system when a fault is detected.
Before the season starts, we run very high-scale load tests to exercise our system’s functionality and ensure there are no regressions from the previous season. We have engineers that scrutinize every web service request our apps and sites make to ensure they are all optimal and there no unexpected calls on high traffic pages.
Beyond building resilient systems, we have an extensive set of runbooks, feature flags and procedures in place if a problem does occur. We use the notion of ‘Defcon’ level to communicate within our team the state of the system:
For each stage, we have a practiced set of remediations that we employ to ensure our users continue having a great experience. For example, if we see our site’s average response time is beyond 400ms, we will reduce the refresh rate from every 15 seconds to every 60 seconds on Matchup pages, lowering load on our system.
The final key to smooth Fantasy is the care and attention of the engineers running the site as well as the hundreds of engineers around Oath who support the team. During peak ‘Draft Season’ — the four weeks leading up to the season — we have engineers on site seven days a week and in the evenings, ensuring we can quickly react to any faults in our system, as well as help customers if they have a problem with their draft. During the first few weeks of the NFL season, it is all hands on deck during NFL games to ensure our system can handle any changes in load or user behavior from the previous season. This is an example of part of the team just before kickoff on the first Thursday of the season, getting ready for gametime:
GeekWire: What changes have you made over the past few years to improve performance?
Campbell: “We’ve added a high-performance Java serving tier to our serving stack, hosted on Kubernetes. While the source of truth for Full Fantasy continues to be served by PHP web services, deployed across a fleet of Linux boxes, we added a dedicated Java-based system for serving current scoring information as fast as possible.
These Java services are 10 times faster than the equivalent PHP services and return to the user up-to-the-second information about the user’s fantasy team. To achieve this level of performance, we synchronize Fantasy team and player information (essentially your roster) to a shared private cloud hosted instance of Vespa. From that point, we use Apache Storm tasks to read from this store and apply the live real-life statistics being accrued by the user’s fantasy players. We store the calculated data into a sharded Redis instance, which is made available to the Java web services for near-instant serving to the user (versus before, we’d calculate this data inside of PHP). This same system is used to also send out personalized scoring notifications, with live scoring information in it (which is unique in the industry).”
GeekWire: What cloud computing service do you use? AWS? Azure?
Campbell: “We are currently hosted on a ‘private cloud’ within the Oath data centers. Nodes are provisioned with OpenStack Ironic and Chef is used to provision the Kubernetes cluster, which Sports is hosted on.”
How many people are working for your teams on a Sunday gameday?
Campbell: “After we get through the first few weeks of the season, [we shift to a] to an on-call model where engineers are paged if any issue occurs with the system. We have a dedicated ‘NOC’ monitoring our site, and we have customer care and our editorial staff working throughout the day answering customer questions and keeping the content of the site up-to-date. We have automated processes for receiving scores from our statistic vendor and updating databases and fantasy scoring information. Engineers and product managers are on Slack constantly checking in with each other to quickly answer questions and ensure the site is running smoothly.”
Highlights from the week in sports tech
- Valve may be moving The International, its massive “Dota 2” esports tournament, from Seattle to Canada.
- Both Sunday Night Football and Monday Night Football posted some of their lowest ratings in the past decade.
- Big college football games coming up in the next few days — ESPN is doing its “Megacast” for the playoff matchups, and introducing a new first down marker camera.
- Here’s a look at how blockchain and esports could be used together.
- Fans will likely have much more control of the way they watch a live sports broadcast in the future.
- The smartwatch industry will likely see its growth continue declining in 2018.
- Here’s how the Warriors’ new Chase Center could play a big role for the team’s “budding dynasty.”
- Speaking of, Golden State is using technology from SyncThink to diagnose concussions.
- The NBA and its broadcasters are using tech to find new ways to connect with fans.
- The Minnesota Vikings are the first NFL team to release a VR app for Oculus.
What to watch this weekend: Tons of great college football this weekend; USC vs. Ohio State at 5 p.m. PT tonight should be good, while UW takes on Penn State and Wisconsin plays Miami on Saturday; the playoffs start on New Years Day.
Thanks for tuning in, everyone! Happy New Year! — Taylor Soper
