Problems with Amazon Web Services’ Direct Connect service took down several customers Friday morning using the cloud leader’s creaky U.S. East region, including its Alexa personal assistant.
The issues began around 7:30am PT, and were unresolved as of this writing. “We are investigating increased packet loss possibly impacting some AWS Direct Connect customers in the US-EAST-1 Region,” AWS wrote on its status page around that time, and over the course of the morning isolated the issues to connectivity problems at data centers in Northern Virginia run by CoreSite and Equinix in Reston and Ashburn, respectively.
AWS Direct Connect is used by hybrid cloud customers to set up a secure connection between AWS infrastructure and the customer’s on-premises infrastructure. This is often faster and cheaper than connecting to AWS over the public internet.
The most visible sign of the outage? Amazon’s Alexa digital assistant was down for a prolonged period of time Friday morning, inadvertently recreating Amazon’s Super Bowl commercial. While that might be annoying, the outage is causing serious problems for users of Atlassian’s developer tools such as JIRA or Bitbucket, and Capital One customers vented their frustrations at being unable to access their accounts.
It’s not clear exactly how those two companies configured their services, but AWS generally encourages customers to set up their workloads across multiple availability zones in the event of problems such as these. That doesn’t explain why Alexa went down, however.
Update 11am: At 10:30am, AWS updated its status page to acknowledge separate problems earlier in the day with AWS Internet Connectivity in Northern Virginia due to “a network facility power loss.” The Washington D.C. area experienced extremely high wind gusts Friday morning, which might explain it.
The AWS Direct Connect issues remain unresolved, according to the status page. But Atlassian said it was beginning to recover from the earlier problems.
We are seeing a recovery with our cloud provider, AWS, and Atlassian products and services are recovering with it. We will continue to monitor and post updates on our Statuspage: https://t.co/pR95HIdc3m
— Atlassian (@Atlassian) March 2, 2018
Update 11:26am: AWS just said the Direct Connect issues have been resolved, but didn’t elaborate on whether or not they were also related to a power outage in the area.