Google’s Sunday Service Outage Caused by an "Incorrectly Applied" Server Configuration

Google app icons on a smartphone display

Google: Here's what caused Sunday's big outage

"In essence, the root cause of [the] disruption was a configuration change that was intended for a small number of servers in a single region", Google said.

Google has provided some details about what caused the massive outage on Sunday that affected major tech brands that use Google Cloud as well as Google's own services, including YouTube, Gmail, Google Search, G Suite, Google Drive, and Google Docs.

Like this story? Subscribe to FierceTelecom! Sign up today to get telecom news and updates delivered to your inbox and read on the go.

The Google Cloud status dashboard states that Google Cloud Networking was experiencing network congestion in eastern US, affecting Google Cloud, G Suite, and YouTube. Google said it had meant to make a configuration change to a small number of servers in one location, but that this change was accidentally applied to many more servers in more regions.

Google has opened up about why some of its services went down in the US on Sunday.

Sloss explained that the regions operating with constrained capacity became clogged up as they attempted to cram inbound and outbound traffic into what capacity remained. Unfortunately, the same network congestion that was plaguing Google services was also responsible for prolonging the fix from rolling out. "The Google teams were keenly aware that every minute which passed represented another minute of user impact, and brought on additional help to parallelize restoration efforts".

The search giant said YouTube, G Suite and Gmail were disrupted as a result of 'high levels of network congestion, ' but that it meant to examine the issue further and make improvements to its systems to prevent a similar event from happening again.

During this time, Sloss estimated that Google Cloud storage systems suffered a 30 per cent drop in traffic - perhaps due to the low-latency deprioritization, we reckon - while YouTube views dropped 2.5 per cent for an hour, and some search queries were slowed. "Approximately 1% of active Gmail users had problems with their account; while that is a small fraction of users, it still represents millions of users who couldn't receive or send email".

Google also insisted that the widespread outages, which primarily affected service in the Eastern United States, but also stretched across the globe, were not a result of cyberattacks.

Google says that it is taking steps to be "guarded against the entire class of issues illustrated by this event".

He apologized for the network outage in his blog post, and said that Google would learn from its mistakes going forward.

Latest News