Skip to main content

Google offers apology and explanation for outage that took multiple services offline


Screen Shot 2014-01-24 at 2.06.15 PM

Earlier today, several of Google’s services went offline. Google has now offered an explanation for the downtime and apologized to affected users. The issue occurred when the internal system responsible for configuring Google’s sites accidentally created in incorrect configuration and brought several other sites.

The system soon self-corrected, which brought downed sites back online.

At 10:55 a.m. PST this morning, an internal system that generates configurations—essentially, information that tells other systems how to behave—encountered a software bug and generated an incorrect configuration. The incorrect configuration was sent to live services over the next 15 minutes, caused users’ requests for their data to be ignored, and those services, in turn, generated errors. Users began seeing these errors on affected services at 11:02 a.m., and at that time our internal monitoring alerted Google’s Site Reliability Team. Engineers were still debugging 12 minutes later when the same system, having automatically cleared the original error, generated a new correct configuration at 11:14 a.m. and began sending it; errors subsided rapidly starting at this time. By 11:30 a.m. the correct configuration was live everywhere and almost all users’ service was restored.

Google says it is putting additional checks and safeguards into place to prevent another instance of this issue.

FTC: We use income earning auto affiliate links. More.

You’re reading 9to5Google — experts who break news about Google and its surrounding ecosystem, day after day. Be sure to check out our homepage for all the latest news, and follow 9to5Google on Twitter, Facebook, and LinkedIn to stay in the loop. Don’t know where to start? Check out our exclusive stories, reviews, how-tos, and subscribe to our YouTube channel