Re Google - there was an "electrical incident" injuring 3 people at a data center earlier

[highplainsdem]

today.

I don't know if that had anything to do with the error messages people are getting from Google tonight, but this is the SFGate story I found on Twitter:


https://www.sfgate.com/news/article/google-electrical-incident-injures-3-17360321.php

Three electricians were critically injured and transported to a local hospital after an "electrical incident" at a Google data center in Council Bluffs, Iowa, according to the Council Bluffs Police Department and Google.

The incident occurred at 11:59 a.m. local time on Monday, the Council Bluffs Police Department told SFGATE. Three electricians were working on a substation close to the data center buildings when an arc flash (an electric explosion) occurred, causing significant burns to all three electricians.

-snip-

Google has 14 data centers in the United States, and 23 in total around the world. The data centers "keep all of Google’s products and services up and running around the clock," according to the company. The Council Bluffs location site is one of Google's largest. It first opened in 2009.

[applegrove]

1. Vibes to them and their loved ones.

[BlueWavePsych]

2. Safety. Safety. Safety.

[lapfog_1]

3. I don't know the particulars of Google DCs

but I can tell you that almost all modern Data Centers have the following.

Dual rail electrical input from different grid power sources to each and every rack, and thus to all equipment.

Every server, switch, and router should have 2+1 redundant power supplies inside the equipment.

A backup battery / generator system that will take over if both grid tied supplies fail... although battery storage may only be sufficient to keep critical switches, routers, and firewalls online while the generator starts up (typically in minutes).

And electrical failure at a substation should not cause an outage at the Iowa DC... and even if that DC went offline because of failure to plan adequately, the rest of the data centers around the world should not be affected even if the load of requests, etc is redistributed to them.

That said, I remember seeing pictures of the very first google servers from the late nineties, early 2000s... and those were complete jokes. cheap, yes... but very fragile.