You may remember that last Friday services that were powered by Amazon’s Web Services went completely down which included Netflix, Instagram and Pinterest and was according to Amazon was due to an electrical spike in the company’s Northern Virginia data center.
“We’d like to share more about the service disruption which occurred last Friday night, June 29th, in one of our Availability Zones in the US East-1 Region. The event was triggered during a large scale electrical storm which swept through the Northern Virginia area.”
Which was the start of the problem the main problem which caused the service to have further outages was that Amazon’s power backup systems glitched.
“All utility electrical switches in both datacenters initiated transfer to generator power. In one of the datacenters, the transfer completed without incident. In the other, the generators started successfully, but each generator independently failed to provide stable voltage as they were brought into service. As a result, the generators did not pick up the load and servers operated without interruption during this period on the Uninterruptable Power Supply (“UPS”) units. Shortly thereafter, utility power was restored and our datacenter personnel transferred the datacenter back to utility power. The utility power in the Region failed a second time at 7:57pm PDT”
Within their power systems Amazon uses two-tiered backup power processes which basically allow them to back up their system with a large electrical generator called UPS power, which powered the servers for as long as it could but actually only lasted 10 minutes, after which Amazon had to spend several hours to reboot the servers.