On Tuesday, 20 October, a Google Cloud data center near London lost power. The europe-west2 region saw a spike in errors and intermittent network failures. The incident began at 06:08 GMT.
Finance was hit hardest. Revolut customers could not withdraw cash, send transfers or top up cards, and Lloyds and HSBC reported problems with their apps and online banking. Downdetector logged hundreds of complaints about the property site Rightmove and Royal Mail. Google later said the impact had been resolved for most customers, though some users were still having trouble.
Why one data center is a lot
People often think of the cloud as something that does not go down. In practice a project usually lives in one region, sometimes in one availability zone, because that is cheaper and simpler. When that place loses power, neither the provider’s brand nor the SLA helps: compensation comes later, while customers leave now.
What you can do without a big budget
- Keep database and file backups in another region or with another provider
- For an important site, spread servers across at least two availability zones
- Put static files and a fallback page on a CDN so visitors see a message instead of an error
- Set up external uptime monitoring with alerts to Telegram
- Write down in advance who does what in an outage: whom to call, where to switch DNS
Our take
A small site does not need a multi-cloud setup, but everyone needs a plan for outage day. Usually a fresh backup somewhere else, monitoring and a clear “we’ll be right back” page are enough. Beyond that, it depends on what an hour of downtime costs you. Source: iXBT.com.