It turns out Google does have extra capacity which is how they were able to bring back service after 100 minutes.

To Google's credit they did notice the issue immediately and they are taking steps to have better reliability in the future.

"After establishing that the core problem was insufficient available capacity, the team brought a LOT of additional request routers online (flexible capacity is one of the advantages of Google's architecture), distributed the traffic across the request routers, and the Gmail web interface came back online," Treynor blogged.

Treynor added that Google is now increasing request router capacity beyond peak demand. They are also doing some router isolation to ensure they don't have the same kind of cascading effect in the future. So if one router goes down in one datacenter it doesn't affect routers in other data centers.

Graceful degradation, another good best practice, is something Google is now going to implement. So traffic will just be slower when trouble pops up, instead of having routers just not accept traffic in response to an overflow condition. This is the same type of technique that works well in my opinion to deal with Denial of Service (DoS) attacks.

"Gmail remains more than 99.9% available to all users, and we're committed to keeping events like today's notable for their rarity,' Treynor wrote.

This incident just goes to show us all that even a big sophisticated organization like Google can mess up on network upgrades.