Everybody Makes Mistakes: The Case of Amazon Web Services

Think you’re the only one who feels “technically illiterate”? Think again. Even the professionals make mistakes! On Tuesday February 28th 2017, you might have noticed some of your favorite websites were acting strangely. No, it wasn’t your imagination and no, it wasn’t foreign hackers. 

It was a typo.

Amazon’s S3 web-based storage service helps hundreds of websites function smoothly. It’s a tool for web developers to store data on the cloud, making editing a lot easier and faster. The server hosts content for upwards of 100,000 websites so the outage was wide ranging in its effects. Ironically, the website most people use to check on these issue isitdownrightnow.com was also down.

These servers have been fairly dependable over the years. So when websites were malfunctioning, many people working within the tech industry demanded an explanation.

Amazon web services released a statement on the incident, which we’ve quoted below:

“…an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.”

When you input a command incorrectly on a piece of programming, it’s not like typing up a Word document– you can’t just press “delete” or “ctrl + z” to fix the problem. You need to refresh the whole server.

Unfortunately, Amazon’s S3 servers were so dependable that they actually had not been refreshed in years. As a result, the fix took Amazon a lot longer than they thought it would– and that’s when folks started to notice all was not as it should be. Just like a computer that has not been fully turned off in a while, refreshing the servers took a while–around five hours to be exact. In the tech world, where speed is the name of the game, five hours feels like a decade.

Luckily, unlike a DDoS attack, only certain parts of certain websites were affected. For instance, I could still create designs on Canva.com but I didn’t have access to my older designs. Although it was disruptive, Amazon’s mistake was not as bad as it could be. Twitter reacted in a largely jovial manner with many observers praising Amazon’s commitment to transparency with regards to this issue.