Disk Failures and Service Interruptions

I am currently employed as the chief architect at Lucidchart. In my spare time (literally) I am also the ops guy. All of our servers are running on Amazon's EC2 cloud. Using the cloud is amazing and frustrating at the same time. Managing hardware, using tape drives, and co-location facilities are all nightmares; on the other hand, so are service outages, network failures, and ephemeral storage drives.

As the CTO of Amazon, Werner Vogels, says, "Everything fails all the time." I would like to give a report of one such failure: how it happened, what was affected, how we got through it, what I did because of it, and how I'll never have to deal with it again.