I recently came across a post about James Hamilton, the star AWS employee who keeps the internet alive while living on a boat, and I thought I would write a blurb about his sage wisdom bestowed upon me during the best session I attended at the AWS re:Invent conference.
If you can either read the rest of this blurb, or watch the session, I recommend you do the latter. In particular, watch James's part; the whole thing is valuable, but James really delivers some good points, backing them up with personal experience.
The main takeaway, for me, was James's 5 points to put in a software system before the system's release. These 5 points are
5. add monitoring and alerting,
4. have incremental deployment,
3. automate testing,
2. test well, and
1. horizontally partition your database.
Each one of these points rings true to me. I personally have had systems succeed or fail because of the implementation or lack thereof due to these points.
I highly recommend that you watch the session.