Tuesday, January 29, 2013

VPC Migration: Post Mortem

All done! Every last one of the servers is running inside of Amazon's VPC. For the most part, everything went as expected. There are just a few loose ends I'd like to note.

Chef Server

Our server configuration is managed by Opscode's Chef. Our Chef setup is not redundant, it needs to move, and all of our other servers depend on it.

I planned to do a dry-run of the migration using an EBS snapshot. Unfortunately, before I could complete the dry-run, the old server died - just my luck. So I scrambled to get everything up and running.

Luckily, Simon Pither had already gone through moving a chef server to another machine. His post was fairly accurate. The only thing that was missing was the solr data that is used for things like `knife status`. To migrate that data, just add /var/cache/chef to the list of directories to copy.

With that blog post, and some quick typing, I was able to move the chef server before anything catastrophic happened.

Security Groups

EC2 security groups and VPC security groups are mutually exclusive. They need to be migrated as well. Because it's such a mind-numbing task to transfer them over one-by-one in the AWS console, and because the rules change during the transition, we had some downtime (<1 minute) due to incorrect security groups.

To avoid this issue, I would recommend writing a script that brings over the security groups programatically. The script may take a little longer, depending on the number of rules and groups, but it will be well worth it.


Instead of waiting for a future sprint to utilize the internal ELBs, I chose to use them right away. Already, I'm loving them. It's nice to not manage the scaling of my own haproxy servers.

If you intend to go through the same migration, I'd be happy to answer questions. Just leave a comment.


  1. Hi Matthew,
    Super helpful series of posts on VPC migration.
    My name is Pradeep - CTO of Vivaki.

    In addition to being a huge fan and customer of Lucidchart, we happen to be looking at a VPC migration as well. Wondering if you can spare a few mins to chat with us about your experience and answer a few questions.

    Thanks in advance. Much appreciated.

    1. This comment has been removed by the author.

    2. Thanks for the compliment. I'd be happy to answer your questions. I'm available on Google Chat with (removed email). Why don't you email me, and we can arrange something?

  2. Hi Mat;

    Quick question, if the chef-server is outside the VPC, how does one manage the provisioning, decommissioning etc of instances inside VPC's?

    Private addressing inside a VPC presents a challenge to bootstrapping does it not?

    1. No, just the opposite. Because your VPC instances can reach anything with a public IP address through the NAT, and your chef server is outside the VPC (presumably in EC2 with a public IP address), then VPC can speak to EC2.

      This situation, if you move 100% into VPC, is only temporary anyway. Once inside the VPC, your chef server will have all the same connectivity you once enjoyed in EC2.


Note: Only a member of this blog may post a comment.