Sharing Amazon Elastic Block Store among multiple instances

Editor’s Note: This whole rigmarole is unnecessary now that you can boot from EBS-backed AMIs that can have essentially unbounded size.  But this trick was fun while it lasted!

I love Amazon’s Elastic Compute Cloud, and have been using it to run research experiments without having to worry about multiplexing computing resources among other members of my research group.  No running top after I login to make sure I’m not stepping on someone else’s experiments: I launch an instance and I get it all to myself.

Sharing storage across instances, however, is tricky.  For my purposes, having a read-only copy distributed among my instances is sufficient; of course adding read/write access makes things substantially trickier.  Yet even given that I was fine with read-only access, none of the solutions that immediately came to mind were satisfactory:

  1. Put everything in the root partition of my instance.  Every instance launched will have this same image, which is great.  The problem is that the root partition is just 10GB and doesn’t seem to be able to be increased.  Also, bundling up an instance is quite slow, making updates painful.
  2. You can get copious amounts of ephemeral storage via /mnt on each instance, but this doesn’t persist across the lifetime of an instance.  I could download a tarball from S3 and extract it to /mnt every time when I launch an instance, but this seems very hackish.
  3. You can get persistent network storage in the form of the Elastic Block Store, but this can only be attached to one instance at a time.  Doh!
  4. You can run NFS or GFS or whatever you want on your instances after you launch them, but this seems like a lot of work.  I’m supposed to be working on research, after all.

What I really wanted was the ability to mount an EBS volume read-only on multiple instances.  Since things are read-only, there won’t be any consistency issues but, still, Amazon doesn’t support this.  Until I discovered a hack to make it possible, using EBS snapshots.

The basic idea is to have a master EBS volume V that you want to replicate with read-only copies across a number of instances.  Upon bootup, each instance makes a snapshot of V and then its own personal volume Vp based on that snapshot.  Each instance can then attach the volume Vp and voila – we’ve got our data replicated across our instances.  No fancy network filesystem or S3 hacks necessary.

What makes all this go is that EBS snapshots are very fast (because they’re lazily constructed).  My master volume V is 10GB in size, and about 7GB full at the moment.  And this whole take-a-snapshot-and-mount-it routine takes less than 10 seconds.  After I’m done with an instance, I have it throw away the snapshot and volume Vp to save space.  But since snapshots are built on diffs, having a bunch of snapshots doesn’t take up much room in S3 (i.e. cost much money) anyway.  Ultimately, EBS is doing exactly what I would want to provide a high performance read-only version of the volume: lazy creation of snapshots makes replication fast, and each snapshot volume functions as a cache to increase read bandwidth.  And all this without any extra engineering on my part!

I put together some Python scripts (with the help of the excellent Boto library) to automate this read-only replication of an EBS volume.  All you have to do is edit some parameters in and then link these scripts into your distro’s boot/shutdown routines; this code is designed to be run from the instance itself.  The code is available under the MIT license (like Boto itself).  The repository includes a copy of Boto 1.7a to keep things self-contained.


5 thoughts on “Sharing Amazon Elastic Block Store among multiple instances

  1. This is great – helped me out just now since nfs kernel support seems to be broken in the version of ubuntu 10.10 I have running on my ec2 installs.

  2. Awesome, thanks Joe! Could you elaborate on how EBS-backed AMIs supersedes this? Your approach allows you to point to a Volume, so that you get a snapshot of the latest contents of that Volume at boot time, while the AWS approach seems to require you to point at a Snapshot, so if your Volume changes, you’d have to manually snapshot and update the AMI each time, correct?

    If I’m wrong please let me know, as I’d love to be doing this the “correct” way.

    • I mostly use the AWS web console, which has a 1-click menu option to create an AMI from a running instance. Just right click on the instance and select “Create Image (EBS AMI)”.

      I think you’re right that using the AWS API you have to create the snapshot first and then roll an AMI from the snapshot.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s