Home Home
It's Amazon EC2 to the rescue!

Gaming Amazon EC2 to get more web hosting for less

March 19, 2011 in Development, Featured

Like countless others, my web sites are running under Amazon EC2, the cloud-based web service. It has been, at least for anything above the free “micro” tier, rock solid. But the costs are certainly not the cheapest cloud hosting around. How do you get around that?

I should start by saying that this post is not for everyone. All of my sites are low-to-low/medium traffic, and, more importantly, aren’t user-driven. This technique would probably not work well for a Facebook-ish site. More on this later.

Amazon EC2 Pricing

A short introduction to EC2: like I mentioned above, it’s an on-demand cloud-based web service. That means you “rent” a virtual server for periods of time, with discounts for one- or three-year terms. The standard pricing tiers are pretty concrete:

Standard on-demand pricing for instances.

Standard on-demand pricing for instances.

Reserved instances are the equivalent to renting in bulk. Essentially, you pay a fee in order to receive a discount on the per-hour charge:

Reserved instance prices.

Reserved instance prices.

Finally, there’s the Spot Instance. And this is where things start to get interesting:

Spot pricing for Amazon instances. Note the difference in prices!

Spot pricing for Amazon instances. Note the difference in prices!

Spot Instances use up what Amazon calls “unused EC2 capacity.” There’s an equilibrium price, called the Spot Price, that automatically adjusts for movements in instance supply and demand. You set your own maximum price, and as long as the Spot Price is lower than your max price, your instance keeps running. However, if the Spot Price goes over your max price limit, it stops. So the recommended usage for spot instances is situations where it doesn’t matter if it’s interrupted. Most people would say that web hosting does not fall into this recommended usage category.

You can view a rolling spot price history chart by logging into your EC2 account:

Amazon's rolling spot history price chart.

Amazon's rolling spot history price chart.

The bottom line is that, excluding the spikes, prices have fluctuated +-3% for the Small instance, +-5% for the Medium instance, and a larger 14% for the Micro instance.

The other bottom line is that, if you set your Spot instance price limit far above the average – say $0.20 for the Small, which averages around $0.030 per hour, your instance can run indefinitely. Meaning you get the benefit of an on-demand instance for a significant discount. For the Small instance, we’re talking $0.085/hr vs. $0.030/hr – that’s 65% off the regular price, making your minimum monthly bill $21.60, an amazing price for a VPS that provides 1.7 GB of RAM. Of course, it’ll be slightly more depending on how much bandwidth, storage space, and I/Os that you consume.

The price, however, is offset by the risk of losing your instance, your data, and your websites.

Managing Risk

I don’t know about you, but my approach to saving money is thus: if the discount is bigger than the amount of risk involved, go for it. And so far, the discount is pretty damn big. But now we have to take into account the possibility of an unexpected surge in demand/shortage of supply, making prices shoot up significantly. For example, if prices jump to $0.30 for the Small, your instance will be up and gone like a fart in the wind.

So we need to find a good way to mitigate the risk. And that method is a good backup system.

Here’s how mine works.

At a very low traffic point of the day, generally around 3-4 am, my backup cron job will kick in and do the following:

- Gzip everything in /var/www
- Dump the entire MySQL database
- Upload the gz’ed public_html backup file to Amazon S3
- Upload the MySQL .sql dump to S3

Since it runs once per day, my worst case scenario is that I lose a day’s worth of data. That’s okay with me. Depending on your setup and needs, you may wish to run backups more often, or customize your backup strategy for specific databases, such as backing up one database every hour and a different one each day. It’s purely up to you.

The Actual Backup

You’ll need a shell script, a php upload script, and Donovan Schonknecht’s S3 PHP class. And a cron job!

I can’t take credit for much of these. The shell script and PHP file that I’ve used comes from this post at kovshenin.com. He’s got a better version of the shell script that also checks and auto-repairs corrupted tables (I’ve been too lazy to add it to this).

The Shell Script

Remember to change the USER and PASSWORD fields in the mysqldump command to your own. I would recommend creating a backup user specifically for this job, with SELECT, FILE, and LOCK TABLES privileges. You can also change the backup file location to something else (default is /ebs/backups/).

#/bin/sh
mysqlname=mysql.`date +%Y-%m-%d`.sql.gz
dataname=data.`date +%Y-%m-%d`.sql.gz
echo Generating MySQL Dump: ${mysqlname}
mysqldump -uUSER -pPASSWORD --all-databases | gzip -c9 > /ebs/backups/${mysqlname}
echo Archiving /var/www
tar -czf /tmp/${dataname} /var/www
echo Uploading ${mysqlname} to S3 bucket
php /ebs/data/upload-any.php ${mysqlname}
echo Uploading ${dataname} to S3 bucket
php /ebs/data/upload-any.php ${dataname}
echo Removing file ${mysqlname}
rm -f /ebs/backups/${mysqlname}
echo Removing file ${dataname}
rm -f /tmp/${dataname}
echo Done.

Don’t forget to give it proper permissions with chmod: “sudo chmod 700 s3.sh”

PHP Upload Script

require("S3.php");
require("S3auth.php");

set_time_limit(0);

$total_backup_days = 1;

if ($argv[1]) {

	// is this a mysql or data upload?
	if (strpos($argv[1], 'mysql') !== false) {
		$file = 'mysql';
		$local_dir = '/ebs/backups/';
	}
	else if (strpos($argv[1], 'data') !== false) {
		$file = 'data';
		$local_dir = '/tmp/';
	}
	else die('File does not match any known patterns.');

	$s3 = new S3($access_id, $secret_key);
	$s3->putBucket($bucket_name, S3::ACL_PRIVATE);
	$s3->putObjectFile($local_dir.$argv[1], $bucket_name, $remote_dir.$argv[1], S3::ACL_PRIVATE);

	$delete_date = date('Y-m-d', time()-(86400*$total_backup_days));

	echo 'Deleting remote file ' . $file . '.' . $delete_date . '.sql.gz' . "\n\n";
	$s3->deleteObject($bucket_name, $remote_dir.$file . '.' . $delete_date . '.sql.gz');
}

You may want to change the $total_backup_days variable to a larger number, depending on how many days of backups you want to keep. I opted for a grand total of…one.

Finally, the S3auth.php file contains some important details to connect to your Amazon S3 bucket:

$access_id = 'ZZZZZZZZZZZZZZZZZ';  
$secret_key = 'SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS/';  
$bucket_name = 'your.bucket';  
$local_dir = '/ebs/backups/';  
$remote_dir = 'backup/'; 

Fairly self-explanatory. The access_id and secret_key comes from your Amazon EC2 account, the bucket is whatever bucket you created in S3, and the remote_dir is the folder inside the bucket.

Cron Job

Use crontab -e or sudo crontab -e to access your cron job. It should probably be something similar to this:

00 3 * * * /ebs/s3.sh

This means the cron job will run s3.sh every day at 3 am.

Final thoughts, like why this won’t work for everyone.

To make your restore process easy, make sure you set up an AMI (Amazon Machine Image) once you’ve finish configuring your environment. That way, if your server crashes and burns, you can fire up another instance of your AMI, copy and extract the files, and import the .sql file to your database.

And if you have a user-driven site, remember that any changes that occur between any given point in time and your last backup time will be lost. Unless you’re backing up every 30 minutes or so (an extremely unrealistic scenario), you stand to lose more than you would probably gain using this method. In this case, you’re far better off paying for on-demand or reserved instance pricing.

***

Submit to reddit!  Digg!  Like this on Facebook!  Tweet this!

Tags

,

Soroush
1/21/2012
4:08 pm

Great stuff – used in conjunction with the original scripts at Konstantin’s blog, got things working in no time. Thanks a lot!

 


angbox: version 5

Subscribe!

Subscribe!


Subscribe to this blog: RSS, Atom
 
angbox

About angbox


Angbox is the personal blog of Cary Ang, a web developer from New Jersey, and an attempt to write about the random things that move him.
 
php.angbox: now open

php.angbox


An effort to teach practical PHP to programmers and non-programmers alike. Not sure where this is going...

The Blog

7

Development


40

Featured


9

Gaming


3

Music


5

Other


6

Personal Finance


19

Travel


Featured Posts

randomlygifted

New randomlygifted update brings more gift ideas and hopefully...


One of my many side projects is randomlygifted, a random gift idea generator. It's a site that constantly searches through Ama...
 
It's called the Last Express for a reason...

The Last Express: Poignant, entertaining, and just short of a ...


Nestled among some of the more recognizable games of 1997 - a list that included Fallout, Final Fantasy 7, and GoldenEye - was...
 
This is Carly Rae Jepsen's EP. LISTEN TO IT NOW.

Curiosity (2012) - Carly Rae Jepsen: Catchy at its Finest


So, there's this new girl in town. Her name's Carly Rae Jepsen. She's Canadian, and she just joined Justin Bieber's new label....
 
AT&T's data plans. FUUUUUUUUUUUUUUUU...

Getting Around AT&T's Tiered Smartphone Data Plans


You've probably heard that AT&T's $30 unlimited* data plans have been dead and gone since the middle of last year, replaced wi...
 

Other Fun Angbox Sites

Clear Note Widget

Clear Note Widget


Clear Note Widget is a sticky note and Post-it app for Android phones. Get it now!
 
TubeSpinner

tubespinner


The best of YouTube, shuffled.
 
randomlygifted

randomlygifted


Let randomlygifted find you the perfect (random) gift ideas.
 
The Nokia Phone Blog

The Nokia Phone Blog


My blog covering the latest developments in the wireless industry, Symbian/MeeGo, and Nokia phones.
 
MyDealSpy

MyDealSpy


The greatest deal aggregator that has ever existed. Okay, I'm a little biased here.
 
PhoneMatchup

PhoneMatchup


An eBay cell phone comparison shopping/price monitoring project. Buy your next cell phone through PhoneMatchup and save!