Last week I did a presentation on Amazon Web Services meet up in Greece. During the presentation I showed how you can start an AMI of fedora 8 base, install apache, php and mySQL and make them all store their data on an EBS volume. This attracted a lot of attention and participants asked me to post this as an online tutorial. This tutorial requires basic knowledge of AWS like running an EC2 instance, creating an EBS volume and assigning an elastic IP to the instance. During the presentation I used ElasticFox which is a firefox plugin that allows you to perform several operations on Amazon through a graphical user interface.
Now before starting the tutorial, I would like to write a few things about the purpose of it. The idea behind it, is to be able to run a server with data stored on EBS volumes. This gives you a lot of flexibility. For example, let’s say that you have two EBS volumes: One 10GB volume for you mySQL database and another 150GB volume for serving files through apache. If your website grows and you need to have a bigger storage just for your apache server you can simply take a snapshot of that volume, create a new volume using that snapshot but with a bigger storage (up to 1000 GB) and attach it back to the server. This is a matter of just a few minutes. Moreover this can happen for CPU problems as well. Once you need more CPU, you can create a new, bigger instance and attach your EBS volumes to that. Amazon offers great options for that, which practically can cover most of internet websites. Should you need more, scaling is much easier since they are many tools out there like RightScale or scalr which can add / remove instances on the fly using very advanced triggers for that.
Now let’s get our hands dirty:)
To get started you need to do the following things:
1. Launch an instance of the following AMI: “ec2-public-images/fedora-8-i386-base-v1.07.manifest.xml“.
2. Create an EBS volume and attach it to the instance (attach it to /dev/sdh).
3. Request a new elastic IP and assign it to the instance.
4. Connect to your new server using SSH.
All the above steps can be done using ElasticFox (or any other tool, even through command line).
Once you have logged in as root to your server the first thing to do (only do this once) is to format your newly attached hard drive (EBS volume actually:)). To do this, run the following command:
mkfs.ext3 /dev/sdh
Then you need to create a directory on your server where the new volume will be mounted.
mkdir /ebs1
To mount the volume to that directory run the following command:
mount -t ext3 /dev/sdh /ebs1
Note for the steps above: If you ever terminate your instance and create a new one, you will only need to create the directory and mount the volume there. The volume would be formatted already and your data will be in there!
Now you can install apache, php and mySQL using yum:
yum -y install httpd php mysql mysql-server php-mysql
To make sure that these services are started each time your instance boots, run the following commands:
/sbin/chkconfig httpd on
/sbin/chkconfig –add mysqld
/sbin/chkconfig mysqld on
You can start apache and mySQL using:
/sbin/service httpd start
/sbin/service mysqld start
Once you have started mySQL it is advised that you do some stuff for security reasons. The first thing to do is to change the root password:
mysqladmin -u root password ‘new-password’ (where new-password is the password you want for root).
Then connect to mySQL:
mysql -u root -pnew-password (replace new-password with the password you specified before).
Remove the test database:
DROP DATABASE test;
Disable anonymous access:
DELETE FROM mysql.user WHERE user = ”;
FLUSH PRIVILEGES;
Then exit mySQL using quit.
Now it’s time to move apache to EBS:
First we need to stop the service:
/sbin/service httpd stop
Then we move httpd to EBS:
mv /etc/httpd /ebs1/httpd
And create a symbolic link to it:
ln -s /ebs1/httpd /etc
Also do this for the document root:
mv /var/www /ebs1/www
ln -s /ebs1/www /var/
And restart apache:
/sbin/service httpd start
Note: If you terminate and start a new instance simply remove the /etc/httpd and /var/www folders and just run the commands to create the symbolic links.
Then we do the same for mySQL:
/sbin/service mysqld stop
mv /var/lib/mysql /ebs1/mysql
ln -s /ebs1/mysql /var/lib
/sbin/service mysqld start
This only moves the mySQL data, not the logs. You can edit /etc/my.cnf to configure where logs are stored and also move the log directory to EBS if you want to.
So now you have a fully functioning server on EC2 that stores and retrieves data to and from and EBS volume. The EBS volume and be backed up almost instantly using a snapshot. If you don’t use innodb for your tables it is advised that you lock your tables while you take the snapshot.
Next steps / thoughts:
Some might want to create a hosting enviroment for many websites. What you can do for this is to edit your httpd.conf file (located on EBS /ebs1/httpd/conf/httpd.conf) and enable Virtual Hosts. What I did is to create a directory /ebs1/sites and then create users on fedora using the following command:
useradd -d /ebs1/sites/mysite1.com user-for-mysite1
Also don’t forget to specify a password for the user using:
passwd user-for-mysite1
Once you have created the user you can point the document root for that specific site to his home directory.
Also to enable the user to upload files to his/her website using SCP you need to allow password access through SSH (be careful on security issues though). To do this, edit /etc/ssh/sshd_config and change the line “PasswordAuthentication no” to “PasswordAuthentication yes”. Also restart sshd by running /etc/init.d/sshd restart.
The final step is to link real domains to your new server. We assume that you have already assigned one or more IPs to your server. To point a domain to a specific IP you need to edit its DNS records and add an “A record” that points to that IP. Once the domain resolves on your server you will be able to use it for your website. If you need to host multiple websites, create one VirtualHost record for each domain (using the appropriate servername and documentroot values) and off you go!
Special thanks to http://agiletesting.blogspot.com/ for the useful information that I found there for this tutorial.
Yes, the presentation…was great 🙂
but thanks for posting the commands here. Very useful and easy to have them in hand, instead of going through all the manual.
John,
I am looking for a solution where data will reside in an EBS but launching an instance will be just that and the EBS attaching. No moves, or slinks. Any ideas?
If you’re interested in more detail on EC2 and EBS, I posted a little writeup on my site about installing the Drupal CMS on EC2: http://www.sunsetlakesoftware.com/2008/09/13/running-drupal-website-amazon-ec2 .
I take the same tack as you did with the step-by-step instructions. Additionally, I include some tips on bundling your own AMI and setting up automatic hourly snapshot backups to S3.
@Nikos: You can do this by customizing your instance and having it mount the EBS at boot time. Also, rightscale provides a way to run scripts for every instance you run so it is configured automatically.
@Brad: Very good tutorial! Especially the instructions for bundling the AMI are very handy since I didn’t include them at all in my post.
I learned the hard way that it’s a little bit of a bad idea to mount the EBS volume under /ebs1 (or as anything else under /).
When/if you later bundle the instance to an AMI, the /ebs1 folder will be included in the image, which means even if you attach the EBS volume to that new instance, you’ll first have to clean out the /ebs1 folder.
Hence it’s better to mount it to /mnt/ebs1, as /mnt is one of the folders excluded by the AMI tools.
I setup an AMI which points Apache/PHP config files, MySQL data, and web app files to /mnt/ebs1/. All I need to do to get another instance running ready for another web app to be deployed is simply to start the instance from my AMI, create and attach an EBS volume based on a prepared snapshot, and then run the following commands:
$ mkdir /mnt/ebs1
$ mount /dev/sdh /mnt/ebs1
$ /etc/init.d/apache2 start
$ /etc/init.d/mysql start
The AMI I was working with runs Ubuntu 8.04, hence the different commands to launch Apache and MySQL from what’s in the article above.
Thanks for the useful tip jimeh!
Thanks for this post, saved me a lot of Googling when I hooked up my first Instance 🙂
Thanks! Following your tutorial I was able to start a LAMP/EC2/EBS server within 10 minutes, joy!
Thanks! Very useful information, crisp and to the point … Jimeh, thanks for the advice, will keep in mind
Cheers John – great clear article and helped with my first venture into using EBS.
hey, is it possible to assign more than one IP in one EC2 instance?
I want to host all my websites in one EC2 instance, is this possible?
Thanks for your help.
This is a Great tutorial. Is it possible to implement a firewall by default in EC2? On the first hand, I tried to install cPanel , But i had some issues with DNS Zone files, After a lot of workaround, I managed to run cpanel successfully. Still iam have some trouble with the subdomains created through cPanel. Anyhow this is a great tutorial and saved me a lot of time.
Hi There,
I hope you don’t mind what might seem to be some very naive newby questions here, but here goes…
I just set up a Basic 32-bit Amazon Linux instance from an EBS backed image with a joomla website attached to a database that is still hosted by my old web host. It’s working, slowly (not sure if that’s the way I set it up or the fact that the instance is in EU-West-1 and the database is in the Western USA) but it works.
I have been looking to set up the database now and on looking for some advice come up with these tutorials about setting up EBS volumes for certain functions.
Ok, does not the fact that my instance is coming from an EBS volume mean that the data gets backed up to that volume – if the instance shuts down wouldn’t the data persist?
If that’s the case why not just set-up the database using the instance that is running?
Cheers,
Dominic
Great tutorial, Jnousis. My hiccup was that elasticFox at this moment, Sept 4th claims not to run with firefox 6.x, so the entire section that was glossed over concerning attaching to the instances etc, didn’t work for me.
Guess, i will have to download the aws command line package and run it all from terminal using those long aws commands that no newbie can remember, lol.
Just wanted to say thank you for this tutorial. I didn’t really understand volumes at first, but this really cleared things up for me. Thanks to you, I survived a major server failure because I had snapshots of both my volumes, and was able to quickly and easily restore my complete server in less than 10 minutes with very little issue. Thanks! I have this page bookmarked, and I come back here very often.
Thanks for this awesome post! I succeeded in every step you described BUT when I’m trying to access my DB on EBS I got this message : Can’t connect to local MySQL server through socket ‘/var/run/mysqld/mysqld.sock’
What would be the problem? I checked that mysql is running and it is…
I’m using the AMI Ubuntu 11.10 Oneiric
Many thanks in advance!
Is using EBS really a better option than managed hosting platform, like Cloudways, for deploy PHP apps on EC2? If someone who is not good with sysadmin thing, what would be a better option for them. Go to EC2 directly or use a managed hosting?