Set Up Your Server Right
A Little Background
Unix administrators have been dealing with the same problems for a long, long time. Users and programs contending for resources, outside attackers trying to get in, crashing services. You name it. There are a lot of excellent utilities out there to handle all of these issues. Unfortunately, they go mostly unused.
Ever have an application, of any language, eat up memory or disk space to the point of slowing the entire system to a crawl? Seen a security breach in one web application compromise a whole server? Had a server encounter performance problems that seem to disappear by the time you make it into top? I’ve joined a number of projects where problems like these are constant, nagging issues. They really don’t need to be.
Assume Your Application is
not “Well-Behaved”
When you are setting up your server, the general rule is: don’t trust your own rails app. Assume it will leak memory, take up hard disk space, try to run commands it shouldn’t, attempt to delete other application’s databases, and so on. Besides the potential for an arbitrary code-execution bug being exploited by a hacker, you also have to keep in mind the standard bugs and your own “oops!” moments that are part of the development process. Thinking like a sysadmin can give your production boxes a nice safety net.
I am going to go through a series of blog posts explaining a lot of tips, tricks, and best practices for setting up a single server and the general small infrastructures. Today we are going to hit the basics — what I consider to be the bare minimum necessary to get an Ubuntu 8.04 LTS server ready to go and serving your rails application.
The Basics
So lets start out with the basics everyone expects.
Step 1: Create Your User & Setup sudo
I prefer to set up users with password access disabled. SSH keys are the way to go. If you are setting up a VPS, your root account pretty much needs to have a password, however. It should be long, random, and stored someplace safe.
root@your_host:~# adduser thomas --disabled-password root@your_host:~# usermod -aG sudo thomas
Step 2: Give Yourself Sudo Access
We set up your user sans password before, so you can’t use a password for sudo.
root@your_host:~# visudo # Add the following line %sudo ALL=NOPASSWD: ALL
Step 3: Update Your System
Before you forget or spend any time setting up or compiling against old packages.
root@your_host:~# apt-get update root@your_host:~# apt-get upgrade
Step 4: Apt-get the World
Pretty much all the basics you will need for ruby/rails/passenger.
root@your_host:~# apt-get install ruby mysql-server libmysqlclient15-dev apache2 rubygems build-essential wget irb1.8 rdoc1.8 libopenssl-ruby1.8 libreadline-ruby1.8 ruby1.8-dev apache2-prefork-dev libapr1-dev libaprutil1-dev git-core
Use a strong password for mysql’s root!
Step 5: Install rubygems
Ubuntu 8.04 has a pretty out of date version of rubygems. Best to just start from the latest tarball.
root@your_host:~# wget http://rubyforge.org/frs/download.php/60718/rubygems-1.3.5.tgz root@your_host:~# tar -xvzf rubygems-1.3.5.tgz root@your_host:~# cd rubygems-1.3.5 root@your_host:~/rubygems-1.3.5# ruby ./setup.rb root@your_host:~# ln -s /usr/bin/gem1.8 /usr/bin/gem root@your_host:~# gem -v 1.3.5
Step 6: Gem Install the World
root@your_host:~# gem install --no-rdoc --no-ri rails passenger mysql rake
Step 7: Create a User for Your Application
Every application should have its own user, for a few reasons. If you are deploying several applications on the same server, you do not want one application to be able to write, or necessarily even read, the files of another. Later on, when we set up PAM resource limits, having separate users will allow us to prevent any single application from dominating the server’s resources. It also provides a handy mechanism for separating cron jobs, and protecting yourself from an “oops!” catastrophe.
This user will both run the application and deploy it, but will not have sudo access — whatever user you run the application as should never, ever, have sudo access. For any reason. Whatever you deploy the application as should ideally also not have sudo access.
What we are going to do here is create an application user, generate ssh keys for it, and point its authorized_keys file to an /etc/deployers file. Given the assumption that you are going to set up multiple rails applications deployed by the same developers on this server, that is just a nice way to share ssh public key definitions among all of the deploy users you will have.
root@your_host:~# adduser someapp --disabled-password root@your_host:~# su - someapp someapp@your_host:~$ ssh-keygen someapp@your_host:~$ exit root@your_host:~# touch /etc/deployers root@your_host:~# ln -s /etc/deployers /home/someapp/.ssh/authorized_keys
Of course, if you are allowing several unrelated developers and/or companies access to the same server you wouldn’t want to share an authorized_keys file like this.
Step 8: Set up Your Application Directory Structure
root@your_host:~# mkdir -p /var/rails/someapp root@your_host:~# chown someapp:someapp /var/rails/someapp/
Step 9: Set up Your Database
Set up a user for every application that only has access to that application’s database. Doing so is not only a good security measure, but it also helps ensure you don’t fail in a spectacularly epic manner.
If you share database users, you are inevitably going to copy your database.yml and/or password file from another app when you set up a new one. You are also inevitably going to forget to modify the database name in that before calling something destructive like rake db:reset. Lets avoid that.
create database someapp_production; grant all on someapp_production.* to 'someapp'@'localhost' identified by 'a_good_password';
The user should not have permission to create databases on the system, which may be a little inconvenient during setup. It is, however, a good security measure and gives a little more protection from sysadmin “oops!” events.
Step 10: Do a Deploy
Run your cap scripts, do your cold deploys, etc. Remember the someapp user isn’t allowed to have sudo access — so just install gems yourself.
Capistrano is currently the best way to handle deployments for rails applications. There are a lot of great guides to using it, so I won’t rehash that part.
If you are using capistrano you should now have a directory structure resembling this:
thomas@your_host:/var/rails/someapp$ ls -l * lrwxrwxrwx 1 someapp someapp 43 Aug 5 03:36 current - /var/rails/someapp/releases/20090805033622 releases: total 20 drwxrwxr-x 14 someapp someapp 4096 Aug 5 03:18 20090805031830 drwxrwxr-x 14 someapp someapp 4096 Aug 5 03:20 20090805032022 drwxrwxr-x 14 someapp someapp 4096 Aug 5 03:33 20090805033331 drwxrwxr-x 14 someapp someapp 4096 Aug 5 03:36 20090805033549 drwxrwxr-x 14 someapp someapp 4096 Aug 5 03:36 20090805033622 shared: total 20 drwxr-xr-x 13 someapp someapp 4096 Aug 5 03:06 cached-copy drwxr-xr-x 2 someapp someapp 4096 Aug 5 03:18 config drwxrwxr-x 2 someapp someapp 4096 Aug 5 03:06 log drwxrwxr-x 2 someapp someapp 4096 Aug 5 02:42 pids drwxrwxr-x 2 someapp someapp 4096 Aug 5 02:42 system
Step 11: Set up Passenger
Follow the excellent instructions & point to the proper path in your apache config.
root@your_host:~# passenger-install-apache2-module
In ubuntu you would set up the http configuration by creating a new file in /etc/apache2/sites-available with the recommended passenger configuration and soft linking it to a file in /etc/apache2/sites-enabled when the app is ready to go up.
root@your_host:~# vim /etc/apache2/sites-available/someapp root@your_host:~# ln -s /etc/apache2/sites-available/someapp /etc/apache2/sites-enabled/ root@your_host:~# /etc/init.d/apache2 restart
Beyond the Basic Setup
So generally it seems like everyone stops at that passenger step. The application is running, everyone is happy, right? Well, there is actually a bunch more to be done to save yourself some late nights later on.
Step 12: Shorewall Firewall
I like to run a firewall on pretty much every server I manage. They are easy to set up, and protect you from, well, you. A firewall doesn’t make it OK to have your database listening on an external IP address, but it can certainly mitigate the damage if that does somehow happen.
Getting shorewall on your server is pretty easy. First we’ll apt-get it, then we’ll copy the example config, tweak it, and start the firewall.
Rule number one with firewalls, routing, and network configuration. Make sure you can get into the system without ssh. If it is a physical server, this isn’t something to do offsite at 2AM. If it is a VPS, make sure you have the necessary credentials to log in via the web console.
The example file used here is for a server with one public facing interface/ip address. If your setup is different, you’ll need to read up on the shorewall docs. There are example files for a two-interface (private/public) server that make the process almost as easy. They are in /usr/share/doc/shorewall-common/examples/two-interfaces.
root@your_host:~# apt-get install shorewall root@your_host:~# cd /etc/shorewall/ root@your_host:/etc/shorewall# cp /usr/share/doc/shorewall-common/examples/one-interface/* ./ root@your_host:/etc/shorewall# vim rules # Add the following two lines SSH/ACCEPT net $FW HTTP/ACCEPT net $FW root@your_host:/etc/shorewall# vim shorewall.conf # Change STARTUP_ENABLED=No to STARTUP_ENABLED=Yes root@your_host:/etc/shorewall# vim /etc/default/shorewall # Change startup=0 to startup=1 root@your_host:/etc/shorewall# /etc/init.d/shorewall start
Done, you have a firewall.
Shorewall has a ton of features. You can read more about it at shorewall.net.
Step 13: PAM Limits
When your Linux box starts to run low on memory it fires up a psychopathic little helper call oom-killer. At first blush this may sound reasonable, fire up something to kill off whatever is taking up the resources. Except that’s not what oom-killer does. It just guesses — and usually ends up killing a lot of things you don’t want it to. Your leaking application might cause oom-killer to kill your apache or mysql process, or, more likely, other unrelated rails processes. Don’t let it get to that point.
Instead, lets set up a simple limit statement in /etc/security/limits.conf.
I can’t provide a recommended number to use here. Every application is different. The number here limits the total amount of RAM the user may use, not each rails process. So, if you have a maximum of three passenger processes that tend to max out at 30MB a piece and a cron job that tends to max out at 40MB during its runs, then a reasonable value here would be 200MB (we don’t want it fussing over small spikes). To set the limit as 200MB, you convert it to KB (204800) and enter it as shown below.
root@your_host:~# vim /etc/security/limits.conf someapp hard as 204800
Now, instead of having the OS guess at which process to kill, the OS will start denying someapp’s memory allocation calls — usually causing the offending application to crash. A far better scenario than setting oom-killer loose. I’ve actually seen troublesome apps hobble along in a configuration like this while the team works on a major refactor & code fixup. So long as the memory leak is infrequent, a configured limits file can make a leaky application seem a lot more stable if, for example, passenger is restarting the rails processes for you.
Step 14: Logging with Syslog
Dropping all of your log files into your RAILS_ROOT/log/production.log may be convenient, but it isn’t necessarily the best way to go. Syslog is a better choice for a production environment. It gives you more control, more configuration options, and will make multi-server setups far easier to debug.
Plus, if you want to get strict about security, an application shouldn’t be able to delete information from its own log file. Let’s just get started with the bare basics. First, install the gem and set up your syslog config.
root@your_host:~# gem install SyslogLogger root@your_host:~# vim /etc/syslog.conf # Add the following two lines to the bottom of the file !rails *.* /var/log/production.log root@your_host:~# /etc/init.d/sysklogd restart
Then, add these two lines to your environments/production.rb file.
require 'syslog_logger'
config.logger = SyslogLogger.new('someapp_production')
And restart your application. You should now see log statements going to /var/log/production.log
That’s a good start
While there is still a bunch more you can do to lock down and monitor your server, this is a great start. It’ll give you a solid, well organized base on which to build, which we will continue doing in the next installment.
- Thomas Meeks
