Envy Labs

A little background

Unix administrators have been dealing with the same problems for a long, long time. Users and programs contending for resources, outside attackers trying to get in, crashing services. You name it. There are a lot of excellent utilities out there to handle all of these issues. Unfortunately, they go mostly unused.

Ever have an application, of any language, eat up memory or disk space to the point of slowing the entire system to a crawl? Seen a security breach in one web application compromise a whole server? Had a server encounter performance problems that seem to disappear by the time you make it into top? I’ve joined a number of projects where problems like these are constant, nagging issues. They really don’t need to be.

Assume Your Application is not “Well-Behaved”

When you are setting up your server, the general rule is: don’t trust your own rails app. Assume it will leak memory, take up hard disk space, try to run commands it shouldn’t, attempt to delete other application’s databases, and so on. Besides the potential for an arbitrary code-execution bug being exploited by a hacker, you also have to keep in mind the standard bugs and your own “oops!” moments that are part of the development process. Thinking like a sysadmin can give your production boxes a nice safety net.

I am going to go through a series of blog posts explaining a lot of tips, tricks, and best practices for setting up a single server and the general small infrastructures. Today we are going to hit the basics — what I consider to be the bare minimum necessary to get an Ubuntu 8.04 LTS server ready to go and serving your rails application.

The basics

So lets start out with the basics everyone expects.

Step 1: If you haven’t already, create your user & setup sudo

I prefer to set up users with password access disabled. SSH keys are the way to go. If you are setting up a VPS, your root account pretty much needs to have a password, however. It should be long, random, and stored someplace safe.

root@your_host:~# adduser thomas --disabled-password
root@your_host:~# usermod -aG sudo thomas

Step 2: Give yourself sudo access

We set up your user sans password before, so you can’t use a password for sudo.

root@your_host:~# visudo
# Add the following line
%sudo ALL=NOPASSWD: ALL

Step 3: Update your system

Before you forget or spend any time setting up or compiling against old packages.

root@your_host:~# apt-get update
root@your_host:~# apt-get upgrade

Step 4: Apt-get the world

Pretty much all the basics you will need for ruby/rails/passenger.

root@your_host:~# apt-get install ruby mysql-server libmysqlclient15-dev apache2 rubygems build-essential wget irb1.8 rdoc1.8 libopenssl-ruby1.8 libreadline-ruby1.8 ruby1.8-dev apache2-prefork-dev libapr1-dev libaprutil1-dev git-core

Use a strong password for mysql’s root!

Step 5: Install rubygems

Ubuntu 8.04 has a pretty out of date version of rubygems. Best to just start from the latest tarball.

root@your_host:~# wget http://rubyforge.org/frs/download.php/60718/rubygems-1.3.5.tgz
root@your_host:~# tar -xvzf rubygems-1.3.5.tgz
root@your_host:~# cd rubygems-1.3.5
root@your_host:~/rubygems-1.3.5# ruby ./setup.rb
root@your_host:~# ln -s /usr/bin/gem1.8 /usr/bin/gem
root@your_host:~# gem -v
1.3.5

Step 6: Gem install the world

root@your_host:~# gem install --no-rdoc --no-ri rails passenger mysql rake

Step 7: Create a user for your application

Every application should have its own user, for a few reasons. If you are deploying several applications on the same server, you do not want one application to be able to write, or necessarily even read, the files of another. Later on, when we set up PAM resource limits, having separate users will allow us to prevent any single application from dominating the server’s resources. It also provides a handy mechanism for separating cron jobs, and protecting yourself from an “oops!” catastrophe.

This user will both run the application and deploy it, but will not have sudo access — whatever user you run the application as should never, ever, have sudo access. For any reason. Whatever you deploy the application as should ideally also not have sudo access.

What we are going to do here is create an application user, generate ssh keys for it, and point its authorized_keys file to an /etc/deployers file. Given the assumption that you are going to set up multiple rails applications deployed by the same developers on this server, that is just a nice way to share ssh public key definitions among all of the deploy users you will have.

root@your_host:~# adduser someapp --disabled-password
root@your_host:~# su - someapp
someapp@your_host:~$ ssh-keygen
someapp@your_host:~$ exit
root@your_host:~# touch /etc/deployers
root@your_host:~# ln -s /etc/deployers /home/someapp/.ssh/authorized_keys

Of course, if you are allowing several unrelated developers and/or companies access to the same server you wouldn’t want to share an authorized_keys file like this.

Step 8: Set up your application directory structure

root@your_host:~# mkdir -p /var/rails/someapp
root@your_host:~# chown someapp:someapp /var/rails/someapp/

Step 9: Set up your database

Set up a user for every application that only has access to that application’s database. Doing so is not only a good security measure, but it also helps ensure you don’t fail in a spectacularly epic manner.

If you share database users, you are inevitably going to copy your database.yml and/or password file from another app when you set up a new one. You are also inevitably going to forget to modify the database name in that before calling something destructive like rake db:reset. Lets avoid that.

create database someapp_production;
grant all on someapp_production.* to 'someapp'@'localhost' identified by 'a_good_password';

The user should not have permission to create databases on the system, which may be a little inconvenient during setup. It is, however, a good security measure and gives a little more protection from sysadmin “oops!” events.

Step 10: Do a deploy

Run your cap scripts, do your cold deploys, etc. Remember the someapp user isn’t allowed to have sudo access — so just install gems yourself.

Capistrano is currently the best way to handle deployments for rails applications. There are a lot of great guides to using it, so I won’t rehash that part.
Capistrano’s Home Page
A Getting Started Guide
Using git with Capistrano

If you are using capistrano you should now have a directory structure resembling this:

thomas@your_host:/var/rails/someapp$ ls -l *
lrwxrwxrwx 1 someapp someapp   43 Aug  5 03:36 current - /var/rails/someapp/releases/20090805033622

releases:
total 20
drwxrwxr-x 14 someapp someapp 4096 Aug  5 03:18 20090805031830
drwxrwxr-x 14 someapp someapp 4096 Aug  5 03:20 20090805032022
drwxrwxr-x 14 someapp someapp 4096 Aug  5 03:33 20090805033331
drwxrwxr-x 14 someapp someapp 4096 Aug  5 03:36 20090805033549
drwxrwxr-x 14 someapp someapp 4096 Aug  5 03:36 20090805033622

shared:
total 20
drwxr-xr-x 13 someapp someapp 4096 Aug  5 03:06 cached-copy
drwxr-xr-x  2 someapp someapp 4096 Aug  5 03:18 config
drwxrwxr-x  2 someapp someapp 4096 Aug  5 03:06 log
drwxrwxr-x  2 someapp someapp 4096 Aug  5 02:42 pids
drwxrwxr-x  2 someapp someapp 4096 Aug  5 02:42 system

Step 11: Set up Passenger

Follow the excellent instructions & point to the proper path in your apache config.

root@your_host:~# passenger-install-apache2-module

In ubuntu you would set up the http configuration by creating a new file in /etc/apache2/sites-available with the recommended passenger configuration and soft linking it to a file in /etc/apache2/sites-enabled when the app is ready to go up.

root@your_host:~# vim /etc/apache2/sites-available/someapp
root@your_host:~# ln -s /etc/apache2/sites-available/someapp /etc/apache2/sites-enabled/
root@your_host:~# /etc/init.d/apache2 restart

Beyond the Basic Setup

So generally it seems like everyone stops at that passenger step. The application is running, everyone is happy, right? Well, there is actually a bunch more to be done to save yourself some late nights later on.

Step 12: Shorewall Firewall

I like to run a firewall on pretty much every server I manage. They are easy to set up, and protect you from, well, you. A firewall doesn’t make it OK to have your database listening on an external IP address, but it can certainly mitigate the damage if that does somehow happen.

Getting shorewall on your server is pretty easy. First we’ll apt-get it, then we’ll copy the example config, tweak it, and start the firewall.

Rule number one with firewalls, routing, and network configuration. Make sure you can get into the system without ssh. If it is a physical server, this isn’t something to do offsite at 2AM. If it is a VPS, make sure you have the necessary credentials to log in via the web console.

The example file used here is for a server with one public facing interface/ip address. If your setup is different, you’ll need to read up on the shorewall docs. There are example files for a two-interface (private/public) server that make the process almost as easy. They are in /usr/share/doc/shorewall-common/examples/two-interfaces.

root@your_host:~# apt-get install shorewall
root@your_host:~# cd /etc/shorewall/
root@your_host:/etc/shorewall# cp /usr/share/doc/shorewall-common/examples/one-interface/* ./
root@your_host:/etc/shorewall# vim rules
# Add the following two lines
SSH/ACCEPT      net             $FW
HTTP/ACCEPT     net             $FW
root@your_host:/etc/shorewall# vim shorewall.conf
# Change STARTUP_ENABLED=No to
STARTUP_ENABLED=Yes
root@your_host:/etc/shorewall# vim /etc/default/shorewall
# Change startup=0 to
startup=1
root@your_host:/etc/shorewall# /etc/init.d/shorewall start

Done, you have a firewall.

Shorewall has a ton of features. You can read more about it at shorewall.net.

Step 13: PAM Limits

When your Linux box starts to run low on memory it fires up a psychopathic little helper call oom-killer. At first blush this may sound reasonable, fire up something to kill off whatever is taking up the resources. Except that’s not what oom-killer does. It just guesses — and usually ends up killing a lot of things you don’t want it to. Your leaking application might cause oom-killer to kill your apache or mysql process, or, more likely, other unrelated rails processes. Don’t let it get to that point.

Instead, lets set up a simple limit statement in /etc/security/limits.conf.

I can’t provide a recommended number to use here. Every application is different. The number here limits the total amount of RAM the user may use, not each rails process. So, if you have a maximum of three passenger processes that tend to max out at 30MB a piece and a cron job that tends to max out at 40MB during its runs, then a reasonable value here would be 200MB (we don’t want it fussing over small spikes). To set the limit as 200MB, you convert it to KB (204800) and enter it as shown below.

root@your_host:~# vim /etc/security/limits.conf
someapp     hard    as      204800

Now, instead of having the OS guess at which process to kill, the OS will start denying someapp’s memory allocation calls — usually causing the offending application to crash. A far better scenario than setting oom-killer loose. I’ve actually seen troublesome apps hobble along in a configuration like this while the team works on a major refactor & code fixup. So long as the memory leak is infrequent, a configured limits file can make a leaky application seem a lot more stable if, for example, passenger is restarting the rails processes for you.

Step 14: Logging with Syslog

Dropping all of your log files into your RAILS_ROOT/log/production.log may be convenient, but it isn’t necessarily the best way to go. Syslog is a better choice for a production environment. It gives you more control, more configuration options, and will make multi-server setups far easier to debug.

Plus, if you want to get strict about security, an application shouldn’t be able to delete information from its own log file.

Lets just get started with the bare basics. First, install the gem and set up your syslog config.

root@your_host:~# gem install SyslogLogger
root@your_host:~# vim /etc/syslog.conf
# Add the following two lines to the bottom of the file
!rails
*.*                     /var/log/production.log
root@your_host:~# /etc/init.d/sysklogd restart

Then, add these two lines to your environments/production.rb file.

require 'syslog_logger'
config.logger = SyslogLogger.new('someapp_production')

And restart your application. You should now see log statements going to /var/log/production.log

That’s a good start

While there is still a bunch more you can do to lock down and monitor your server, this is a great start. It’ll give you a solid, well organized base on which to build, which we will continue doing in the next installment. Be sure to subscribe to the RSS feed so you don’t miss it!

Photo Credits: Server Rack, Cate… what a mess!, It’s.. Brobee!, A Brand New Day, Olive the mini dachshund