A little background
Unix administrators have been dealing with the same problems for a long, long time. Users and programs contending for resources, outside attackers trying to get in, crashing services. You name it. There are a lot of excellent utilities out there to handle all of these issues. Unfortunately, they go mostly unused.
Ever have an application, of any language, eat up memory or disk space to the point of slowing the entire system to a crawl? Seen a security breach in one web application compromise a whole server? Had a server encounter performance problems that seem to disappear by the time you make it into top? I’ve joined a number of projects where problems like these are constant, nagging issues. They really don’t need to be.
Assume Your Application is not “Well-Behaved”
When you are setting up your server, the general rule is: don’t trust your own rails app. Assume it will leak memory, take up hard disk space, try to run commands it shouldn’t, attempt to delete other application’s databases, and so on. Besides the potential for an arbitrary code-execution bug being exploited by a hacker, you also have to keep in mind the standard bugs and your own “oops!” moments that are part of the development process. Thinking like a sysadmin can give your production boxes a nice safety net.
I am going to go through a series of blog posts explaining a lot of tips, tricks, and best practices for setting up a single server and the general small infrastructures. Today we are going to hit the basics — what I consider to be the bare minimum necessary to get an Ubuntu 8.04 LTS server ready to go and serving your rails application.
The basics
So lets start out with the basics everyone expects.
Step 1: If you haven’t already, create your user & setup sudo
I prefer to set up users with password access disabled. SSH keys are the way to go. If you are setting up a VPS, your root account pretty much needs to have a password, however. It should be long, random, and stored someplace safe.
root@your_host:~# adduser thomas --disabled-password root@your_host:~# usermod -aG sudo thomas
Step 2: Give yourself sudo access
We set up your user sans password before, so you can’t use a password for sudo.
root@your_host:~# visudo # Add the following line %sudo ALL=NOPASSWD: ALL
Step 3: Update your system
Before you forget or spend any time setting up or compiling against old packages.
root@your_host:~# apt-get update root@your_host:~# apt-get upgrade
Step 4: Apt-get the world
Pretty much all the basics you will need for ruby/rails/passenger.
root@your_host:~# apt-get install ruby mysql-server libmysqlclient15-dev apache2 rubygems build-essential wget irb1.8 rdoc1.8 libopenssl-ruby1.8 libreadline-ruby1.8 ruby1.8-dev apache2-prefork-dev libapr1-dev libaprutil1-dev git-core
Use a strong password for mysql’s root!
Step 5: Install rubygems
Ubuntu 8.04 has a pretty out of date version of rubygems. Best to just start from the latest tarball.
root@your_host:~# wget http://rubyforge.org/frs/download.php/60718/rubygems-1.3.5.tgz root@your_host:~# tar -xvzf rubygems-1.3.5.tgz root@your_host:~# cd rubygems-1.3.5 root@your_host:~/rubygems-1.3.5# ruby ./setup.rb root@your_host:~# ln -s /usr/bin/gem1.8 /usr/bin/gem root@your_host:~# gem -v 1.3.5
Step 6: Gem install the world
root@your_host:~# gem install --no-rdoc --no-ri rails passenger mysql rake
Step 7: Create a user for your application
Every application should have its own user, for a few reasons. If you are deploying several applications on the same server, you do not want one application to be able to write, or necessarily even read, the files of another. Later on, when we set up PAM resource limits, having separate users will allow us to prevent any single application from dominating the server’s resources. It also provides a handy mechanism for separating cron jobs, and protecting yourself from an “oops!” catastrophe.
This user will both run the application and deploy it, but will not have sudo access — whatever user you run the application as should never, ever, have sudo access. For any reason. Whatever you deploy the application as should ideally also not have sudo access.
What we are going to do here is create an application user, generate ssh keys for it, and point its authorized_keys file to an /etc/deployers file. Given the assumption that you are going to set up multiple rails applications deployed by the same developers on this server, that is just a nice way to share ssh public key definitions among all of the deploy users you will have.
root@your_host:~# adduser someapp --disabled-password root@your_host:~# su - someapp someapp@your_host:~$ ssh-keygen someapp@your_host:~$ exit root@your_host:~# touch /etc/deployers root@your_host:~# ln -s /etc/deployers /home/someapp/.ssh/authorized_keys
Of course, if you are allowing several unrelated developers and/or companies access to the same server you wouldn’t want to share an authorized_keys file like this.
Step 8: Set up your application directory structure
root@your_host:~# mkdir -p /var/rails/someapp root@your_host:~# chown someapp:someapp /var/rails/someapp/
Step 9: Set up your database
Set up a user for every application that only has access to that application’s database. Doing so is not only a good security measure, but it also helps ensure you don’t fail in a spectacularly epic manner.
If you share database users, you are inevitably going to copy your database.yml and/or password file from another app when you set up a new one. You are also inevitably going to forget to modify the database name in that before calling something destructive like rake db:reset. Lets avoid that.
create database someapp_production; grant all on someapp_production.* to 'someapp'@'localhost' identified by 'a_good_password';
The user should not have permission to create databases on the system, which may be a little inconvenient during setup. It is, however, a good security measure and gives a little more protection from sysadmin “oops!” events.
Step 10: Do a deploy
Run your cap scripts, do your cold deploys, etc. Remember the someapp user isn’t allowed to have sudo access — so just install gems yourself.
Capistrano is currently the best way to handle deployments for rails applications. There are a lot of great guides to using it, so I won’t rehash that part.
Capistrano’s Home Page
A Getting Started Guide
Using git with Capistrano
If you are using capistrano you should now have a directory structure resembling this:
thomas@your_host:/var/rails/someapp$ ls -l * lrwxrwxrwx 1 someapp someapp 43 Aug 5 03:36 current - /var/rails/someapp/releases/20090805033622 releases: total 20 drwxrwxr-x 14 someapp someapp 4096 Aug 5 03:18 20090805031830 drwxrwxr-x 14 someapp someapp 4096 Aug 5 03:20 20090805032022 drwxrwxr-x 14 someapp someapp 4096 Aug 5 03:33 20090805033331 drwxrwxr-x 14 someapp someapp 4096 Aug 5 03:36 20090805033549 drwxrwxr-x 14 someapp someapp 4096 Aug 5 03:36 20090805033622 shared: total 20 drwxr-xr-x 13 someapp someapp 4096 Aug 5 03:06 cached-copy drwxr-xr-x 2 someapp someapp 4096 Aug 5 03:18 config drwxrwxr-x 2 someapp someapp 4096 Aug 5 03:06 log drwxrwxr-x 2 someapp someapp 4096 Aug 5 02:42 pids drwxrwxr-x 2 someapp someapp 4096 Aug 5 02:42 system
Step 11: Set up Passenger
Follow the excellent instructions & point to the proper path in your apache config.
root@your_host:~# passenger-install-apache2-module
In ubuntu you would set up the http configuration by creating a new file in /etc/apache2/sites-available with the recommended passenger configuration and soft linking it to a file in /etc/apache2/sites-enabled when the app is ready to go up.
root@your_host:~# vim /etc/apache2/sites-available/someapp root@your_host:~# ln -s /etc/apache2/sites-available/someapp /etc/apache2/sites-enabled/ root@your_host:~# /etc/init.d/apache2 restart
Beyond the Basic Setup
So generally it seems like everyone stops at that passenger step. The application is running, everyone is happy, right? Well, there is actually a bunch more to be done to save yourself some late nights later on.
Step 12: Shorewall Firewall
I like to run a firewall on pretty much every server I manage. They are easy to set up, and protect you from, well, you. A firewall doesn’t make it OK to have your database listening on an external IP address, but it can certainly mitigate the damage if that does somehow happen.
Getting shorewall on your server is pretty easy. First we’ll apt-get it, then we’ll copy the example config, tweak it, and start the firewall.
Rule number one with firewalls, routing, and network configuration. Make sure you can get into the system without ssh. If it is a physical server, this isn’t something to do offsite at 2AM. If it is a VPS, make sure you have the necessary credentials to log in via the web console.
The example file used here is for a server with one public facing interface/ip address. If your setup is different, you’ll need to read up on the shorewall docs. There are example files for a two-interface (private/public) server that make the process almost as easy. They are in /usr/share/doc/shorewall-common/examples/two-interfaces.
root@your_host:~# apt-get install shorewall root@your_host:~# cd /etc/shorewall/ root@your_host:/etc/shorewall# cp /usr/share/doc/shorewall-common/examples/one-interface/* ./ root@your_host:/etc/shorewall# vim rules # Add the following two lines SSH/ACCEPT net $FW HTTP/ACCEPT net $FW root@your_host:/etc/shorewall# vim shorewall.conf # Change STARTUP_ENABLED=No to STARTUP_ENABLED=Yes root@your_host:/etc/shorewall# vim /etc/default/shorewall # Change startup=0 to startup=1 root@your_host:/etc/shorewall# /etc/init.d/shorewall start
Done, you have a firewall.
Shorewall has a ton of features. You can read more about it at shorewall.net.
Step 13: PAM Limits
When your Linux box starts to run low on memory it fires up a psychopathic little helper call oom-killer. At first blush this may sound reasonable, fire up something to kill off whatever is taking up the resources. Except that’s not what oom-killer does. It just guesses — and usually ends up killing a lot of things you don’t want it to. Your leaking application might cause oom-killer to kill your apache or mysql process, or, more likely, other unrelated rails processes. Don’t let it get to that point.
OOM-Killer really isn't your friend.
Instead, lets set up a simple limit statement in /etc/security/limits.conf.
I can’t provide a recommended number to use here. Every application is different. The number here limits the total amount of RAM the user may use, not each rails process. So, if you have a maximum of three passenger processes that tend to max out at 30MB a piece and a cron job that tends to max out at 40MB during its runs, then a reasonable value here would be 200MB (we don’t want it fussing over small spikes). To set the limit as 200MB, you convert it to KB (204800) and enter it as shown below.
root@your_host:~# vim /etc/security/limits.conf someapp hard as 204800
Now, instead of having the OS guess at which process to kill, the OS will start denying someapp’s memory allocation calls — usually causing the offending application to crash. A far better scenario than setting oom-killer loose. I’ve actually seen troublesome apps hobble along in a configuration like this while the team works on a major refactor & code fixup. So long as the memory leak is infrequent, a configured limits file can make a leaky application seem a lot more stable if, for example, passenger is restarting the rails processes for you.
Step 14: Logging with Syslog
Dropping all of your log files into your RAILS_ROOT/log/production.log may be convenient, but it isn’t necessarily the best way to go. Syslog is a better choice for a production environment. It gives you more control, more configuration options, and will make multi-server setups far easier to debug.
Plus, if you want to get strict about security, an application shouldn’t be able to delete information from its own log file.
Lets just get started with the bare basics. First, install the gem and set up your syslog config.
root@your_host:~# gem install SyslogLogger root@your_host:~# vim /etc/syslog.conf # Add the following two lines to the bottom of the file !rails *.* /var/log/production.log root@your_host:~# /etc/init.d/sysklogd restart
Then, add these two lines to your environments/production.rb file.
require 'syslog_logger'
config.logger = SyslogLogger.new('someapp_production')
And restart your application. You should now see log statements going to /var/log/production.log
That’s a good start
While there is still a bunch more you can do to lock down and monitor your server, this is a great start. It’ll give you a solid, well organized base on which to build, which we will continue doing in the next installment. Be sure to subscribe to the RSS feed so you don’t miss it!
Photo Credits: Server Rack, Cate… what a mess!, It’s.. Brobee!, A Brand New Day, Olive the mini dachshund
Related posts:
Great article guys!.
We wish you the best with Envy Labs too.
Best regards,
Martín
Nice article. Can you talk about monitoring tools that are recommended?Looking forward for the second Installment.
Dumma
Great article! If only you just created a bunch of chef recipes for this ;)
@Tawheed: Agreed, in due time! Teach the concepts first, then make it easy. ;)
Always useful to see other people’s deployments. A++++ would click again.
This comment was originally posted on Reddit
For the monitoring part (part 2?), you might want to take a look at these munin plugins: http://github.com/barttenbrinke/munin-plugins-rails/
I’d add a quick section for a basic iptables setup as well, with holes for the appropriate services.
This comment was originally posted on Hacker News
Instead of soft linking to a sites-available file, why not use apache’s a2ensite?
He’s using Shorewall for that.
This comment was originally posted on Hacker News
Thanks Thom, What about tripwire? It’s usually a good pick when setting up a new server.
My 3 most important rules for Linux servers are: 1. Run software update! 2. Don’t allow everyone to access sensitive ports. Moving SSH to some nonstandard port is not enough. Use iptables to limit access to it from only certain static IPs. 3. Run SELinux! If you think its only purpose is to make your life more difficult, you have a lot more studying to do. (Also, make sure you don’t use a distro that comes with a badly broken selinux config)
This comment was originally posted on Hacker News
It’s kinda fluffy. Most of the advice is good, but basic.One peeve I have is that it’s pushing a firewall solution on the local box. In general, that’s best done off-host. Even better are MAC security solutions like AppArmor, which is really easy to set up (frankly much more so than a non-trivial set of firewall rules) and much more capable in general. Typical applications need access to only a handful of paths and capabilities that are well-defined in the documentation.
This comment was originally posted on Hacker News
Considering how firewalls and AppArmor do completely different things, I’d say you need both.
This comment was originally posted on Hacker News
Use iptables to limit access to it from only certain static IPs.Or just disable password logins entirely.
This comment was originally posted on Hacker News
How hard is it to provide attribution for the photos used? e.g.http://www.flickr.com/photos/thomashawk/287666827/
This comment was originally posted on Hacker News
After setting up users and disabling the root account, I usually set up public key authentication and disable password logins. It’s convenient AND secure (OpenSSL bugs aside).
This comment was originally posted on Reddit
Yeah, that’s really much more secure ;-)http://www.debian.org/security/2008/dsa-1571
This comment was originally posted on Hacker News
Set Up Your Server Right, Part 1 « Envy Labs http://is.gd/240H4
This comment was originally posted on Twitter
Set Up Your Server Right, Part 1 « Envy Labs http://snipr.com/ot9od
This comment was originally posted on Twitter
It is attributed. Double-check the bottom of the blog post.
This comment was originally posted on Hacker News
This would only protect you against password guessing, but not against any remotely exploitable security holes in the SSH (or any other) daemon.
This comment was originally posted on Hacker News
We could automate all these steps using scripts. A very good example is the scripts by RightScale – http://github.com/rightscale/right_link/tree/master
This comment was originally posted on Hacker News
For us geeks out there: Set Up Your Server Right, Part 1 http://bit.ly/1lQ6eY
This comment was originally posted on Twitter
Turn off password logins and move ssh to a different port.
This comment was originally posted on Hacker News
He should have used something like puppet so he can repeat this process on any server.
This comment was originally posted on Reddit
Every sysadmin has their own fingerprints. I read some of the suggestions and kind of cringe since they don’t match my preferences (not that they’re absolutely bad). One of my current aspirations is to "use the tools the way they were intended". Rather than perverting the core components or skinning over them with other admin tools, I believe it’s important to appreciate the system as designed and take advantage of the toolset provided.1) sudo NOPASSWD is the moral equivalent of making that user root. I don’t care for it. My preference would be to configure sshd_config with "PasswordAuthentication no" and continue to use passwords in the regular way, with password complexity enforcement if your distribution supports it. I’m not going to blanket condemn ssh public key auth, since it’s generally smart. However, you can not enforce a requirement that the public key be encrypted on the remote host. If someone is undisciplined with the distribution of their public keys they can end up turning a single box exploit into free reign of the network, particularly if NOPASSWD is enabled.
2) I prefer manually editing iptables rules myself, using -I and -D. It’s certainly not intuitive at first blush, more like using "ed" than "vi". I’ve gained an appreciation for using the "-m comment -comment ‘this rule does blah’" construct. The benefit of this model is that an unfamiliar sysadmin won’t immediately know that a host has another FW package installed, but if a change is needed everything is documented and commented in an "iptables -L" — a standard diagnostic command. Furthermore, firewall rule generators often create unreadable rulesets, making diagnosing specific problems tough.
A specific nit I have against Shorewall is that it requires (last time I checked) you to edit a bunch of different files, using boilerplate recipes that aren’t really much simpler than just writing the rules yourself. It could be a big win if you had multiple platforms to support, I suppose.
This comment was originally posted on Hacker News
I’ve never been a fan of changing SSH’s port.If you’re going to firewall SSH, changing the port is redundant.. The only reason to change the port is to prevent brute force attacks, and the firewall will do that for you.
This comment was originally posted on Hacker News
re: 1) sudo NO PASSWD…So the problem I have with this is that it collides with using Puppet to manage your users–Puppet will have to know about the users’ Unix passwords.
It should be fairly straightforward to write a script that tests for SSH keypairs with an empty passphrase, simply try to authenticate an SSH agent by loading the user’s key.
This comment was originally posted on Hacker News
I can’t speak to puppet, but I would expect the proper thing to do is to run puppet as root, or possibly create a puppet user with NOPASSWD if you must. There’s really no reason to not run as root if you’re going to run NOPASSWD ALL, since anyone with the slightest bit of deviancy in mind (or just annoyance with not being root) is going to "sudo su -", and sudo is done.You can probably write a script that searches for empty-phrase keypairs, but this would be a client side thing. The private key is not disclosed to the server, and all operations involving the private key happen post-decoding, so this is unenforceable server-side. It would be an interesting policy/hygiene enforcement tool but would be of little or no use for security purposes.
This comment was originally posted on Hacker News
Set Up Your Server Right, Part 1 « Envy Labs http://ff.im/-6eVl3
This comment was originally posted on Twitter
First thing I do is configure the backup software!It defies belief that a lot of people don’t consider this in these guides.
This comment was originally posted on Hacker News
[...] to Design Online Stores – Inspect Elem 40 (NEW) High Quality Adobe Photoshop Tutorials Noupe Set Up Your Server Right, Part 1 Envy Labs In-Field Labels jQuery Plugin The Little But Really Useful Guide to Creativity Why Teens Dont Tweet [...]
Set Up Your Server Right, Part 1 « Envy Labs http://bit.ly/GNbRv
This comment was originally posted on Twitter
Set Up Your Server Right, Part 1 « Envy Labs: http://bit.ly/11xdPR
This comment was originally posted on Twitter
Set Up Your Server Right, Part 1 « Envy Labs: http://bit.ly/11xdPR RT @barryroodt
This comment was originally posted on Twitter
* List all installed applications and libraries with versions* Subscribe to all security advisories of those components
* Look for problems that hit your setup
* Take patch/corrective action when necessary
* OS update is good, but is not the final word in what you need to patch. You need to do your own homework
This comment was originally posted on Hacker News
[...] Set Up Your Server Right, Part 1 « Envy Labs [...]
Still…. it really cleans the logs up. It’s not safer, but the logs do become easier to read in a pinch.
This comment was originally posted on Hacker News
Set Up Your Server Right, Part 1 http://bit.ly/Nm913 #postrank #entrepreneur
This comment was originally posted on Twitter
Set Up Your Rails Server Right: http://bit.ly/SERAm
This comment was originally posted on Twitter
“root@your_host:/etc/shorewall# vim rules”
Couldn’t agree more. :-)
On a system with multiple administrators sudo provides an audit trail.
This comment was originally posted on Hacker News
And disable root login from ssh.
This comment was originally posted on Hacker News
In addition to the advantage that notaddicted mentioned, you can also limit the user(s) to a specific command, set of commands, and so forth. Using sudo in that fashion doesn’t give a user root access to the entire system, so it’s quite different from "being" the root user.
This comment was originally posted on Hacker News
Huh? The firewall logs? Got grep -v?
This comment was originally posted on Hacker News
D’oh, upvoted instead of clicking reply =(What I mean is that in order to manage user accounts, the management tool (Puppet) will have to know what the encrypted password is so it can insert it into /etc/shadow. Otherwise you have no password and must rely on NOPASSWD in sudoers if you want to log into that managed machine and use sudo.
If the system doesn’t have a password for you in /etc/shadow, sudo can’t authenticate you via getpwent or whatever.
So your only two options are to write a tool for users to update their password in Puppet directly/indirectly, or allow NOPASSWD and religiously check for empty passphrases on SSH keypairs.
This comment was originally posted on Hacker News
The thing is that a lot of the simple scripts only bang on 22. Moving the port means that a lot of scripts will move on to another server.
This comment was originally posted on Hacker News
@Donnnie: Habit mostly. I’ve just gotten used to soft-linking the world. ;)
@Rob Ottaway: I may go there. Tripwire isn’t something I set up on every server, though it is certainly a good practice.
love the brobee pic. Yo Gabba Gabba!
[...] Set Up Your Server Right, Part 1 « Envy Labs [ server linux howto webdev ] [...]
Your syslog target is “!rails” but the target you declare in the Ruby configuration is ’someapp_production.’ I’m pretty sure those have to be the same.
Set Up Your Server Right, Part 1 « Envy Labs http://bit.ly/33fjKa
This comment was originally posted on Twitter
[...] Set Up Your Server Right, Part 1 « Envy Labs [...]
К слову, лучший способ обезопасить себя от навязчивых мобилок – задействовать Подавитель телефонов
sudo ALL=NOPASSWD: ALL
looks too scary o_O