.. / OpenBSD + https for $3 / month

Security at any price... ;-)

I recently realized it was time to overhaul my web server. The previous instance was setup years ago; that machine was expensive, obsolete and embarrassingly overpowered.

While I was futzing with it anyway, I decided to switch from Ubuntu to OpenBSD, and move my hosting to a country with reasonable privacy protections. It was also well past time to move to HTTPS.

I like the fact that OpenBSD has a stable and well-documented base installation. Rather than reinvent init and package management, the developers have been solving problems that I have hit over the years. Below, when I configure httpd, you'll see what I mean.

What this guide isn't

In this guide, I walk through the steps I take to spin up a VM for my low-traffic personal web server. Beyond conserving memory to drive the price of the VM instance down, performance tuning is out of scope. Similarly, you won't find any information about dynamically generating pages, hosting wordpress, or other such things -- static pages suffice for me so far.

This guide also isn't going to help you NSA-proof your box. I like to think they'll waste more money breaking in than they'll get out of me in taxes for the next few years. I haven't run the math. I'm sure it's depressing. I digress.

Finally, this guide won't magically improve the security of the public cloud. By the time you read and understand it, you should understand how a malicious cloud provider could transparently proxy https connections to your webserver, or simply man-in-the-middle your management console, intercepting your passwords and keys in the process. I do explain how to set up encryption of data at rest. Assuming the cloud provider is not current compromised or actively harvesting such information, this at least prevents simple copying of the base OS image after the fact.

Choosing a cloud provider

I found a few articles on the topic, which gave me a list of acceptable countries, and then went hunting for a cloud provider. I only found one option that is (1) less expensive than my old t1.micro instance at AWS, (2) not US based and (3) OpenBSD compatible.

To make a long story short, I went with CloudSigma. After signing up, I found this reddit thread that recommends vultr, tilaa, serveraptor, ramnode bytemark and transip. I did not have chance to look at them.

CloudSigma claims each of their international divisions is structured as an independent company, and let you choose whether you want your account to be governed under Swiss, Australian or US law. I'm no lawyer, but I have to wonder if the Swiss arm would cave if the US arm was facing an injunction or an NSL. It's better than nothing, I suppose.

I liked the idea of hosting in Australia (moving compute to Europe is becoming cliché, after all). Despite their strong privacy laws, the Australian government keeps trying to censor the Internet, so I settled on the Zürich data center.

NOTE: Since this guide was written, the Swiss passed a referendum that will increase monitoring of cross-border traffic. I plan to move my hosting elsewhere, which is a shame, since I have had good luck with CloudSigma.

Provisioning a VM

I've found that the smallest-available instance size is plenty for serving web traffic. This site is hosted on a 0.25GHz VM with 0.25GB of RAM. The disk image is 1GB. With current reserved pricing, this VM runs a bit more than $3/month.

The price calculator estimates $1.35/month, but it assumes "free tier" pricing discounts. Unfortunately, you have to hit $10/month spending across your account to get that price, and we'll be well below that with just one small VM.

I enabled HostCPU, NUMA and Multi-CPU in the CloudSigma settings. This all worked fine. For the initial install, I recommend cranking up the RAM and CPU. Once the install is complete, shut the VM down, and set them to their desired final values.

Downloading OpenBSD

The next step is getting the OpenBSD install image to CloudSigma. They helpfully have the image in their library already; if you trust them, you can just use that one. Otherwise, you can download it yourself from openbsd.org, or buy a CD-ROM set.

Of course, CloudSigma could just check the fingerprint of your uploaded ISO, and then replace it with their secret malicious image at install time. Remember that note above about public cloud security?

Note that, at some point during installation, when it unpacks from the ISO, the OpenBSD installer will complain that it cannot validate the packages. That's by design; the installer assumes that anyone that bothers to tamper with the packages will also bother to tamper with the installer, so it does not bother to validate the signatures of packages that came from the same source as the installer. I found this confusing, and initially thought the images were broken.

This brings us back to actually downloading the ISO. You can grab a copy from a mirror, though you'll quickly notice that none of the mirrors are encrypted or authenticated in any way. The OpenBSD team has made a conscious decision not to trust https' public key infrastructure when signing their releases. This probably seems crazy to you now, but once you see how the sausage is made (below, when we setup our key), it will make a lot more sense.

Anyway, instead of https they use a tool called signify, which needs a trusted copy of the OpenBSD public key to be of any use. If you have a trusted installation of OpenBSD 5.5+ lying around, it'll contain the key. Of course, that still doesn't help if we don't trust the cloud provider not to man-in-the-middle the image after we upload it. Bootstrapping trust is hard.

At this point, it is probably best to loosen your tinfoil hat a bit, and just use the image CloudSigma provides. If you must download from a mirror, be sure to get the big one (installXX.iso), since the net install image won't have enough disk space to verify package signatures at installation time. Of course, it downloads the packages via an unauthenticated connection.

Installing OpenBSD

Honestly, at this point, you should just hop over to the excellent OpenBSD installation guide. That, plus these rough notes should allow you to exactly reproduce my setup.

Be sure to drop into a shell and turn on whole-disk encryption before beginning the install process.

The partition table should look like this:
Partition Size
/ 400M
swap 80M
/tmp 50M
/var 100M
/var/www ~4000M

When it asks for a "short" hostname, just go ahead and type the fully-qualified domain name in. That will save some trouble later.

You won't need the comp (compilers) package, games, or any of the X11 stuff. bsd* and base will suffice. I also install man.

Setting up https

In order to get https working properly, we'll need to do two things: Set up a web server, and obtain a valid certificate. OpenBSD's base image includes httpd, a simple, security-oriented web server, which we use here.

Certificate generation overview

We use Let's Encrypt to get a free certificate. Let's Encrypt is based on the ACME protocol, which, as we configure it, roughly works as follows:

  1. We need to prove to the certificate authority (CA) that we have control over the web server associated with our domain name.
  2. We request a certificate, and the CA responds with a challenge -- a random number -- that we publish at a well-known URL on our web server.
  3. The CA uses a number of different client machines to request the challenge via http.
  4. If all the clients receive the correct challenge, the CA will issue a certificate for us.

The most obvious problem with this protocol is that any attacker capable of reliable man-in-the-middle attacks against your server can also obtain valid certificates for your domain, and therefore setup a proxy that decrypts all your encrypted traffic in route!

The main risk for the attacker is detection: The certificate displayed in the web browser won't match the one on your server. Most people do not manually check certificates, so this is not that big of a risk to the attacker in practice. In particular, if the attacker is interested in a subset of the users of the site, and can easily control which users' traffic is decrypted, then it is unlikely anyone involved in the configuration of the server will encounter the attacker's certificate.

The other way for an attacker to be detected is for the CA to publish a list of all the certificates it grants. This would allow site owners to periodically check to see if a certificate for one of their domains was unexpectedly issued. This is not a great defense against targeted attacks, since there is a window of vulnerability between the point in time when the certificate is issued and the point in time where it is revoked.

Note that this sort of attack impacts all https traffic, not just sites that use the ACME protocol. Also, it is apparently standard practice to use the ability to publish files on web servers as a proxy for proving ownership of servers; ACME just helps automate the (already weak) process. Oh well, it's not like people trust this stuff with anything important, like credit card numbers... The important thing is that it's now really convenient to set up https, and some security is better than none, right?

Choosing an appropriate client and installing prerequisites

The official Let's Encrypt client is written in python, and pulls in a bunch of packages using pip (a python package manager), amongst other things. It would be a shame to go to all of this effort to have a lean-mean VM image with carefully vetted software, and configure it to auto-update code from who-knows-where each time it needs to renew its certificate.

Fortunately, there is an alternative. letsencrypt.sh, is a bash script with minimal dependencies that provides the same basic functionality as the python client.

We'll need to configure the OpenBSD package manager, and install bash and curl. While we're at it, I also install git and rsync, which I use to publish the web site:

export PKG_PATH=http://mirror.switch.ch/ftp/pub/OpenBSD/$(uname -r)/packages/$(machine -a)/

pkg_add bash curl 
pkg_add git   # to clone letsencrypt.sh
pkg_add rsync # to publish website

cd ~ ; git clone https://github.com/lukas2511/letsencrypt.sh

Next, we need to tell the Let's Encrypt client which domain name we are trying to set up. Create a file called domains.txt in the root directory of the Let's Encrypt checkout:

cd ~/letsencrypt.sh/ ; hostname > domains.txt

We also need to configure it for use with httpd, and to provide a contact email in case there is a problem with the certificate.

Create a file ~/letsencrypt.sh/config.sh

WELLKNOWN=/var/www/htdocs/.well-known/acme-challenge
CONTACT_EMAIL=YOUR@EMAIL.ADDRESS

Next, setup httpd by creating /etc/httpd.conf

ext_addr="*"
server "YOUR_HOSTNAME" {
  listen on $ext_addr port 80
#  listen on $ext_addr tls port 443
#  tls {
#    certificate "/root/letsencrypt.sh/certs/YOUR_HOSTNAME/fullchain.pem"
#    key "/root/letsencrypt.sh/certs/YOUR_HOSTNAME/privkey.pem"
#  }
}

types {
        include "/usr/share/misc/mime.types"
}

Then run this to customize the file, or hand-edit it:

sed -i s~YOUR_HOSTNAME~`hostname`~ /etc/httpd.conf

Finally, enable httpd:

echo 'httpd_flags=' >> /etc/rc.conf.local
/etc/rc.d/httpd start
Schedule a daily cron job to freshen the certificate by creating a new file /etc/daily.local

#!/bin/sh
mkdir -p /var/www/htdocs/.well-known/acme-challenge
PATH=${PATH}:/usr/local/bin /root/letsencrypt.sh/letsencrypt.sh --cron
pkill -SIGHUP httpd

Let's Encrypt issues keys with short lifetimes; the cron job automatically replaces the key when necessary. letsencrypt.sh checks the certificate expiration date, and does not request a new certificate until the old one is at risk of expiring. The pkill line forces httpd to re-read its configuration file, and to reload the certificate.

You'll need to setup your domain name to point to the IP address of your server. You can find the IP by running ifconfig. The dynamic IP addresses issued by CloudSigma seem to be stable, unless you stop and start your virtual machine instance. Alternatively, you can pay extra for a static IP, but doing so will significantly increase the cost of hosting your virtual machine.

Once the DNS entry is created, and you can access your server via http, create the certificate by manually invoking the cron job:

. /etc/daily.local

Now that you have certificates, you can enable https:

sed -i~ 's~^#~~' /etc/httpd.conf
/etc/rc.d/httpd restart

At this point, you should be able to access your web site via https.

Errata / TODO

This guide leaves you with a network facing service that runs as root. Ideally, letsencrypt.sh would be installed as its own user with its own special crontab.

I suspect it is possible to remove the dependency on bash, since ksh is similar.

Drop me a line at sears+bsd-https@cs.berkeley.edu to comment on this article.