bad network

(basic) SSH Keys Part 2: Agent forwarding, Adding Keys

2020-08-20T00:00:00+02:00

There's a million and one things you can do with ssh keys, but as a non-power-user I find 99% of the time I'm either trying to get a shell on a different machine or moving a file between machines I trust. So for a long time I didn't really touch ssh configuration.

But it's always good to make brief exceptions to laziness to quickly check whether you could be more lazy, and there are plenty of nice ways to save yourself some typing and fuss sprinkled around.

For simple added SSH key quality of life for someone new to using keys, there are two configuration keywords in particular you might enjoy trying out first.

Here's my setup for a jump through a proxy to a machine in my home:

Host simurgh
HostName 192.168.1.199
ProxyJump jumphost.example.com
ForwardAgent yes
AddKeysToAgent yes

ForwardAgent will allow the remote destination to use the ssh agent on my local machine, rather than needing to supply keys of its own. I don't keep any keys on this remote host, and this means I won't miss them.

AddKeysToAgent will add the key used to make this connection to my ssh agent, so I will have continued access to it for other purposes. (You can specify a lifetime or expiration if you prefer).

Taken together, these allow me the flexibility to connect from this remote machine outward without thinking too hard about it, or to repeatedly connect from the local to the remote machine during a work session without having to type passphrases until I'm ready to throw things.

If you're a newer user who maybe only has one or two machines under your control, though, using ForwardAgent in particular will mean you won't need to store private keys on any hosts you don't own to reap their benefits, which might make you feel more comfortable experimenting with more interesting ways to use them!

ssh-agent

To take advantage of those two keywords, we will need a running ssh agent process on the local machine. An ssh agent holds private keys to be used automatically for authentication. This can be safer than using keys generated without passphrases, since you'll need to provide your passphrase to add the key to the agent, but it can make actual use of keys much more convenient.

If you logged into your local machine with xenodm, you may already have keys in your ssh agent available to forward and a running agent collecting any new ones you use. In that case, our setup above likely already works and you will be able to use your keys on the remote machine as soon as you connect, yay!

But if you're working in the terminal or otherwise have not made available the key you want, you'll need to load keys into your ssh agent. To walk through this we'll need ssh-agent(1) running, and we'll either use ssh-add(1) directly or take advantage of making ssh connections configured with AddKeysToAgent.

Let's first see how a connection works without those last two configuration lines using our two new keywords. Here, we're simply connecting through the jump host for a no-frills ssh session using a key.

valefor$ ssh simurgh    
Enter passphrase for key '/home/pamela/.ssh/id_ed25519': 
Enter passphrase for key '/home/pamela/.ssh/id_ed25519':

Ehhh. I don't know about this. Because both the jump host and the destination machine need to be checked against my key, I'll have to provide passphrases for both legs of the connection. I'm going to move a whole bunch of stuff back and forth today. One side needs more cute hippo pictures; the other side needs more reaction gifs starring angry elephants... I may be at this a while. I'm not going to type my passphrase in twice each time I connect!

Let's first make sure ForwardAgent and AddKeysToAgent are both set to "yes" for the connection we wish to make in the config file, as shown in the first section.

Are there any keys in the local ssh agent to start? You can get this list using ssh-add -L (and empty it with -D again to experiment).

valefor$ ssh-add -L
Could not open a connection to your authentication agent.

OK, we don't have an ssh agent running at all! So let's get that going.

valefor$ ssh-agent  
SSH_AUTH_SOCK=/tmp/ssh-Ns6XKrcJ2DZc/agent.17087; export SSH_AUTH_SOCK;
SSH_AGENT_PID=71952; export SSH_AGENT_PID;
echo Agent pid 71952;

As explained in ssh-agent(1), these are the shell commands we need to set the environment variables and set up our agent. But this isn't going to do us much good for our purposes; it's not meant to be run like this and used to authenticate repeatedly. Instead, let's try using "eval" as described in the manual.

valefor$ eval `ssh-agent`
Agent pid 13373

Now we have actually used what ssh-agent gave us, rather than just admiring it once, and we have a running agent, ready for keys. A process can't change the environment variables of its parent process, so any direct use of ssh-agent (i.e. "ssh-agent ksh" to open a new shell with these settings) will only apply to that one process and its children. Evaluating these commands instead allows us to load the environment variables ssh-agent gave us and then continue on to issue other commands as we like, with ongoing access to our agent.

Remember how I said you could look at /etc/X11/xenodm/Xsession to see how xenodm loads in keys upon login? That's done using eval, too, and your keys will be available to you in every fresh xterm window you open! If you're planning to connect via shell and want to be able to use an agent every time, you could even add eval 'ssh-agent' to your .profile (though this will also kill attempts to forward a different ssh agent if you later connect to this machine remotely, so it's probably not a great default).

Adding keys to ssh-agent

valefor:~$ ssh-add -L
The agent has no identities.

The agent is running on the local machine now, but it's empty. I'll use ssh-add to provide identities to the agent, teaching it about some of the private keys this machine holds.

valefor:~$ ls .ssh/        
authorized_keys config          id_ed25519.pub  id_rsa.pub
backups.pub     id_ed25519      id_rsa          known_hosts

valefor:~$ ssh-add         
Enter passphrase for /home/pamela/.ssh/id_rsa: 
Identity added: /home/pamela/.ssh/id_rsa (pamela@valefor.blah)
Identity added: /home/pamela/.ssh/id_ed25519 (pamela@valefor.blah)

You can specify particular keys with ssh-add private-key-file to add them individually, or simply running ssh-add by itself will load in the usual (i.e. default) suspects for you automatically. There are now two different private keys known to my ssh agent, and if I ssh to a host where one of these keys is authorized, barring specialty settings on either side, I'll simply be ushered directly to a remote prompt like a hecking VIP!

Now, let's try emptying out the agent and instead add a key automatically by using the AddKeysToAgent keyword we specified in that first configuration file.

valefor:~$ ssh-add -D
All identities removed.

Connecting to remote host "simurgh" should automatically furnish the key we use for the connection to our local agent because of AddKeysToAgent, and let us use the key there as well, thanks to ForwardAgent. Let's go ahead and connect to the remote host, and look at what these two handy configuration lines have done for us.

valefor:~$ ssh simurgh
Enter passphrase for key '/home/pamela/.ssh/id_ed25519': 
Enter passphrase for key '/home/pamela/.ssh/id_ed25519': 
Last login: Tue Aug 11 01:31:04 2020 from 192.168.1.1
OpenBSD 6.7 (GENERIC.MP) #3: Tue Aug 11 09:21:14 MDT 2020

Welcome to OpenBSD: The proactively secure Unix-like operating system.

Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code.  With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.

simurgh$ ssh-add -L
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII42PVL153xszd1SLBiNB7fTwQvIa8HKXxymx5JBbUZz    pamela@valefor.blah

Yay! We had to type the passphrase because the key we needed wasn't already in the agent, but the local agent is being forwarded successfully. I can now connect from simurgh to anywhere this particular key grants me access. But will the key that was added when we made this connection stick around even after the connection closes?

simurgh$ exit
Connection to 192.168.1.199 closed.


:::text
valefor:~$ ssh-add -L
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII42PVL153xszd1SLBiNB7fTwQvIa8HKXxymx5JBbUZz    pamela@valefor.blah

Yep, loaded into the agent and ready for the next use. I won't need to type my passphrase if I connect to simurgh a second time. My agent will handle authentication for me for as long as it holds that particular key.

If I connect somewhere else using ForwardAgent now, my key (and any others my agent might hold at that point) will be available for that remote host's use, even if it wasn't the key used to authenticate that session. This can really come in handy both from an "I don't want to think about which key I use where" perspective, and for working with more complex configurations to suit your specific needs and habits.

There are many options and directives you can play with, but hopefully this gives you something simple and useful to try out when first setting up a workflow using ssh keys.

(Basic) SSH Keys Part 1: Simple setup + ProxyJump

2020-07-23T00:00:00+02:00

Last year, curious, I asked a group of users and developers how they manage their SSH keys. Most people responded that they keep one unique set of keys per machine. Some have a work pair and a home pair. Some have keys specific to automated processes, or particular functions. Keys for common scheduled tasks are popular.

For my part, I have simply been using a single key on my laptop to reach much of the outside world. But how to go the other direction? Today, I'd like to add a keypair onto a machine in a datacenter where I have an account, so I can use that to reach a machine at my home from anywhere.

Here's what I hope is an approachable guide to get a beginner started with keys. We will create a set of keys to use first for a simple connection, then one where a third party is needed to access machines or access files from a private home network (like my folder of Fiona the Hippo baby pictures) from the outside world.

On Keypairs

We create pairs of public and private keys with ssh-keygen(1). By default these both live in ~/.ssh/ on your local machine. You should add the contents of the public key (only!) to the remote authorized_keys file (~/.ssh/authorized_keys) on any machines to which you plan to connect.

Everyone has a tortured metaphor for public key encryption, and here's mine:

Do you remember playing with "invisible ink" as a child? If you mail people a bottle 
of your special ink blend, you can go to one of them, introduce yourself, and ask them to
sign something for you using that ink. Only you know the secret formula, so only you will
know whether to hold the paper to a light bulb, brush grape juice over it or hold
it under a black light to make their signature visible. You can then show them their own 
signature as proof that you knew the secret.

In this metaphor, the remote server was never a child and is therefore very impressed by your wizardry, but you can read about how this actually works in RFC 4252 and friends.

When you connect to a remote server from your local machine, you present public key options to the server to compare with the set of keys they recognize. (Password authentication is your fallback, unless disabled). If a match is found, the remote machine will use the shared public key to randomly generate an encrypted message for your local private key to unlock. If you can decrypt it and present your answer, the remote server then knows you're using the correct private key. It would take it absolutely ages to determine what that private key is, but it will trust that the key is your own and grant you access.

The upshot of all this is that you can share your public key around however you like to gain access without needing a secure method to share a password, and you can memorize a passphrase connected to one key usable in multiple places, rather than individual passwords for each machine. Several people know my laptop's public key and can give me access to any remote machines they wish. (And they are actually doing that, so I'm a big fan!)

Generating Keypairs

My crypto instructor emphasized two things to the point of annoyance:

Don't create your own cryptographic solutions in a vacuum. Their weaknesses will include every single thing which might occur to someone else but didn't occur to you.
Your perfect solution would be something which can never be guessed. We deal in guesses which take an extremely long time.

Neither point is particularly relevant here, except as an evergreen reminder that it's best to use the cryptographic standards which are well-studied and to accept that they are not infallible. But the weakest link in our privacy might well be us, and whether we successfully keep our private keys inaccessible. Retiring and replacing keys periodically won't hurt.

We're also limited by compatibility... your encryption method of choice may not be available everywhere. At the moment, we have a number of options for use with OpenSSH:

 -t dsa | ecdsa | ecdsa-sk | ed25519 | ed25519-sk | rsa

If you run ssh-keygen without specifying details, you currently receive rsa keys. I use ed25519, and so do the others I asked, but if you connect to something running a positively ancient incarnation of ssh you might be out of luck with those and need to generate a different type.

I'll make us an example keypair to use, sticking with the defaults.

valefor:~$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/pamela/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/pamela/.ssh/id_rsa.
Your public key has been saved in /home/pamela/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:G0c/gGyAd2XVGd7QqMvIiLlDCALuK6lht/CpIA5bAhI pamela@valefor.blah
The key's randomart image is:
+---[RSA 2048]----+
|     ..  .o...o= |
|.   . .o.o   .+o.|
|E    . .+ o  .. .|
|.o.    . . o.    |
|+. . . oSo.oo.   |
|o.  . + .+o o.   |
|*+o. . ..        |
|O*+ o o          |
|=o.+   .         |
+----[SHA256]-----+

So now I have two new files corresponding to the two keys stored within ~/.ssh, and a lovely free octopus tentacle artwork. Score!

I also created a second set of keys for more specific purposes at the same time, using ssh-keygen -t to specify ed25519. I'll easily be able to tell these sets apart, since by default they're always named id_(type) and id_(type).pub. OpenSSH will look for files in ~/.ssh named any of the id_* combinations it offers by default. If you choose to rename or move your keypair, use ssh -i (/path/file) to point at it. I know people who separate out keys they expect to rotate frequently, or use at particular events, for example.

When generating keys, you're given the chance to password-protect access to the file containing your private key (and should probably do so). That's just done locally. The passphrase can be updated later, or even left empty and the key's use restricted in other ways. You can be prompted for your passphrase with each use of the key, or unlock the private key just once at the start of each X session thanks to ssh-agent(1). This should be offered automatically on your next login if you are using one of the default names for your keys. (If interested, you can see how the keys are loaded in at /etc/X11/xenodm/Xsession, and my next post includes a basic intro to use of ssh-agent/ssh-add on the command line).

A private key file's format is nothing more than the appropriate big long string of gibberish, with ominous BEGIN PRIVATE KEY and END PRIVATE KEY comments on either end. But with this implementation, what information am I actually handing out in my public key to all my troublemaker friends?

valefor:~$ cat .ssh/id_rsa.pub                                                     
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz0SXRq6tRtc+kXFeT6TTD4ryJpi+11Z3W53gbHgv
J2py34PMjABxA4CM5nQr+fxpncuKUYmkp85k68Cfwt9QzngQekCKzDVp9krCkSnuIhvKNWEDt1Lh7Fa
hK6C5nsGJ2YXfQef50Aq92uQR7R8izLMzWh+EBJi1n5rNmtWJRDOPFL+NIJ+8KER+Yh5ZiELhi0vdLS
GzGd8WbUO9arqmjvMQxrmmn46t5tHTbZqAGuquYGmGV+bq55cEBweaBTW5J+Fer4WPn36uUXfbhqerz
NMRD+BOXYsNz3GfVDLF8lMc6YmFJJEbheFNHtlJChEGQM2AR5EecVNShI8AT9 pamela@valefor

Clear as mud. Take the contents of your .pub file and edit ~/.ssh/authorized_keys on your destination to add them, or ask whoever is creating your account to do it for you. (Protip: use :r in vi, r in ed, C-x i in emacs, C-r in nano to just slap in the entire contents of a file. You laugh, but it took me over a year to notice I could do this. If that's also you, you're welcome!)

Your authorized_keys file expects each public key on a separate line (not required but please put a space between, yikes!) and ultimately takes the following form:

[options, or leave blank] key-type KEY user@keyorigin

The last field is just a simple generated comment. You can change it to anything which will remind you where the associated private key lives, or even add in a more traditional comment using # on its own line.

That's all I needed. Now I can ssh or scp to that remote machine and I'll be asked for the passphrase for my key if it's not already loaded into my agent, and then I'm in! All done!

ProxyJump

Oh no! I'm not actually in! This only takes me as far as the firewall at my house. The machines inside, including the one containing the folder of adorable baby hippo pics I'm trying to reach, are on a local subnet. I can't just type in the IP address I know... it's internal to the wrong network.

I'm extremely lazy. I definitely don't want to ssh into my firewall and ssh from there to the correct machine, yuck! I want a way to punch straight through, and ideally I'd get to set it up and never think about it again.

There are two straightforward ways to deal with this. (There are probably more. I know of two). You can set up port forwarding, where specifying a specific port at the end of your remote IP address will tell the firewall which machine you want to contact on the inside, like dialing an extension on the phone. This is within the realm of pf(4) and we provide some instructions in the FAQ.

Or you can use the ssh ProxyJump configuration directive. It's still port forwarding, but designed for this purpose specifically. At this point we're moving over to looking at information contained within ssh_config(5), but we actually only need a few lines of configuration to get the job done here. (Note that sshd_config(5) also exists, if you want to explore your options on the daemon end of things, too).

Our ultimate goal here is to be able to type "ssh destination" and reach the correct machine, without fussing over a login to the machine(s) in-between, and without exposing the session itself to that intermediary (merely using it for authentication).

We can do this on a single-use basis on the command line by adding our key to the authorized_keys file on the "jump host" machine in the middle as well as the destination, and then making a jump like so:

ssh -J jumphost destination

If your key is authorized on both machines, this should just work. You don't need a private key on the jump host. Generate your local keys, put the public key onto both remote machines' authorized_keys lists and you're there. If you'll need more than one jump, separate with commas.

Let's be real. As simple as that was, after a few times I'll still want a permanent shortcut. Sometimes I work within test VMs and other complicated networks, and I'd rather just type the name of the machine I want and go.

For this case, I will always want the same jump host, and just one jump, so I am first going to edit ~/.ssh/config to create an entry that tells OpenSSH what I want to do whenever I try to connect to the remote machine "antlion".

My bare minimum configuration here would be:

Host antlion
ProxyJump jumphost.example.com

I know antlion by its internal IP on a different subnet, so let's add that in as the HostName for ssh to use when talking to the jumphost.

Host antlion
HostName 192.168.1.99
ProxyJump jumphost.example.com

Now we're getting somewhere. When I type "ssh antlion" on this machine, it will connect to the jumphost machine (my firewall, an ocean away) and then onward to the correct machine on the internal network. With HostName I've given the jumphost an address I'm certain it knows how to reach. This prevents me from needing to fuss with anything on jumphost.example.com, such as adding an /etc/hosts entry, where I might not have root access.

Now I can type "ssh antlion" and connect to it, without thinking about what's in-between. Perfect. Fiona the Hippo awaits!

ThinkPad BIOS update on a stick!

2020-07-02T00:00:00+02:00

Hi, I'm pamela@! Peter offered me some space to write about OpenBSD and networking, and I come bearing some how-tos which hopefully assume less prior knowledge than most, and maybe later some thoughts about more esoteric development topics.

I currently maintain the OpenBSD changelog so I can talk shop, but working on mostly loaned equipment and not working in tech, I don't get enough practical experience. I have to keep notes about some odd things! But I was recently given a nice new laptop which should keep getting fresh updates for years to come. \o/ It has a brand new BIOS version, so let's slap that on here and write out how we did it.

Wrangling your BIOS Update ISO

This laptop is a ThinkPad x395, and I can go to the Lenovo support site and see there was a new BIOS version released on 10 Jun. We want the bootable CD image and can give it a quick check with sha256(1) just in case.

epaaj:~/Downloads$ sha256 r13uj44wd.iso
SHA256 (r13uj44wd.iso) = 35179a1c85e84e489a29cea995d1cd5ac98121647c0c52ad63444f278a645115

No optical drive on this model, so we will use the handy GetElTorito utility from ports to use this CD image on our USB stick. Just install it with pkg_add(1):

epaaj:~$ doas pkg_add geteltorito

The name refers to the "El Torito" bootable CD-ROM standard from the mid-1990s, which introduced a "Boot Record" volume descriptor and created a format for bootable CD-ROM discs containing one more more bootable images. GetElTorito, under the hood, is simply a perl script which extracts the first boot image from within an .iso. Since we can boot off any USB stick (in theory), we only need the data.

epaaj:~/Downloads$ geteltorito r13uj44wd.iso > x395_1.18.fs
Booting catalog starts at sector: 20 
Manufacturer of CD: NERO BURNING ROM VER 12
Image architecture: x86
Boot media type is: harddisk
El Torito image starts at sector 27 and has 104448 sector(s) of 512 Bytes
Image has been written to stdout ....

We now have the BIOS update file x395_1.18.fs ready to go.

Creating bootable media

I'm going to be using a USB stick to do this transfer, and since my USB sticks are generally also borrowed, that means I've quite literally just picked something random up off a table and should make sure there's nothing important on it.

Here's the update to the dmesg:

umass0 at uhub0 port 6 configuration 1 interface 0 "SanDisk Ultra USB 3.0" rev 3.00/1.00 addr 2
umass0: using SCSI over Bulk-Only
scsibus4 at umass0: 2 targets, initiator 0
sd2 at scsibus4 targ 1 lun 0: <SanDisk, Ultra USB 3.0, 1.00> removable serial.07815591150714116163
sd2: 117312MB, 512 bytes/sector, 240254976 sectors

Plenty of room, and now we know it's using sd2. Let's take a peek using disklabel(8)! That will show us the partition layout in use and tell us about how it's formatted.

epaaj:~$ doas disklabel sd2 
# /dev/rsd2c:
type: SCSI
disk: SCSI disk
label: Ultra USB 3.0
duid: 0000000000000000
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 255
sectors/cylinder: 16065
cylinders: 14955
total sectors: 240254976
boundstart: 0
boundend: 240254976
drivedata: 0

16 partitions:
      #              size           offset  fstype [fsize bsize   cpg]
      c:        240254976                0  unused

Oh. There's nothing at all! This one is either entirely unused or has been cleaned up thoroughly. The c: partition is just the disk in its entirety. If there were data to worry about on here, we'd expect to see one or more partitions in addition with filesystem types specified.

Here's an example that already has some assorted junk on it:

epaaj:/mnt$ disklabel sd2        
# /dev/rsd2c:
type: SCSI
disk: SCSI disk
label: Cruzer Dial
duid: 0000000000000000
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 255
sectors/cylinder: 16065
cylinders: 3818
total sectors: 61341696
boundstart: 0
boundend: 61341696
drivedata: 0

16 partitions:
        #       size    offset  fstype [fsize bsize   cpg]                       
        c:  61341696                0  unused                         
        i:  61341664               32   MSDOS

So this stick does have one existing MSDOS partition filling the stick. in this case I would want to mount that specific partition as follows:

epaaj:~$ doas mount -t msdos /dev/sd2i /mnt

Just to look through and make sure there's nothing important on there. If I don't care, though, we don't actually need to mount it and rummage around. We will be overwriting the entire stick to make a bootable version, and the original contents can simply be blown away.

We're going to use dd(1) to move our image onto the stick. Our input file is the image just created with geteltorito, the output file is the raw device corresponding to our USB stick, and optionally we can set a larger block size of a megabyte for speed.

Make sure you're using the correct device as output before you start! Double check the number, and then specify 'r' for raw (direct access to the device) and end with 'c' indicate we are working with the whole disk at once. My USB stick is still designated sd2, and I will be transferring the image to /dev/rsd2c.

epaaj:~/Downloads$ doas dd if=x395_1.18.fs of=/dev/rsd2c bs=1m
51+0 records in
51+0 records out
53477376 bytes transferred in 1.398 secs (38228412 bytes/sec)


epaaj:~$ doas disklabel sd2
# /dev/rsd2c:
type: SCSI
disk: SCSI disk
label: Ultra USB 3.0
duid: 0000000000000000
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 255
sectors/cylinder: 16065
cylinders: 14955
total sectors: 240254976
boundstart: 0
boundend: 240254976
drivedata: 0

16 partitions:
#                size           offset  fstype [fsize bsize   cpg]
  c:        240254976                0  unused                    
  i:           104416               32   MSDOS

Now we are ready to boot from this stick! For this machine, that involves pressing Enter and F12 to temporarily select the stick to boot from, and then updating the system. It takes maybe 5 minutes with one restart near the beginning, and then you're all set.

Cleanup

I'm not going to do it with this one, as it's a waste of a fairly large USB stick (which I do not own) but once you're done with your BIOS update, you can grab one of the installXX.fs images from your local mirror and turn your USB stick into a bootable recovery tool. Just pull the one for your architecture and use dd(1) exactly the same way, overwriting the raw device (/dev/rsd?c) as before. You can make yourself a pretty USB stick necklace and always be able to get a shell or fresh install. I once had an encrypted disk fail when I only had unreliable wireless internet and no other machines... now I really like making sure I have one of these to hand.

Historical: My first OpenBSD Hackathon

2017-06-21T00:00:00+02:00

This is a story about encouragement. Every time I use the word "I", you should think "I as in me, not I as in the author".

In 2003, I was invited to my first OpenBSD Hackathon. Way before I was into networking, I was porting software to my favourite OS. Specifically, I was porting games.

At that time, porters weren't really considered doing "real work". We weren't writing software from scratch. We couldn't say "this code is crap, let's do it correctly". We were taking software that someone else wrote, and massaging it to run on OpenBSD. If we were lucky, our patches would be accepted at the upstream project, so the next update would be easier. But still, the other porters were friendly and we worked together to solve important problems.

On the first night most of the hackathon attendees end up at the bar for food and beer, and I'm sitting next to Theo de Raadt, the founder of OpenBSD. At some point during the evening, he's telling me about all of these "crazy" ideas he has about randomizing libraries, and protections that can be done in ld.so. (ld.so is the part of the OS that loads the libraries your program needs. It's, uh, kinda important.) Theo is encouraging me to help implement some of these ideas! At some point I tell Theo "I'm just a porter, I don't know C."

Theo responds with "It isn't hard, I'll have Dale (Rahn) show you how ld.so works, and you can do it." I was hoping that all of this would be forgotten by the next day, but sure enough Dale comes by. "Hey, are you Peter? Theo wanted me to show you how ld.so works" Dale spends an hour or two showing me how it works, the code structure, and how to recover in case of failure.

At first I had lots of failures. Then more failures. And even more failures. Once, I broke my machine so badly I had to reinstall it. I learned a lot about how an OS works during this. But, I eventually started doing changes without it breaking. And some even did what I wanted! By the end of the hackathon I had came up with a useful patch, that was committed as part of a larger change.

I was a nobody. With some encouragement, enough liquid courage to override my imposter syndrome, and a few hours of mentoring, I'm now doing big projects. The next time you're sitting at a table with someone new to your field, ask yourself: how can you encourage them? You just might make the world better.

Thank you Dale. And thank you Theo.

ASNs and PI Space

2017-04-25T00:00:00+02:00

I do a lot of networking, and it makes sense for me to finally get my own personal allocation of IP addresses and an ASN. It looks like a confusing process, but it is pretty straightforward.

First, the cost. In the RIPE region there is no cost for ASNs, but there is a 50euro/year cost for each PI allocation. RIPE has no more PI IPv4 space available, but does have PI IPv6 space. You'll need to arrange to have a sponsoring LIR, and they may charge you for the service. You'll need a physical location to have a machine, and you'll need at least two connections, connecting to two different ASNs.

Secondly, you'll need a number of pieces of documentation. I made requests for an ASN and PI IPv6 space. The official rules are at RIPE's website, but this is the short list of what I sent in:

Copy of my ID or Passport
Signed End User contract with my Sponsoring LIR
Justification for having PI adresses (aka, what will I do with it)
Proof that I will be multi-homing the ASN, so Peering Contracts with at least 2 ASNs

Once I received the ASNs, I created the domain, as-set, aut-num, and route6 objects, so I exist network-wise. I also registered the traditional "as<num>.net" domain name, created the traditional email addresses, and PeeringDB entry. Time from first applying to receiving the assignments, 7 days.

ASN:                                 205969
IPv6:                    2001:67c:12f4::/48
PeeringDB: https://peeringdb.com/asn/205969

OpenBGPD Large Communities

2016-10-16T00:00:00+02:00

On Friday, I committed support for Large Communities to OpenBGPD. This is a draft-RFC that I am pretty excited about.

Back in the early days of The Internet, when routers rode dinosaurs to work and nerds weren't cool, we wanted to signal to our network neighbours certain information about routes. To be fair, we still do. But, back then everyone had 16 bit ASNs, so there was a simple concept called 'communities'. This was a 32bit opaque value, that was traditionally split into two 16bit values. Conveniently, we were able to encode an "us" and a "them", and perform actions based on what our neighbours told us.

But, 16bits is pretty limiting. There could only be ~65'000 possible networks on The Internet total? Eeek. So, we created 32bit ASNs. 4 billion networks is seen as a quite reasonable limitation. However, you can't really encode a 32bit "us" and a 32bit "them" value into 32bits of total space. Something called "Extended Communities" was invented, but it tries to solve everything except the case of a 32bit ASN signalling to another 32bit ASN.

Enter Large Communities. This is 3 32bit values. The first one is the "owner" of the namespace. Normally, you would put in your own ASN, or the ASN that you wish to signal. The second two 32bit values are opaque and only have meaning from the originating operator, but normally people will use "myasn":"verb":"noun" Or "myasn":"noun":"verb". Either way, it fits very nicely.

Having previously ran a 32bit ASN, it became quickly obvious the lack of suitable communities was a critical problem. It was even the way to request an "old style" 16bit ASN from RIPE, "I need to use communities". Even the ability to say "do this to that ASN" was ugly, since you couldn't really communicate who the community was supposed to matter to. Clearly, we The Internet Community screwed up by not addressing this need earlier.

OpenBGPD in OpenBSD -current has support for Large Communities, and this will be available in the 6.1 release and later. This was based partially on a patch from Job Snijders, thanks!

Presenting at BSDCan 2016

2016-02-26T00:00:00+01:00

This morning I recieved a nice email.

Subject: BSDCan 2016 - you are chosen

So, I'll be at BSDCan in Ottawa Canada this June. (You'll want to attend, there will be lots of other interesting talks, including one about an implementation called "switchd" for OpenBSD).

I'll be talking about an "Bidirectional Forwarding Detection (BFD) implementation and support in OpenBSD". This is some code that I'm working on, which will support the protocol in the kernel and daemons such as OpenBGPD.

I'm still in the process of writing the code (the kernel side is functional, still need to communicate with the other parts of the system), but this will be put into production well before the conference.

Come see me in Canada!

Conference submission below:

Bidirectional Forwarding Detection (BFD) implementation and support in OpenBSD. Or: A new protocol actually /did/ improve our routing.

Abstract: Bidirectional Forwarding Detection (BFD) is a protocol that allows detecting faults in links or routes. This is similar to GRE keep-alives, but is actually supported on real routers. Contrary to traditional link-state detection, BFD works on the next-hop IP address; so one can detect failures of some peers that do not affect the link state.

Full description: Internet links fail. This is a truism as old as Internet links. When a link fails, traffic gets dropped until the failure is detected and traffic can be re-routed. Detection of failures can be quite tricky however, since they are not always directly visible. Most systems use link state or a form of keep-alives for detection of failures. Link state detection does not help when there are active devices between a router and the other system, such as a switch or long distance links which use MPLS. The in-protocol BGP timers can also be quite long (a common default is 90 seconds) which is a lot of traffic when one are sending 10Gbps or even faster rates.

BFD is a new protocol that exists outside of existing routing protocols, but can communicate the status to all protocols. This allows for a single keep-alive to detect the health of a single link, without having to depend on a keep-alive in each and every protocol being used. As this is part of the "parent" interface, this does not introduce another layer in the network configuration. And since the link-state is only per next-hop IP, one can mix and match BFD and non-BFD neighbours on the same interface. This is extremely useful for routers connected to an Internet Exchange Point, which can have hundreds of peers spread over 10 or more physical locations.

A clever description of this is described in a draft RFC, which introduces automagic configuration of BFD between parties allowing for stronger resilience when there are many potential neighbouring networks without the overhead of manual configuration.

I will be discussing of the implementation of the BFD protocol for OpenBSD, problems discovered in both the protocol and network stack, use cases and production experience.

IPv4 vs IPv6 peering

2016-02-19T00:00:00+01:00

During a conversation on IRC, I started to think about our network peers that have IPv6, but don't send us any of those routes.

The disjoint in IPv4 peers vs IPv6 peers is extremely noticeable with BT (http://bgp.he.net/AS2856), but is common with almost all network entities.

BGP Peers Observed (v4): 200
BGP Peers Observed (v6): 16

As best I can tell, there are two basic reasons why I would receive IPv4 routes and not IPv6 routes from a peer.

First, is that peer does not announce any IPv6 routes to anyone. Maybe they don't have an allocation, maybe they have one but aren't using it. Who knows. Personally, I don't see much of a problem with that; even though that network really should get working on IPv6 for themselves.

The second reason is they only announce IPv6 to (at least) one transit provider, but not to their network peers. Maybe it's just to keep others from squatting on "unused space", maybe it's not really used, so they don't put any effort in. More commonly though, I believe it is from a (general) lack of effort put into IPv6. Looking at my own IXP connected routers, I see many ASes that are announcing IPv4 to the route server but not announcing their IPv6 networks to the route server.

Why is that? Did they get a great price on IPv6 traffic from their transit providers? Are they not as interested in traffic going over the Exchanges? Simply forgot? Is their backbone not prepared for IPv6 yet, and this is something they plan on fixing in the future?

While I don't think IPv6 is the end-all-be-all magical fix to all of our networking woes, I think the ship has sailed and we all need to get up to speed on IPv6. Network interconnections is one of the lowest levels of infrastructure, and needs to be done extremely early on in an IPv6 deployment to ensure high quality networking for users of that protocol.

RIPE71 Conference After-Action

2015-11-23T00:00:00+01:00

Last week was RIPE71 held in Bucharest, Romania. Since I was interested in a tutorial on Monday morning, and the conference officially ended Friday at 12:30pm, we arrived Sunday afternoon and left Friday afternoon. After checking in to the hotel, we visited the RIPE Atlas Probe Hackathon, which was happening over the weekend. Hung out with Florian@, who had made a port of the RIPE Atlas Tools, which I started testing. Fellow OpenBSD developer Paul Irofti lives in Bucharest, so we made sure to have some drinks with him at a pub near the hotel. The pub had fantastic hot chocolate (yum!).

Monday morning, I took the "NetDevOps - Ansible 101 to Network Nirvana" tutorial. I've previously written my own automation tool (update.sh, not publicly available), and have used Chef; but I was curious what they would show us. There are some neat tricks available in Ansible and in Python, so I should really play with these some more. In the afternoon the Opening Plenary and general talks started. IMHO, the most interesting one I saw was "BGP Security at Internet Exchanges: a practical experiment", which tested ingress filtering for peers on an IXP. Quite a few networks don't aggressively filter peers.

Monday night was the first of the official social events, at one of the hotel restaurants. Drinks and food were available for everyone, and we all got to meet each other.

Tuesday, we started again. Some very nice grouping of blocks today, with measurement as the first block (including the most dense understandable presentation I've ever seen "Spatial Representation of Broadband Networks in New Zealand"). The mid-morning session showed us how Facebook does automation, then how a smaller Datacenter does automation, then Leslie Carr showed us how to actually do it ourselves. Lots of fun (and useful!) gifs in the presentation.

I ended up skipping part of the afternoon sessions on Tuesday for some side conversations and need to catch up on the pdf's and videos of the ones I missed. I did however, agree to give a presentation about OpenBSD/OpenBGPd, for Thursday.

The Tuesday night social event was at a Nightclub, which we rented out for the entire evening. Very nice venue, good music (without being too loud!). The food was nice, but there were some queueing issues. Two plates of food, then lots of dessert. Then more food. Hmm. Would have been nice to re-arrange the order of those. Some excellent conversations, as normal.

Wednesday was a bit slower than normal. This was the first day of the Working Group sessions. I saw almost no presentations, as I kept getting involved in discussions about OpenBSD and OpenBGPd. During Wednesday, I also worked on my presentation, as well as setting up some BGP testing framework with Martin Winter (of Netdef).

Wednesday night we were on our own for social times. A group of us went to Caru cu Bere, which is a nice traditional Romanian restaurant. Extremely yummy food. A+, would eat again.

Thursday, bright and shiny at 9am we're at it again. Routing WG had a few interesting presentations, including a method to simply declare what kind of peering relationship you have. This would force the correct settings to be set on both sides of a peering, but would not be communicated beyond that. Interesting idea, but requires some major changes to the ecosystem, so I'm not sure how feasible this actually is....

Mid morning was the Open Source WG. GoBGP has quite a few interesting features, and the author is very nice and friendly. Spent some time talking with him afterwards, will possibly do some collaboration in the future. I gave my presentation "Current Status of OpenBSD/OpenBGPd", which went extremely well. Extremely favourable response from the room, as well as a bunch of comments and discussions with me afterwards; almost missed lunch ;).

Thursday afternoon was more discussions about OpenBGPd, and some things I'd like to see in BGP. Annoyingly, I have to take meds to fly, so I had to go to my room and take my meds and lie down for a while. After that, get ready for the fancy dinner on Thursday.

The RIPE 71 Dinner was held at the Palace of Parliament, which is the second largest parliament building in the world. Since it is an actual parliament building, there was airport-style security to enter, which caused airport-style lines to get in. Once in, however, a delightful dining room, with food and drink to match. After dinner, music and dancing. The dance floor was obscenely loud, but judging by how full it was, people liked it.

Friday morning, the last day. Geoff Huston gave a presentation "Today's Mobile Internet", which described what may be a serious driver of Networking in the next 5 years. And, Christian Scheele talked about concerns over the new FCC and EU directives over Wireless firmwares, "Forced Firmware Lockdown". A technical report of the conference, then the ending remarks from the RIPE NCC Chair.

Overall, a good conference. Interesting presentations, met lots of nice people, got to see some old friends, and made lots of good contacts. People are interested in the upcoming features of OpenBSD and especially OpenBGPd. I can't count the number of times I explained how funding for OpenBSD happens, and the best way for them to contribute money (the OpenBSD Foundation). I've also been convinced to submit presentations to UKNOF35, and RIPE72.

On the flight home, I got retweeted by Emo Philips, which I consider to be one of my career highs.

OpenBGPd and route filters

2015-11-06T00:00:00+01:00

Many moons ago, OpenBGPd was extensively used throughout the networking world as a Route Server. However, over the years, many have stopped using it and have migrated away to other implementations. Recently, I have been getting more involved with the networking community, so I decided to ask "why". Almost exclusively, they told me "filter performance".

Many IXP operators made a point to tell me that they liked the syntax and use of OpenBGPd. They were very concerned that there was apparently only a single piece of software that was now being used for Route Servers, and even some of them used multiple versions of it to "satisfy" their desire for divergent implementations. (insert 'eek' here)

So, I decided to look into why the filters were so slow.

Turns out, we were being very naive with the filters and checking every filter against every rule, even when we couldn't match. This is very common in the first implementation of a filter and even showed up in the initial version of PF. Obviously, filter performance in PF is rather critical to having a performant firewall so I looked at its implemetation. Turns out, it was rather simple and straight-forward.

PF's filter performance is known as "skip steps", which exploit the fact that some rules will never match. As an example, if you write a rule like "allow from AS 65111", this can never match if a peer has AS 65333. In such a case, we need to skip all of those rules.

There are two parts to skip-steps. First is when you parse and load the filters, you need to calculate where to skip to, if a group doesn't apply. This happens at load time. The second part is when we are iterating through the filters, is to detect when we get to a rule that can't apply, and skip to the next group. That happens while applying the rules.

Andy from LONAP (the 15th largest IX in the world, according to wikipedia), gave me the last configuration they used with OpenBGPd. This had 314 peers configured, and about 35,000 prefix filters. For additional torture, I added the IRR ruleset for a friend's network, which brought the total prefix rules up to around 600,000 prefixes filters. To this configuration, I sent 65,000 routes.

In OpenBSD 5.8, processing these routes would take approx 35 minutes on my laptop. With the change I committed today (Nov 6), the same processing takes only 30 seconds.

Stuart Henderson (a fellow OpenBSD developer, and one of the people that reviewd and OKed my change) adds a valuable comment in the discussion on Undeadly:

"Turns out, it was rather simple and straight-forward." - Henning pointed out a few times that it was only ever meant to be a quick first implementation, to be replaced with something better! Such a nice idea to use skip-steps here, with hindsight it's blindingly obvious, but that's often the way.

Since skip-steps come from PF, we can look in that direction for guidance about performance. An old Undeadly article about PF rule optimization talks about this ("Ordering rulesets to maximize skip steps") and the principles are the same here. You want to group the rules together to reduce the number of evaluations done.

Take these rules, for example:
allow quick from AS 112 prefix { 192.31.196.0/24, 192.175.48.0/24, 2001:4:112::/48, 2620:4f:8000::/48 }
allow quick to AS 112 prefix { $MY_PREFIXES }

allow quick from AS 2818 prefix { 212.58.224.0/19, 212.58.224.0/20, 212.58.240.0/20, [...] }
allow quick to AS 2818 prefix { $MY_PREFIXES }

allow quick from AS 8330 prefix { 199.212.92.0/23, 199.212.90.0/23,  199.80.128.0/17 [...] }
allow quick to AS 8330 prefix { $MY_PREFIXES }
And here are the skip-step groupings used in the code:
#define RDE_FILTER_SKIP_DIR            0
#define RDE_FILTER_SKIP_GROUPID        1
#define RDE_FILTER_SKIP_REMOTE_AS      2
#define RDE_FILTER_SKIP_PEERID         3
You can probably make a good guess at the meanings. So the rules above are not optimal - a simple reordering as below would put all of the "from" rules together and allow half of the lines to be skipped in one go. (Fewer than half of the rules, because the prefix lists are expanded to multiple rules, but hey there needs to be some room for borrowing something else from PF for future improvement ;-)
allow quick from AS 112 prefix { 192.31.196.0/24, 192.175.48.0/24, 2001:4:112::/48, 2620:4f:8000::/48 }
allow quick from AS 2818 prefix { 212.58.224.0/19, 212.58.224.0/20, 212.58.240.0/20, [...] }
allow quick from AS 8330 prefix { 199.212.92.0/23, 199.212.90.0/23, 199.80.128.0/17 [...] }

allow quick to AS 112 prefix { $MY_PREFIXES }
allow quick to AS 2818 prefix { $MY_PREFIXES }
allow quick to AS 8330 prefix { $MY_PREFIXES }
Nowadays PF has a ruleset optimizer that is aware of skip-steps and will reorder rules to take advantage of them automatically. (In fact, in some cases with PF it can still be better to disable the optimizer and do this by hand; you might know something about your traffic that pfctl doesn't). In the case of IXP route-server BGP filters, they're likely to be much larger than any PF ruleset you're ever going to see, so a ruleset optimizer would be relatively slow and take a lot of memory. Fortunately with BGP filters, they're already going to be machine-generated from routing registries or local databases, so it's a lot easier to order them correctly from the start.