Hiawatha & Public Key Pinning (HPKP)

In light of the recent spate of certificate authority controversies, the next entry in my series of Hiawatha tutorials will focus on one particular countermeasure: HTTP Public Key Pinning, or HPKP for short. In a nutshell, HPKP helps to avoid the scenario where an antagonist issues an SSL certificate for your domain which is signed by a rogue CA. Normally the client browser will implicitly trust any cert which is signed by a valid certificate authority, even if that CA happens to be, say the Hong Kong Post Office or the China Certification Authority, for example.

To avoid this, HPKP tells a capable client via an HTTP header that, for your particular domain, it should trust only certs signed by a particular set of keys. After the first time they’ve visited your site, even if an adversary issues a certificate for your domain which isn’t signed by one of the keys in your HPKP list, the browser will behave as if the certificate is untrusted and issue the appropriate warning to the user. While imperfect, this method can at least present a fairly effective hurdle to would-be attackers looking to harass your users.

Example scenarios

On a good day, everything works as anticipated.

CA Exploit Example (Normal)

The client makes a request and is able to reach the expected website. Yes, I know this diagram is oversimplified and not 100% technically accurate. It’s an illustration. Shut up.

If a bad actor is able to issue certs issued by a compromised or complicit CA however, they are capable of spoofing your site very convincingly. The user will even see the little lock icon in their browser, reinforcing the perception that everything’s just fine.
CA Exploit Example (Compromised)

If an attacker is able to succeed in tampering with DNS and providing an apparently valid certificate, the victim may never even realize they’re interacting with a malicious website.

HPKP can throw a wrench in the works for an adversary by supplementing the normal chain of trust. If the attacker’s certs weren’t signed by one of your pinned keys, the intended victim’s browser refuses to connect and will display a prominent error message instead.
CA Exploit Example (+HPKP)

With HPKP in play, as long as the client has connected to the real server at least once, their browser will reject the attacker’s certificate and warn the user. Not ideal perhaps, but it definitely beats the alternative.


It’s important to understand that HTTP public key pinning is not without inherent risks and drawbacks, so it should be approached with an appropriate level of prudence and foresight. First and foremost, HPKP works by essentially providing an alternative chain of trust for your website which acts to compliment and even supersede the normal trust process. Therefore, the same mechanism that can foil a would-be attacker will also work against you should you ever lose or mishandle your keys. If a user’s browser has ever connected to your website with HPKP in place, you can’t simply reissue your certificates against a different key without running afoul of the same mechanism. As such it’s extremely critical that, if you wish to utilize HPKP, you take serious precautions to avoid this scenario.

The information presented in this article is provided “as is” without any representations or warranties, express or implied. Should you choose to follow any of the advice provided, you understand that you do so entirely at your own risk; I accept no responsibility nor liability for damages of any kind which may result from following said information.

TL;DR: An incorrectly implemented HPKP setup could lead to fire and brimstone coming down from the skies, rivers and seas boiling, forty years of darkness, earthquakes, volcanoes, the dead rising from the grave, human sacrifice, dogs and cats living together, and/or mass hysteria. Please proceed with due caution.

Planning and precautions

In order to adequately implement HPKP in your own environment, you must make certain preparations. These include producing and pinning more than one key, and providing appropriate safeguards to ensure their integrity and availability in case something goes sideways. Depending on the importance of (and dollar value attached to) a particular website, what might be considered reasonable provisions could range from the simplistic to the extravagant. I’ll explain…

In the draft specification itself, it’s suggested that at least two certs be pinned: the active cert, and a backup cert. I’d recommend taking this a step further, especially if the site is mission critical, and making a 3rd key to place in escrow. If you’re dealing with a simple personal or hobby site, this might just mean an encrypted backup on your fileserver, or printing it out in a monospace font and sticking it in a fire safe with your other important documents. If it’s for something more grandiose, you might park it with a 3rd party security firm or perhaps store it in an encrypted USB key in a safe deposit box. Yeah, this is important stuff. Once you start using public key pinning, you can’t really roll it back—especially if your website uses full time TLS +HSTS.

With this all in mind, here’s what I believe to be a sensible approach to preparing for HPKP:

  1. Create three keys, and perhaps also corresponding certificate signing requests (CSRs), from a secure host.
  2. The first key will be used to acquire the production TLS certificate, and placed on your live webserver(s).
  3. Store the second key in an encrypted database within arm’s reach, e.g. in a KeePassX database on a system with redundant storage (or similar—just make sure it’s safe and you can retrieve it when you need it).
  4. The third key will be your failsafe key; “Break glass in case of Murphy’s Law” territory. It should be stored separately from the other keys, and if possible on a different media type.
  5. If this is part of your business process, also include instructions for key retrieval in your relevant DR and/or BC documents.

Creating your keys and CSRs

First, you’ll need to generate multiple keys. Again, I recommend three for additional peace of mind, though you should have at least two. If you’ve already got one in production at the time you wish to implement HPKP, simply consider that one your first cert instead of generating it.

I prefer to generate the keys and CSRs at the same time, so that they’ll be ready and waiting if you need them. For instance, if your CA was compromised and you need to quickly issue a new certificate against a new CA. You’ll be like a Scout: always prepared.

Edit: The content below describes the process of manually generating the multiple CSRs and corresponding fingerprints in preparation for deploying HPKP. In the time since writing this article, I’ve written a script which will automatically generate all necessary keys, CSRs and fingerprints for you. If that sounds more appealing, see my other article: A script to simplify HPKP for details. Otherwise, please read on.

For this step you’ll need a secure host computer, hopefully running something UNIX-y, with a relatively up-to-date version of OpenSSL. Since you’re making multiple key/CSR combos at once, you can also save a lot of time by a) making sure that the host you’re creating them on has plenty of available entropy, and b) you can include all of the necessary information in-line. For instance, you could do something like the following (change the parts in bold to fit your own circumstances):

mkdir csr && chmod 0700 csr && cd csr && \
openssl req -out example_tld.csr -new -sha512 -newkey rsa:4096 -nodes -keyout example_tld.key -subj "/C=US/ST=California/L=San Francisco/O=MyCompany, Inc./OU=Network Operations/CN=example.tld" && \
openssl req -out example_tld.standby1.csr -new -sha512 -newkey rsa:4096 -nodes -keyout example_tld.standby1.key -subj "/C=US/ST=California/L=San Francisco/O=MyCompany, Inc./OU=Network Operations/CN=example.tld" && \
openssl req -out example_tld.standby2.csr -new -sha512 -newkey rsa:4096 -nodes -keyout example_tld.standby2.key -subj "/C=US/ST=California/L=San Francisco/O=MyCompany, Inc./OU=Network Operations/CN=example.tld" && \
for key in `ls *.key`; do openssl rsa -in $key -outform der -pubout | openssl dgst -sha256 -binary | base64 > $key.fingerprint ; done

Here’s the breakdown of what each part does:

  • mkdir csr && chmod 0700 csr && cd csr – create a directory (‘csr‘ in this example), make sure only the creating user can traverse or manipulate it (‘0700‘), and enter said directory (‘cd csr‘).
  • openssl req -out example_tld.csr -new -sha512 -newkey rsa:4096 -nodes -keyout example_tld.key – generate a new CSR (‘example_tld.csr‘) and key (‘example_tld.key‘) pair, using SHA512 hashing (‘-sha512‘), the RSA key size being 4k (‘rsa:4096‘).
  • -subj ".." – Rather than answering questions about the certificate request interactively, feed the information from the command line…
    • /C= | Two-character country code, e.g. /C=US
    • /ST= | State, e.g. /ST=California
    • /L= | Location / city, e.g. /L=San Francisco
    • /O= | Organization, e.g. /O=My Company, Inc.
    • /OU= | Organizational unit / department, e.g. /OU=Network Operations
    • /CN= | Common name / hostname, e.g. /CN=example.tld
  • for key in `ls *.key`; do openssl rsa -in $key -outform der -pubout | openssl dgst -sha256 -binary | base64 > $key.fingerprint ; done – Iterate through each key we’ve created and generate a fingerprint for each one.

You should end up with three sets of CSRs, keys, and fingerprints:

$ ls -lah
total 36K
drwx------ 2 chris chris  220 Apr 11 01:31 .
drwx------ 3 chris chris  180 Apr 11 01:31 ..
-rw-r--r-- 1 chris chris 1.7K Apr 11 01:31 example_tld.csr
-rw-r--r-- 1 chris chris 3.2K Apr 11 01:31 example_tld.key
-rw-r--r-- 1 chris chris   45 Apr 11 01:31 example_tld.key.fingerprint
-rw-r--r-- 1 chris chris 1.7K Apr 11 01:31 example_tld.standby1.csr
-rw-r--r-- 1 chris chris 3.2K Apr 11 01:31 example_tld.standby1.key
-rw-r--r-- 1 chris chris   45 Apr 11 01:31 example_tld.standby1.key.fingerprint
-rw-r--r-- 1 chris chris 1.7K Apr 11 01:31 example_tld.standby2.csr
-rw-r--r-- 1 chris chris 3.2K Apr 11 01:31 example_tld.standby2.key
-rw-r--r-- 1 chris chris   45 Apr 11 01:31 example_tld.standby2.key.fingerprint

Installing your host certificate

After you’ve provided your CSR to a certificate authority and gone through the usual certificate issuance process, you’ll wind up with a signed host cert for your own domain, most likely a handful of intermediate certificates, and possibly also a root certificate. You won’t need the root cert, so ignore it. Concatenate the rest in the following order:

  1. Your host cert
  2. Intermediate cert(s), if any, in descending order towards root
  3. Your key, created along with the CSR

For example, if you’d purchased a PositiveSSL cert:

cat example_tld.crt COMODORSADomainValidationSecureServerCA.crt COMODORSAAddTrustCA.crt > example_tld.bundle

Next, make a directory where you’ll store your certs. I typically use something like ‘/etc/ssl/localcerts/‘, depending on the conventions of the underlying OS:

sudo mkdir -p /etc/ssl/localcerts/

Change the permissions so that only root is allowed to traverse or write into it:

sudo chown root:root /etc/ssl/localcerts/
sudo chmod 0640 /etc/ssl/localcerts/

Prep a blank file for your cert bundle with the correct permissions already in place:

sudo touch /etc/ssl/localcerts/example_tld.pem
sudo chmod 640 /etc/ssl/localcerts/example_tld.pem

Then populate it with your certificate chain and the matching private key from the CSR creation process:

sudo cat example_tld.bundle example_tld.key > /etc/ssl/localcerts/example_tld.pem

Configuring Hiawatha

With all of the other considerations handled, all that’s left is to configure Hiawatha. This involves only two aspects: making sure TLS is working properly, and sending the appropriate headers. For the first item, you need only set the appropriate cert bundle from either an HTTPS binding, or within a vhost stanza (if you will be serving multiple HTTPS domains from the same IP address via SNI).

Host-level HTTPS

Here’s an example of a ‘binding‘ stanza within Hiawatha (change the parts in bold to meet your needs):

Binding {
        Port = 443
        SSLcertFile = /etc/ssl/localcerts/example_tld.pem
        MaxKeepAlive = 128
        TimeForRequest = 5,15

Domain-level HTTPS (vhosts)

And as applied to a vhost context, where this will be a secondary vhost on a system hosting multiple secure domains (again, change the bold parts as necessary):

VirtualHost {
        Hostname example.tld, www.example.tld
        WebsiteRoot = /srv/www/vhosts/example_tld
        SSLcertFile = /etc/ssl/localcerts/example_tld.pem
        EnforceFirstHostname = yes

Setting HPKP headers

Now all you need to do is set the keys you’d like to pin as a custom header within your vhost. If you followed my previous directions regarding creating your keys and CSRs, you’ll also have a number of corresponding ‘.fingerprint’ files:

csr$ cat *.fingerprint

Place the contents of these fingerprint files within a CustomHeader entry in your virtual host like so:

VirtualHost {
        Hostname example.tld, www.example.tld
        WebsiteRoot = /srv/www/vhosts/example_tld
        CustomHeader = Public-Key-Pins: pin-sha256="MaSqJjZZOBp0ExzbEXeQLLo55VkMFV5NdcCLp33Iaeo="; pin-sha256="4EEqHjRU2k7Dw+uL8NEOG9YRrTkacl/ZdjtFxF+OJSo="; pin-sha256="P6IFDksuunk6GF3iRYyVBhJ4yMkv5vIoFhJlpCHpCLk="; max-age=15768000; includeSubDomains

Let’s deconstruct the header and examine what each component is doing:

  • Public-Key-Pins: – Tell an HPKP capable browser to pin the following…
  • pin-sha256= – SHA256 fingerprint of the key to be pinned, one entry for each, surrounded by double-quotes.
  • max-age= – How long the browser should store and honor these HPKP values before they expire, in seconds.
  • includeSubDomains – Whether or not the pinning should apply to the entire domain, e.g. ftp.example.tld, blog.example.tld, mail.example.tld, etc. or just to the current domain. If you choose to use the ‘includeSubDomains‘ option on a root domain like ‘example.tld’, do so with care, as it can obviously have wide repercussions. On the other hand, this option is much more benign when applied only to a subdomain like ‘blog.example.tld’, where only that domain and its own subdomains (like ‘static.blog.example.tld’ or ‘cdn.blog.example.tld’) would be affected.

Once you’ve finished making your changes, all you need to do is put them into effect:

sudo /etc/init.d/hiawatha restart

Wrap up

Assuming you’ve done everything correctly, your site should function as usual, with the notable difference that it’s now much more resistant to sophisticated man-in-the-middle attacks. You can verify the result by simply examining the headers:

$ curl --head https://example.tld/
HTTP/1.1 200 OK
Date: Sat, 11 Apr 2015 22:36:55 GMT
Server: Hiawatha
Connection: keep-alive
X-Frame-Options: deny
Public-Key-Pins: Public-Key-Pins: pin-sha256="MaSqJjZZOBp0ExzbEXeQLLo55VkMFV5NdcCLp33Iaeo="; pin-sha256="4EEqHjRU2k7Dw+uL8NEOG9YRrTkacl/ZdjtFxF+OJSo="; pin-sha256="P6IFDksuunk6GF3iRYyVBhJ4yMkv5vIoFhJlpCHpCLk="; max-age=15768000; includeSubDomains
X-Random: 222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222

That’s it, you’re done. Your users are better off, and you’ve done your part to make the Internet a slightly safer place. Feel free to leave any questions or suggestions in the comments.

Leave a Reply

Your email address will not be published.