Why client certificates?

RS wrote about Preventing drive-bys with client certs and although we'd discussed this method for some time, I hadn't deployed it yet. However, some recent log-spelunking had led me to determine that I liked the idea of a second layer of protection on some of my sites.

Just for clarification, this is being implemented as a first-layer of authentication and only will grant access to the basic functionality of the server. There will be further login requirements beyond that. However, as Rob notes in his post, there is significant advantage to keeping the wild internet away from the login screen (or anything else they might be able to exploit). So, as Rob says, this is drive-by prevention.

Previous to this exercise, I'd occasionally used nginx's built in authentication to require another user name and password before accessing the site, but this is a bit tedious due to the way that browser prompts and password systems (such as 1Password) work. Client-side certificates don't have this problem, at least on the Apple ecosystem, that authorization is tied to the device auth.

(TL;DR: put all predecessors of the client CA in the CRL)

Intermediate CAs

As RS's article is sufficient to get you going, why is this article necessary? Because I have a tendency to take opportunities like this to explore unnecessarily complex models, such that I can understand how the internals work and can employ them as needed in the future.

In this particular case, I have run an internal CA now for over a decade. This is used only on internal communications, although before Let's Encrypt, it was also used for securing web sites that weren't going to be accessed by people outside of the organization (and family and friends). With LE dropping the marginal cost of certificates to zero (and the fixed cost to the automation that RS and I have already created), the private CA hasn't been getting a lot of use.

A few years back, my original CA cert expired and it caused a notable amount of pain (this was still before LE was in general release), mostly because I needed to send out new certificates to each of my users and make sure they installed them on all of their various devices. In order to head this off for the future, I wanted a long-running CA certificate, something with an expiration beyond the date of my likely retirement, so something in the 2040's...

In the "real world" Root CAs have frequently had this kind of duration. For example the DigiCert High Assurance EV Root CA expires in 2031, and was originally issued in 2006; so, basically 25 years. However, they also have significant security on their Root CAs (for example, physical HSMs holding the keys) and do not issue normal certificates directly from the Root, but instead issue from intermediate CAs that have much shorter expirations. So, I figured I could approximate this using a secure USB storage device (tamper-proof and encrypted with a long PIN) for my Root, and by issuing certificates off of intermediates with much more limited lifespans.

For the few servers I've used this on, it has worked well. I trust the Root certificate on each of my devices and then the web servers send the intermediate (signed by the root) and the server certificate along with it.

Intermediate CAs and debugging client-side certificates

Since I already have the intermediate CAs (it turned out I'd created one for client certificates and another for server certificates originally), it seemed like an easy enough exercise to take RS's recipe and apply it to my client certs. I generated a new private key and a new client certificate for myself and then went about the configuration.

Simple enough, I took the intermediate CA and uploaded it to my server, along with the current Certificate Revocation List (CRL) and placed them into the ssl_client_certificate and ssl_crl stanzas respectively, making sure to turn ssl_verify_client on; as well.

That didn't work. No good error message, just an SSL error from both Safari and Chrome. I should note here that debugging client certificates requires quitting your browser frequently. If you don't there's often some piece of state that will either create false negatives or false positives. So, while testing, basically quit your browser and re-start it between every attempt. However, I've found no befit to restarting the machine.

I tried adding the Root CA to the Intermediate certificate in ssl_client_certificate, but this was unnecessary (as it should be if the client contains an authorized copy of the root, and the server is explicitly instructed to use the intermediate certificate as the base for authorization).

I'll note here that adding debug to the end of my nginx config's error_log line would have been helpful at this point. There were definitely errors occurring and only the server knew what they were.

Once I turned on debug, it was clear that the CRL verification was causing difficulty and so I validated that by commenting the ssl_crl line in the nginx config and restarting the server and my browser and things worked.

No CRL is probably a bad idea, so I did some looking around and it became clear that nginx wanted to check both the CRL of the Root and the Intermediate, so I concatenated the two CRL files and uploaded them to the server, and now the server is working fine.

In retrospect it made sense. If you're going to ask for client certificate verification and you're going to provide a CRL, you should provide CRLs all the way to the root.