PKI certificate management

As I said, I have a very rough understanding, but a lot of gaps.

In this post, I will try to explain:

• The different file formats that can be used to store certificates.
• How the files are structured, and how they differ from one another.
• How to generate new certificates, and inspect existing ones, using the openssl command line tool.
• What a certificate chain is, and how to inspect it.
• The difference between a TLS certificate and an SSH certificate.

Hopefully by the end of the post, you'll have a clearer idea of what certificates are, and how to interact with them.

Before we start exploring the different ways PKI certificates can be generated, stored, verified, and used, let's just take a step back and start with an introduction to PKI certificates in general.

Public Key Infrastructure certificates are most commonly used for securing TCP and HTTP communication via TLS. They are primarily used to:

• Encrypt end-to-end communication.
• Establish trust between a client and a server, so that the server's identity can be verified.

The certificates themselves normally contain metadata about the owner (such as name, location, etc.), as well as a public key used for encryption. The certificate is accompanied by a private key, which makes decryption possible.

Normally, a PKI certificate manifests as a pair of files on disk. One containing the certificate, and another containing the private key.

Most TLS certificates are in fact X.509 certificates. X.509 is a standard for certificate structure which defines which fields are included in the certificate. X.509 certificates can be stored in a variety of different file formats, which is the main cause of my confusion about which file types are used to store certificates.

The certificate itself is comprised of three parts:

1. Information about the certificate, such as the issuer and the distinguished name the certificate is for.
2. The public key, used for encrypting data.
3. The private key, used for decrypting data.

Normally, when a certificate is generated, its information and public key are stored in one file (normally just referred to as the certificate), and the private key is stored in another file.

X.509 certificates are typically stored in base64 encoded ASCII files which use the *.pem, *.crt and *.cer file extensions for the public key portion interchangeably. The private key is typically stored in a file with the *.key file extension. Whenever you see one of these files, you're looking at a base64 encoded X.509 certificate, irrespective of what the file extension might be.

Before we dive into generating new certificates in the next section, it's worth briefly mentioning what a certificate request is. Certificate requests require a basic understanding of certificate authorities.

A certificate authority is a reputable company who digitally signs certificates to indicate that they are from a trustworthy source. When a certificate is used for authentication and encryption, its authenticity can be verified by checking that the certificate's signature was generated by the original certificate authority. This verification process is discussed in more detail later in this post.

Normally, when a new certificate is generated, it is signed by a certificate authority. When generating certificates in this way, the artifacts of the certificate generation process are files which actually represent a certificate request, and not a certificate.

The certificate request is sent to the certificate authority, who returns a signed certificate which is ready to use. When experimenting with certificate generation locally, this certificate request and signing step can be skipped, and a certificate can be generated directly with no signing. This is known as a self-signed certificate, which is perfectly usable, but would fail certificate verification checks as no trusted authority has signed it. More on this later.

New X.509 certificates can be generated using the openssl command line tool. Parameters for certificate generation can be provided via command line arguments, interactive responses, or via a config file.

For this example, we will start with the following config file:

The distinguished name in this configuration identifies the owner of the certificate. As well as containing things like organisation name and location, the distinguished name also includes the Common Name. This is the hostname at which the certificate will be used, and forms an important part of certificate verification. In this case, the certificate we're generating could only be used to encrypt traffic on localhost.

Then, we can use openssl to generate a new X.509 certificate with the following:

Let's just examine each of the command line arguments:

• req: this command creates and processes certificate requests.
• -x509: generate an X.509 certificate that is self-signed, as opposed to a certificate request that would need to be signed by a certificate authority.
• -nodes: do not encrypt the private key.
• -newkey rsa:4096: indicates that a new certificate request and private key should be generated, and that the RSA algorithm should be used with a key length of 4096 bits.
• -keyout private.key: the file to write the private key to.
• -out certificate.pem: the file to write the certificate and public key to.
• -config openssl.conf: the config file to use for certificate parameters.

The result of this command is two files: a private key, and a certificate file containing a public key.

The certificate looks like this:

The private key looks like this:

Now that we've successfully generated a certificate and private key, we can inspect these files as follows:

Here you can see the issuer, validity, and algorithm of the public key. Similarly, the private key can be inspected with:

Checking the private key yields less useful information (for an operator, at least), but you can also check the consistency of the key using:

In order to verify the authenticity of certificates, it's necessary to inspect the issuer (or certificate authority) of a certificate. In the example above, the distinguished name of the issuer is C=UK, L=London, O=Form3, CN=localhost. This distinguished name identifies the certificate authority, who must be trusted in order for the authenticity of a certificate to be verified.

The certificate authority has a root certificate, which is used to generate a signature for each certificate that it issues. Combining the signature of a certificate with the public key of the issuer's root certificate allows a signature to be verified. Sometimes this is as simple as combining the signature/key of an issued certificate and a root certificate, and sometimes there are intermediate certificates between the issued certificate you're verifying and the ultimate root certificate. Intermediate certificates are typically also issued by the certificate authority, but with a shorter expiry time. This makes them less vulnerable to compromise, and easy to rotate and revoke.

On a Linux operating system, a list of root certificates can be found in /etc/ssl/certs. Each certificate authority is represented by its own root certificate, which can be inspected in the same way we inspected the certificate we generated earlier.

For example, inspecting the GoDaddy root certificate looks something like this:

We can tell this is a root certificate, because the certificate was used to sign itself. We can verify this self-signed signature as follows:

Similarly, if we try to verify the issuer signature for the certificate we generated, we get a verification error:

This error message indicates that the certificate we generated was "self-signed". The GoDaddy root certificate is also self-signed, however this is characteristic of a root certificate. The root certificate is trusted, because it was installed by the operating system. Our certificate is less trustworthy, because we just generated it on the fly.

TLS certificates in use on the Internet are signed by one of the root certificates installed on your computer, which allows their authenticity to be verified.

This can be demonstrated by inspecting a certificate of a website protected by TLS, and then verifying it with its root/issuing certificate.

The TLS certificate of a website can be inspected by using openssl s_client, which normally expects input from stdin (hence the echo -n |):

Note that the Common Name of this certificate was google.com, indicating that this certificate can only be used to encrypt traffic to this domain name. Even if the certificate is verified as authentic, it cannot be used to serve TLS traffic from any other domain in a trusted capacity.

A certificate file can be generated for verification with:

If you inspect this certificate (openssl x509 -in google.pem -text), you will see that this certificate was issued by "Google Trust Services". A quick search of your root certificate directory (cat /etc/ssl/certs | grep google) will show that Google Trust Services is not a root certificate authority. This means that there is a chain of issuing certificates between the one in use at google.com and the certificate authority that signed the first certificate in the chain.

We can download the certificate chain separately using openssl as follows:

The original certificate can then be manually verified against its root certificate via the certificate chain with:

The verification of Google's certificate against the root certificate installed in your operating system demonstrates the difference in trust between the public certificates in use on the Internet for TLS, and the self-signed certificates you might generate locally. The fact that the certificates in use online can be verified against known certificates on your computer demonstrates that they have been issued to a trustworthy server.

According to man ssh-keygen, the certificates used for SSH are a different, and much more simple, format than X.509 certificates used for TLS. SSH certificates are used in a similar way to the X.509 certificates used in TLS: they consist of public and private keys, but the format is different to the certificates described in this post.

When you connect to a server that offers TLS, the server will be configured to send you its public certificate and will encrypt data with its private key. I'm not going to delve into the details of how TLS works here, but instead demonstrate how you might configure a simple web server with the materials it needs to make TLS possible. I'll be using nginx running in a Docker container to provide a small example.

First of all, we'll need some static content to serve as our web page:

Next, we'll need to configure an nginx server to serve this web page:

Then, we can package nginx with our web page and config as follows:

Now, we can build and run this image with:

OK, now we can test our server by making a web request:

Now that we've got a functioning web server, we can try to add TLS to it. The first thing to do is to update the nginx configuration with TLS details:

Now that we've got a functioning web server, we can try to add TLS to it. The first thing to do is to update the nginx configuration with TLS details:

This configuration uses the certificates we generated earlier. To re-cap, we generated these files using openssl:

The certificate files will also need to be present in the Dockerfile:

We can then build and run the container in a similar way to before:

Now, if we try to get the webpage via HTTP, it fails:

And, if we try to get the webpage via HTTPS, it also fails:

This is because we're using a self-signed certificate. Web browsers, curl, and most HTTP clients will fail if you try to make requests over TLS and the server presents a self-signed certificate. In this case, there's no way to know if you can trust the server or not. However, if we try again and tell curl to ignore self-signed certificates (-k), we are successful:

If you were working with self-signed certificates regularly, or with a private certificate authority whose root certificates aren't installed automatically, it is possible to import custom root certificates into your system's list of trusted certificates. Doing so would mean that clients like curl would recognise the authenticity of your certificate, rather than failing to verify its signature.

So there you have it! Most TLS certificates are X.509 certificates, and whether you see them in *.pem, *.crt, or *.key files they're all likely to be in the same format. Certificates can be generated, inspected, and verified using the openssl command, and this applies to both self-signed certificates you might generate for testing, as well as certificates signed by a trusted certificate authority. Using X.509 certificates for TLS on web servers is relatively straightforward, and easy to configure in server applications like nginx.

I hope you've found this post useful, and walk away from it slightly less confounded than I was when I started writing it!