Why Are Certificates Used for VPN? Best Practice Authentication

Previous posts in this series

1. Understanding Cryptographic Terms - Cryptography 101
2. All About Certificates - The Anatomy of a Certificate
3. Which Types of Certificates Exist and How are They Created?
4. What is the Subject Alternative Name and Why is it Important for VPN?
5. Why Are Certificates Used for VPN? Best Practice Authentication

So far in our deep dive series on certificates, we have discussed everything there is to know about certificates themselves, but three important questions remain: How exactly are certificates used with a VPN connection, how do they make VPN connections more secure and why is a correct Subject Alternative Name (SAN) value so important?

In this post, we explore the role certificates play in securing a VPN connection, as well as the steps you can take when choosing a VPN client to ensure your organisation is kept protected.

The Key Exchange

One common misconception is that a certificate-based connection uses private and public keys to directly encrypt data sent over the VPN. That is not the case. In fact, whether a VPN connection is certificate based, pre-shared key based, or neither does not change the way a VPN connection operates when it comes to data encryption. Not even a pre-shared key is ever used to directly encrypt VPN traffic.

Session keys

Instead, VPN traffic is encrypted with a key that is different every time you connect the VPN. This key is called a session key, as it is only valid for the current session. Oftentimes, keys are not even valid for the entire session, as there are time and traffic limits that force a new session key to be created during a single session if that session lasts long enough or handles enough traffic.

What is a key exchange?

A session key is negotiated using a key exchange (KE) method. One of the most well known KE methods is called the Diffie–Hellman (DH) key exchange. We won't dive into the details of how a DH key exchange actually works or how other key exchange methods work, but the following description more or less applies to all key exchanges:

1. Both sides know one or more shared values, or one side chooses them and sends them to the other side

2. Both sides each create a random secret value

3. Each side performs some calculations with its secret value and some of the shared values

4. The calculation results are exchanged between both sides, the secrets themselves are not

5. More calculations are performed using the exchanged results, the secrets, and possibly some of the shared values

6. The result of that last calculation is the session key

Both sides end up with exactly the same session key – despite the fact that each side used a random secret key in calculations that was never directly exchanged with the other side, only as part of a intermediate calculation result.

Note that the entire exchange can take place over an unencrypted, even a public data channel. If attackers capture all data being exchanged, they still won't be able to calculate the session key, as the final step (5) requires the knowledge of one of the two secrets from step (2) and those secrets were never exchanged, they are only known to the two endpoints.

Without knowing either secret, the session key cannot be calculated. And the calculations in step (3) are performed in such a way that it is virtually impossible to get back the secret from the calculation result.

Think of the following scenario:

Someone multiplied a value X with itself Y-times divided the result by Z (whole number division), only to keep the reminder R of that last calculation. Even if you know the value of X, the value of Z and the reminder R, there is no way you can tell what the value of Y was. That's because there is an infinite number of possible Y-values that all would have resulted in exactly the same reminder value. But only using the right Y-value will give you the correct session key in the end.

The Man in the Middle Attack

In theory a key exchange guarantees perfectly secure session keys, except if there is a Man in the Middle (MitM) attack.

How do MitM attacks work?

If the two sides communicating are Alice and Bob, the session key is only secure if it can be guaranteed that Alice was talking to Bob the entire time and Bob was only talking to Alice.

But what if there is another person in-between Alice and Bob? (Let's say Mallory.) When talking to Bob, Mallory pretends to be Alice and when talking to Alice, she pretends to be Bob.

In this case Alice would have negotiated one session key with Mallory and Bob would have negotiated another. Alice thinks she is sending data to Bob, but in fact she is sending it to Mallory, who decrypts that data with the session key Alice has calculated and then re-encrypts the data with the session key Bob has calculated.

Conclusion: As our example shows, the entire key exchange concept is useless, if you cannot ensure that you are exchanging data with the right person, i.e. the right client/server.

Cyber attackers (i.e. Mallory) can use MitM attacks to intercept data between an end user (i.e. Alice) and a VPN gateway (i.e. Bob)

Where authentication comes in

The exchanged data must be authenticated and that's where pre-shared keys (PSK) or certificates come into play. The task of a pre-shared key or certificates is not encryption, it is authentication.

If you remember the first part of the series, we discussed how private and public keys can be used to sign data ("Signatures as Proof of Indemnity"). By signing the exchanged data with the private keys of Alice and Bob, Mallory has no chance to perform a MitM-attack, because she does not have access to the private keys of either side.

If Mallory altered the exchanged data but kept the signatures in place, the signatures would no longer match. Mallory would have to also replace the signatures after altering the data but she cannot do that either.

PSK authentication

In case you were wondering how a pre-shared key would achieve authentication: The final session key, some previously exchanged random values (often called Nonces or Salts) and the pre-shared key are all fed into a keyed hash function and the resulting hash value is exchanged and then compared.

Mallory might be able to calculate the correct session key without being noticed, as the exchanged data is not signed in the pre-shared key scenario, but she won't be able to calculate that final hash as she does not know the pre-shared key.

Moreover, the way both sides feed the random values into that keyed hash function is slightly different for either side, so each side ends up with a different hash to show the other side. Each side knows how the other side has calculated the hash, so it can do the same and compare the results. As before, there is virtually no way to get back the input values from the calculated hash, so it won't help Mallory to know the exchanged hashes, she still cannot retrieve the pre-shared key from those values.

Proof of Identity

To verify the signatures of the signed key exchange, the other side needs access to the public key, which is embedded in a certificate. This certificate is simply exchanged as part of the key exchange – after all, a certificate is public and contains no private data whatsoever.

Using the public key from the certificate, the receiver can check if the signature is correct.

How to ensure a certificate is correct

Consider this: What would stop Mallory from manipulating the key exchange data, calculating a new signature using a private key only she has access to and then also replacing the certificate with one that contains the public key matching the private key she used?

Exactly, nothing!

That's why the endpoints also have to verify the certificate to be correct, otherwise the signatures are worthless. And for a certificate to be considered correct, two conditions must be met:

1. The certificate must be signed by a trusted entity that verified the information within the certificate is correct and whose signature ensures that no later manipulation of that information is possible.

2. It must be the correct certificate for the communication endpoint, as of course, when you are talking to the server vpn.example.com, the certificate you receive should be a certificate that authenticates example.com and not a certificate that authenticates www.example.org, as that would be completely different certificate for a completely different server.

Subject Alternative Name (SAN)

The second point is what the Subject Alternative Name (SAN) is for. This field tells the receiver of the certificate which server identities this certificate may authenticate.

If the server you are currently talking to is not listed in the SAN field, the certificate does not provide any proof of identity for that server. It may as well be a certificate that Mallory stole from some other server, together with the private key, and which she is now using trying to spoof your VPN server, so she can get access to login data of one of your VPN users.

A good VPN client will always verify the SAN field and warn the user if the SAN field values don't match the current communication endpoint.

This is the same thing every browser does when you open a website with HTTPS: If the SAN values don't match the current server address, the browser will warn you and if you decide to ignore this warning, you are continuing at your own risk, because then your key exchange is not fully secured against all possible attacks.

Clients who don't validate the SAN field are not just less secure, they are in violation of standard X.509v3 (RFC 5280). If a web server has been set to strict cryptographic validation, the browser won't even allow you to manually override this issue.

In that case the SAN field MUST match the server address, otherwise the connection is immediately terminated.

VPN Tracker puts your security first

Instead of performing certificate checks itself, which is always error prone, as you may forget to verify certain conditions or have misunderstood the standard, VPN Tracker relies on the built in certificate infrastructure that Apple provides in all their operating systems.

This infrastructure strictly follows all required standards and thousands of security experts worldwide keep a close eye on what Apple is doing to immediately point out any standard violations or otherwise incorrect behavior.

By using Apple's infrastructure, VPN users also automatically profit from any security fixes Apple might be releasing in the future without having to update VPN Tracker itself.

Since Apple's infrastructure is complicated to use, many clients take shortcuts and implement their own verification. However, as expected, these checks are often incomplete. They are sometimes based on older standards (X.509 v2 instead of v3) and also tend to ignore more recent security advisories, which request enhances to the verification behavior even in-between standard version updates, e.g. if a new problem has been discovered with the current standard or a new attack has been found.

In particular, the SAN fields are very often ignored by VPN clients, as a lot of setups use incorrect certificates that do not contain correct SAN values. This means verifying the SAN field would trigger a warning on first connect but by not showing this warning, those clients trade security for a tiny amount of comfort – after all, the warning is only shown once and can then be ignored for the future. If the waring shows up again later, the certificate has changed and this is always something that should alarm the user.

The best way to avoid those warnings is to generate certificates with correct SAN entries. This actually isn't too hard, as our last certificate blog post demonstrates. If the SAN field is correct, there won't be a warning and if a warning appears in the future, this almost always means something is wrong, signalling the user to contact their administrators before deciding to override this warning on their own. For example, maybe the server address was changed and someone forgot to update the certificate, or maybe the certificate was replaced and the new one is missing correct SAN entries.

Always keep in mind: A single unprotected key exchange can be all that it takes for an attacker to break into your VPN setup. Once this has happened, the attacker may be able to install Trojans or Ransomware and the resulting damage can be severe and extremely expensive for your business.

Why VPN Tracker?

VPN Tracker is the best secure remote access solution for Mac, iPhone and iPad and is compatible with the most popular VPN gateways, including UniFi, NETGEAR, TP Link, Draytek, and many more.