OpenVPN: The protocol that can do everything, but nothing really well

When you mention SSL, the first protocol that comes to people's mind is OpenVPN. Which is rather funny, considering the fact that OpenVPN is no SSL VPN protocol. SSL VPN protocols are protocols like SSTP, SonicWALL SSL or Cisco AnyConnect. A SSL VPN protocol is a protocol that uses TLS (aka SSL) to establish a secure connection and then tunnels network traffic over that TLS connection. OpenVPN is never doing that. OpenVPN can use TLS as part of the session key creation but it never uses TLS to transport any network traffic.

OpenVPN in fact defines an own VPN connection management and traffic tunneling protocol, that is completely independent of TLS. Just like other VPN protocols, it uses a random session key each time to encrypt and authenticate tunneled network packages. The OpenVPN protocol can run over either TCP or UDP, and as always, UDP is recommended.

Session Key Negotiation

To avoid Man-in-the-Middle (MitM) attacks, OpenVPN can use one of three authentication schemes:

1. Static Key

A secret, static key has to be known by client and server and is used as part of the session key calculation. This mode is pretty much like IKE/IKEv2 with a pre-shared key (PSK). In this mode, TLS is not used at all. In this mode the session key is just calculated out of two random values (one chosen by client, one chosen by server) and the secret static key data. Note that this mode has been deprecated in OpenVPN 2.6 but it's still supported by all clients and servers.

2. Mutual Certificates

In this mode a TLS connection is negotiated on top of the OpenVPN connection protocol. Both, the server and the client require a certificate and must prove each other's identity. Once the TLS connection is up, each side just generates a session key and sends it to the other side via that protected TLS connection which is by itself encrypted and ensures data integrity. Other than for the key exchange, the TLS connection is also used for automatic configuration (choosing a better cipher, assigning the user an IP address and remote network routes) and user authentication (username + password), both optional. Once the protocol negotiations are done, the TLS connection is not used at all anymore and further protocol management and traffic encapsulation is performed directly by the OpenVPN protocol, as in case (1).

3. Server Certificate + Client Login

This mode works just like mode (2) but the client does not require a certificate. Only the server is authenticated towards the client, just like in case of a web browser HTTPS connection, therefor the client must provide client authentication later on by sending username and password to the server.

Config Overkill

The creators of OpenVPN certainly had no experience in developing VPN protocols, or network protocols in general, as they added pretty much every feature you can even think of to this protocol, so the protocol ends up having more configuration options than any other widely used VPN protocol, and that's not really a good thing, as there are many ways to ensure that clients and servers don't interact correctly due to configuration errors or your connection won't work well or isn't as secure as you think it is. Many of these options are very exotic or barely every needed to begin with.

Also, when it comes to security, less is usually better and simpler is often saver than complex. This makes OpenVPN the exact opposite of WireGuard, which went the opposite direction and tried to keep config to a minimum and ensures maximum compatibility between clients and servers, while at the same time keeping the protocol as simple as possible, to ensure it's easy to implement and offers little attacking surface.

OpenVPN != OpenVPN

While other VPN protocols stay the same for decades and may only get a few optional extensions when absolutely needed, OpenVPN constantly adds new options, deprecates old options and keeps changing the core functionality of the protocol, constantly breaking backward compatibility or even worse, adding clunky options to somehow retain backward compatibility. New features are often not very well thought out and then have to be revised several times before they are even useful, creating plenty of bad debts in the code and causing troubleshooting to become even harder.

It was actually foreseeable that many of these innovations would be needed at some point before the first version of OpenVPN has even been released, which of course raises the question of why this was not planned from the outset? Nothing has been learned here from the mistakes or positive developments of other VPN protocols in the past. Pretty much every mistake that the developers of VPN protocols have ever made can also be found in OpenVPN, usually even more than once and even worse than with other protocols.

OpenVPN 2.2, OpenVPN 2.4, and OpenVPN 2.6 all have a different default behavior, all offer different config options and deprecate existing options, and they are only partially forward and backward compatible. And OpenVPN 3.0 will be very different from OpenVPN 2.x in many aspects. The fact that there is already a version 3.x in the making shows how little long term compatibility and stability this protocol offers and how many bad design decisions there were made in the past.

Diving Down the Rabbit Hole

From a technical point of view, the OpenVPN protocol is a jumble of many different ideas, none of which have really been thought through to the end. E.g. the protocol defines a package format that could be extended to support as many new packet types as needed but instead of that, there is an entirely different messaging system for checking client/server capabilities, as well as another one for automatic configuration and for user authentication. Then there are special traffic packets that are actually no traffic (like keep-alive pings) and that are only identified by length and fixed value, there are three incompatible ways how compressed packets can be encapsulated in the data stream, and so on. I think you get the point. It's a bit like every time something new was needed, a new data transport system was invented instead of extending an already existing one.

Also user authentication has been hardcoded to always be a username and a password. Things like OTP or 2FA were not even considered to exist. Further there is no way for the server to let the client know it expects user authentication; this is found in the static client config but that config could be outdated and the dynamic config is only sent after successful user authentication. And if dynamic protocol negotiation fails, there is no even a way to signal that error, instead the server signals "User Authentication Fails" (which is a lie) and just adds another error message at the end that explains the true reason (like "Cipher negotiation failed") but most clients will show the incorrect "Your login was rejected" message, despite the login being correct.

OpenVPN has two modes called TLSAuth and TLSCrypt. These two modes are basically the TLS modes (2) and (3), but on top of that, they also use a shared secret like mode (1) does to just authenticate or authenticate and encrypt the TLS handshake. If that mode is not configured correctly on both sides, the server will not understand the client message and simply never reply at all, making it impossible to find out if the problem is with the TLSAuth/-Crypt configuration or if there is a network issue and the server never gets the requests or the server replies don't make it back to the client, not unless you have log access to the server logs and on many VPN gateways the OpenVPN server log is not exposed (or very hard to get hold of).

Why the success?

There are several reasons why OpenVPN was a success despite all its problems:

• The protocol is entirely implemented in user space; it does not require any special kernel support, unlike IPsec or PPTP, an is thus very portable to operating systems, very firewall friendly, and easy to integrate into existing systems.
• It offers better performance than TCP based SSL VPN protocols.
• It is not a proprietary protocol of a single VPN gateway vendor.
• The full source code is available under GPL-2 license.
• The config file format (albeit constantly changing) is documented and thus an official config file format does exist, unlike with most other VPN protocols.
• The protocol does support automatic configuration and user authentication.

All of the above points also apply to WireGuard, with the exception of the last point. WireGuard is certainly the much better designed VPN protocol, and way faster than OpenVPN, but the entire configuration is static and it only supports client authentication with a private key, yet no way for users to log in with a username and password, which are probably the two biggest drawbacks of WireGuard. Of course, vendors can add their own configuration and authentication method and then simply use WireGuard as the line protocol, but that just creates a new bunch of proprietary, incompatible WireGuard variants. Also OpenVPN is simply older and thus had a head start.

Would you recommend OpenVPN?

Why VPN Tracker?

VPN Tracker is the best secure remote access solution for Mac, iPhone and iPad and is compatible with the most popular VPN gateways, including UniFi, NETGEAR, TP Link, Draytek, and many more.