What Is IPSec?

How the IPSec protocol protects data over VPN networks

IPSec concept immage

Funtap/Getty Images

IPSec, which stands for Internet Protocol Security, is a suite of cryptographic protocols used to protect data traffic over Internet Protocol (IP) networks.

Internet Protocol (IP) networks — including the World Wide Web as we know it — lack encryption and privacy. IPSec VPNs were designed to address this weakness, by providing a framework for encrypted and private communication on the web.

Here's a closer look at what IPSec is and how it works with VPN tunnels to protect data over unsecured networks.

A Brief History of IPSec

When the Internet Protocol was developed back in the early '80s, security was not high on the list of priorities. However, as the number of internet users continued to grow, the need for greater security became evident.

To address this need, the NSA (National Security Agency) began sponsoring the development of security protocols back in the mid-80s under the Secure Data Network Systems (SDNS) program. This led to the development of Security Protocol at Layer 3 (SP3), and eventually, the Network Layer Security Protocol (NLSP). Several more engineers worked on this project throughout the '90s, and IPSec grew out of these efforts. IPSec is now an open source standard as a part of the IPv4 suite.

How IPSec Works

When two computers want to establish a VPN connection, a few things need to happen to create the encrypted tunnel. They must agree on a set of security protocols and encryption algorithms, and exchange cryptographic keys to unlock and view the encrypted data.

That's where IPSec enters the picture. IPSec works with VPN tunnels to establish a private two-way connection between devices. IPSec is not a single protocol; rather, it's a complete suite of protocols and standards that work together to help ensure the confidentiality, integrity, and authentication of internet data packets flowing through a VPN tunnel.

Here's how IPSec creates a secure VPN tunnel:

  • It authenticates data to ensure data packet integrity in transit.
  • It encrypts internet traffic over VPN tunnels so data can't be viewed.
  • It protects against data replay attacks which can lead to unauthorized logins.
  • It enables secure cryptographic key exchange between computers.
  • It offers two security modes: tunnel and transport.

VPN IPSec protects data flowing from host-to-host, network-to-network, host-to-network, and gate to gateway (aka tunnel mode, when an entire IP packet is encrypted and authenticated).

IPSec Protocols and Supporting Components

The IPSec standard can be broken down into core protocols and supporting components. Here's a look at each.

IPSec Core Protocols

  • IPSec Authentication Header (AH): This protocol is used to protect the IP addresses of the computers involved in a data exchange. This ensures that bits of data are not lost, changed, or damaged during transmission. AH also verifies that the person who sent the data actually sent it, protecting the tunnel from infiltration by unauthorized users.
  • Encapsulating Security Payload (ESP): The ESP protocol provides the encryption part of the IPSec, which ensures the confidentiality of data traffic between devices. ESP encrypts the data packets/payload and authenticates the payload and its origin within the IPSec protocol suite. This protocol effectively scrambles internet traffic, so that anyone looking at the tunnel can't see what's there.

ESP can be use to both encrypt and authenticate data, whereas AH can only be used to authenticate data.

IPsec Supporting Components

  • Security Associations (SA): Security Associations and policies are used to establish the different security 'contracts' used in an exchange. These contracts might define the type of encryption and hashing algorithms to be used. These policies are often flexible, allowing devices to decide how they want to handle things.
  • Internet Key Exchange (IKE): For encryption to work, the computers involved in a private communication exchange need to share encryption keys. IKE allows two computers to securely exchange and share cryptographic keys when establishing a VPN connection.
  • Encryption and Hashing Algorithms: A cryptographic key works using a hash value, which is generated using a hash algorithm. AH and ESP are generic in that they don't specify a particular type of encryption. However, IPsec often uses the Message Digest 5 (MD5) or the Secure Hash Algorithm 1 (SHA-1) for encryption. 
  • Anti-replay protection: IPSec also incorporates standards to prevent the replay of any data packets that are part of a successful login process. This prevents hackers from using replayed information to replicate the login themselves.

    IPSec is used as a complete VPN protocol solution on its own, or as an encryption protocol within L2TP and IKEv2.

    Tunneling Modes: Tunnel and Transport

    IPSec has the ability to send data either using tunnel or transport mode. These modes are closely related to the type of protocols used, either AH or ESP.

    • Tunnel mode: In tunnel mode, the entire packet is protected. IPSec wraps the data packet in a new packet, encrypts it, and adds a new IP header. It is commonly used in site-to-site VPN set ups.
    • Transport mode: In transport mode, the original IP header remains and is not encrypted. Only the payload and ESP trailer are encrypted. Transport mode is often used in client-to-site VPN set ups.

    As far as VPNs go, the most common IPSec configuration you'll see is ESP with authentication in tunnel mode. This allows internet traffic to move securely and anonymously inside a VPN tunnel over unsecured networks.