In which mode of IPSec is the data within an IP packet encrypted while the header information is not?

IPSec is a set of communication rules or protocols for setting up secure connections over a network. Internet Protocol (IP) is the common standard that determines how data travels over the internet. IPSec adds encryption and authentication to make the protocol more secure. For example, it scrambles the data at its source and unscrambles it at its destination. It also authenticates the source of the data. 

The Internet Engineering Task Force developed IPSec in the 1990s to ensure data confidentiality, integrity, and authenticity when accessing public networks. For example, users connect to the internet with an IPSec virtual private network (VPN) to access company files remotely. The IPSec protocol encrypts sensitive information to prevent unwanted monitoring. The server can also verify that the received data packets are authorized.

IPsec can be used to do the following:

  • Provide router security when sending data across the public internet.
  • Encrypt application data.
  • Authenticate data quickly if the data originates from a known sender.
  • Protect network data by setting up encrypted circuits, called IPsec tunnels, that encrypt all data sent between two endpoints.

Organizations use IPSec to protect against replay attacks. A replay attack, or man-in-the-middle attack, is an act of intercepting and altering ongoing transmission by routing data to an intermediary computer. IPSec protocol assigns a sequential number to each data packet and performs checks to detect signs of duplicate packets. 

IPSec encryption is a software function that scrambles data to protect its content from unauthorized parties. Data is encrypted by an encryption key, and a decryption key is needed to unscramble the information. IPSec supports various types of encryptions, including AES, Blowfish, Triple DES, ChaCha, and DES-CBC. 

IPSec uses asymmetric and symmetric encryption to provide speed and security during data transfer. In asymmetric encryption, the encryption key is made public while the decryption key is kept private. Symmetric encryption uses the same public key for encrypting and decrypting data. IPSec establishes a secure connection with asymmetric encryption and switches to symmetric encryption to speed up data transfer.

Computers exchange data with the IPSec protocol through the following steps. 

  1. The sender computer determines if the data transmission requires IPSec protection by verifying against its security policy. If it does, the computer initiates secure IPSec transmission with the recipient computer.
  2. Both computers negotiate the requirements to establish a secure connection. This includes mutually agreeing on the encryption, authentication, and other security association (SA) parameters. 
  3. The computer sends and receives encrypted data, validating that it came from trusted sources. It performs checks to ensure the underlying content is reliable. 
  4. Once the transmission is complete or the session has timed out, the computer ends the IPSec connection. 

IPSec protocols send data packets securely. A data packet is a specific structure that formats and prepares information for network transmission. It consists of a header, payload, and trailer.

  • A header is a preceding section that contains instructional information for routing the data packet to the correct destination. 
  • Payload is a term that describes the actual information contained within a data packet.
  • The trailer is additional data appended to the tail of the payload to indicate the end of the data packet. 

 Some IPSec protocols are given below.

Authentication header (AH)

The authentication header (AH) protocol adds a header that contains sender authentication data and protects the packet contents from modification by unauthorized parties. It alerts the recipient of possible manipulations of the original data packet. When receiving the data packet, the computer compares the cryptographic hash calculation from the payload with the header to ensure both values match. A cryptographic hash is a mathematical function that summarizes data into a unique value. 

Encapsulating security payload (ESP)

Depending on the selected IPSec mode, the encapsulating security payload (ESP) protocol performs encryption on the entire IP packet or only the payload. ESP adds a header and trailer to the data packet upon encryption. 

Internet key exchange (IKE)

Internet key exchange (IKE) is a protocol that establishes a secure connection between two devices on the internet. Both devices set up security association (SA), which involves negotiating encryption keys and algorithms to transmit and receive subsequent data packets. 

IPSec operates in two different modes with different degrees of protection. 

Tunnel

The IPSec tunnel mode is suitable for transferring data on public networks as it enhances data protection from unauthorized parties. The computer encrypts all data, including the payload and header, and appends a new header to it. 

Transport

IPSec transport mode encrypts only the data packet's payload and leaves the IP header in its original form. The unencrypted packet header allows routers to identify the destination address of each data packet. Therefore, IPSec transport is used in a close and trusted network, such as securing a direct connection between two computers. 

VPN, or virtual private network, is a networking software that allows users to browse the internet anonymously and securely. An IPSec VPN is a VPN software that uses the IPSec protocol to create encrypted tunnels on the internet. It provides end-to-end encryption, which means data is scrambled at the computer and unscrambled at the receiving server. 

SSL VPN 

SSL stands for secure socket layer. It is a security protocol that protects web traffic. An SSL VPN is a browser-based network security service that uses the built-in SSL protocol to encrypt and safeguard network communication. 

What is the difference between IPSec VPN and SSL VPN?

Both security protocols work on different layers of the open systems interconnection (OSI) model. The OSI model defines the layered structure of how computers exchange data on a network. 

IPSec protocols apply to the network and transport layers in the middle of the OSI model. Meanwhile, SSL encrypts data on the topmost application layer. You can connect to an SSL VPN from a web browser but must install separate software to use IPSec VPNs.

AWS Site-to-Site VPN is a fully managed service that creates a secure connection between your data center or branch office and your AWS resources using IPSec tunnels. When using Site-to-Site VPN, you can connect to both your Amazon Virtual Private Clouds (VPC) as well as AWS Transit Gateway, and two tunnels per connection are used for increased redundancy. AWS Site-to-Site VPN brings many benefits such as:

  • Visibility into local and remote network health with performance monitoring.
  • Secure and easy migration of local applications to the AWS cloud.
  • Improved application performance when integrated with  AWS Global Accelerator.

Get started with AWS VPN by signing up for an AWS account today.

IPsec Authentication Header (AH) is a security protocol used to protect data sent over a network. Its core focus is around data integrity and authentication. It is also responsible for authenticating IP packets and helps protect against network attacks. 

The IPsec Encapsulating Security Payload (ESP) protocol protects data confidentiality and data origin authentication. Both IPsec AH and IPsec ESP focus on encryption with the difference coming in the use of both protocols in the IPsec modes which we will discuss below. IPsec Tunnel mode and IPsec Transport mode. 

In order to get a better understanding of the differences in each IPsec transport mode, let’s first discuss the use cases for them.

The IPsec AH tunnel mode sets up a secure connection between two communication endpoints on the internet. This is the most common mode to use when connecting to a VPN server. While the AH protocol establishes a VPN tunnel without encrypting data, it instead provides integrity of the data packets.

The IPsec ESP tunnel mode encrypts and encapsulates IP packets while also providing authentication and integrity. This protocol is used by VPN tunnels to see if data packets have been tampered with while in transit. This allows VPN connections to be routed through untrusted networks while maintaining encrypted data packets.

In order to configure IPsec tunnel mode, you will first need to use a different protocol such as IKE (Internet Exchange Key) to negotiate the parameters that will be used in order to secure the communication between peers. The IKE setup process is broken down into 2 phases: 

  • IKE Phase 1: The initial IKE phase establishes a secure tunnel between channels. The main purpose in the first step is to authenticate IPsec peers and to negotiate security associations (SA). 
  • IKE Phase 2: Once a security association has been established, the next step is to negotiate authentication and encryption, thus encrypting the entire packet which is then included in the payload or the transmission of data from the intended message.

IPsec AH transport mode is a security protocol used to protect data through your network, but it doesn’t make a secure connection. It encrypts the data being sent without checking for integrity or authentication, which makes it faster than IPSec AH Tunnel Mode. However, it is far less secure.

IPsec ESP transport mode secures data sent over a network, providing privacy by encrypting it, and this protocol provides authentication and integrity. It is used by VPN tunnels to ensure that data is secured while in transit without having to establish a secure connection between two points on the internet. 

The final destination in ESP transport mode is typically the host. The other point to take into consideration is that ESP transport mode encrypts the data only and not the original headers.

IPsec transport mode secures traffic from one system to another. There is also a two-step configuration process similar to tunnel mode.

  • IKE Phase 1: The key focus here is on the negotiation of the secure channel between two systems using an ISAKMP security association or Internet Security Association and Key Management Protocol. 
  • IKE Phase 2: In this step, the IKE peers dynamically negotiate the authentication and encryption algorithms to secure the payload.

Transport mode is seen as less secure than tunnel mode because the IP header is not encrypted.

IPsec tunnel mode sets up a secure connection, while IPsec Transport Mode only encrypts the data being sent without establishing a secure connection.

In transport mode, the sending and receiving hosts establish a connection before exchanging data. In tunnel mode, a second IP packet is sent in a completely different protocol. This protects data packets from being inspected or modified in transit. 

The advantages of tunnel mode over transport mode are that it can work through Network Address Translation (NAT) and that the entire original IP packet is hidden. NAT maps a private IP address to a public IP address by modifying network address information in the IP header of packets across a traffic routing device while in transit.  

The major disadvantages of tunnel mode are additional overhead from encapsulation, an inability to defend against attacks on weak integrity protocols, and that transport mode may be more compatible with some firewalls. 

Examples of such attacks include SYN floods which is a type of distributed denial-of-service (DDoS) attack. SYN floods send massive requests to overwhelm a server, rendering the system unavailable to receive legit traffic. It also prevents the completion of the TCP three-way handshake between client and server needed for a secure connection. 

In general, tunnel mode is better when both endpoints are behind a NAT device, and transport mode is preferable when there is no NAT or if the network uses pre-NAT devices with address translation only at the IP packet level. In most cases, transport mode will provide better security with less overhead.

In order to know when to use either tunnel mode or transport mode, you should consider where each endpoint is located in relation to the internet. If you are both behind the NAT device, then tunnel mode is better because it establishes a connection while transport mode simply encrypts packets.

However, if only one of the endpoints is behind the NAT device, you’ll need to use transport mode so both hosts can communicate securely with each other.

Both tunneling and transport mode encrypts data, but when implementing one over the other, you should consider whether there are NAT devices between two connected networks. If no NAT device exists, use transport mode. If a NAT or pre-NAT firewall exists, use tunnel mode.

For example, if you’re using a pre-NAT firewall and your endpoint is located in the same private network as the server, use transport mode. If your endpoint is located behind a NAT device or on a different network than the server you’ll be connecting to through IKEv2, then use tunnel mode.

Transport mode works best for firewalls that do not translate IP addresses in the packet header and for cases where transports mode is more compatible with certain firewalls.

The main advantage of IPsec transport mode is that it is more compatible with certain firewalls and it offers higher levels of security. In addition, transport mode does not require a secure connection to be established between two endpoints and has less overhead because it does not encapsulate packets.

The main disadvantage of IPsec transport mode is the difficulties it has with NAT traversal or UDP encapsulation. The User Datagram Protocol (UDP) is a technique of adding network headers to the packets and helps with load balancing to better distribute network traffic.

The main advantage of IPsec tunnel mode is that it creates a secure connection between two endpoints by encapsulating packets in an additional IP header. Tunnel mode also provides better security over transport mode because the entire original packet is encrypted.

The main disadvantage of the IPsec tunnel mode is that it requires a secure connection to be established between two endpoints and tends to create more overhead because the entire original packet must be encapsulated. In addition, transport mode may perform better than tunnel mode on some types of networks and with certain firewalls.

In order to know which mode is best for you, consider your network environment. You might also want to consider an IPsec VPN to create encrypted tunnels and secure remote access to an entire network, whether on-premises or from corporate headquarters.

Perimeter 81’s IPsec VPN enables organizations to work safely from anywhere in the world by establishing a secure connection between devices.

Perimeter 81’s IPsec VPN leverages the principles of Zero Trust to provide a stronger level of security across the network. This allows admins to create policies based on authentication factors such as Multi-Factor Authentication (MFA) and 256-bit encryption.

Instantly deploy your entire network with Perimeter 81’s IPsec VPN. See how radically simple it is for yourself. Request a demo today.