IP Security

You cannot talk about VPNs without saying something about IP Security (IPSec). IPSec is a framework of open standards. It is not bound to any specific encryption or authentication algorithm keying technology. IPSec acts on the network layer, where it protects and authenticates IP packets between participating peers such as firewalls, routers, or concentrators. IPSec security provides four major functions:

• Confidentiality The sender can encrypt the packets before transmitting them across the network. If such a communication is intercepted, it cannot be read by anybody.
• Data integrity The receiver can verify whether the data was changed while traveling the Internet.
• Origin authentication The receiver can authenticate the source of the packet.
• Antireplay protection The receiver can verify that each packet is unique and is not duplicated.

Encryption
When packets are traveling on the Internet, they are vulnerable to eavesdropping. Clear-text messages can be intercepted and read by anybody. Therefore, to keep the data secure, it can be encrypted. For encryption to work, both the sender and the receiver need to know the rules that were used to encrypt the original message. There are two types of encryption:

• Symmetric
• Asymmetric

With symmetric key encryption, each peer uses the same key to encrypt and decrypt data. With asymmetric key encryption, each peer uses a different key to encrypt and decrypt the message. Both the Data Encryption Standard (DES) and Triple DES (3DES) require a symmetric shared secret key. The problem is then to give those keys to both users. The keys can be sent by mail, courier, or public key exchange. The easiest method to exchange the key is Diffie-Hellman public key exchange. This key exchange provides a way for the users to establish a shared secret key, which only they know, although they are sending it over an insecure channel.
Public key cryptosystems rely on a two-key system:

• A public key, which is exchanged between the users
• A private key, which is kept secret by the owners

The Diffie-Hellman public key algorithm states that if user A and user B exchange public keys and combine them with their private keys, the end result should be the same. This is shown in Figure 1.

Figure 1. Diffie-Hellman Key Exchange

Figure 1 is greatly simplified to ensure that the concept of Diffie-Hellman key exchange is clear. There are different variations to this algorithm, know as DH groups 1 through 7. During tunnel setup, VPN peers negotiate which DH group to use.

Encryption can also be accomplished by using the Rivest, Shamir, and Adelman (RSA) algorithm. The RSA algorithm uses an asymmetric key for encryption and decryption. Each user generates two keys: a private key and a public key. The users keep the private key for themselves and exchange the public key. To send an encrypted message to the other end, the local end encrypts the message by using the remote end’s public key and the RSA encryption algorithm. This message is then sent to the other end, where it is decrypted using that site’s private key. With RSA encryption, the opposite can also be true. The remote end can encrypt a message using its own private key, and the receiver can decrypt the message using the sender’s public key. This RSA encryption technique is used for digital signatures and is covered in more detail later.

Data Integrity
Data integrity is also a critical function of VPN because data is sent over a public network and can be intercepted and modified. To guard against this interception, every message has an attached hash. This hash guarantees the integrity of the message. The receiver checks this by comparing the received hash with the hash it calculates from the message itself. If both values are equal, the message has not been tampered with. However, if there is no match, the receiver knows that the message was altered.
IPSec uses the Hashed Message Authentication Codes (HMAC) protocol to calculate the hash. At the sender’s end, the message and the shared key are sent through a hash algorithm, which produces a hash value. Basically, this hash algorithm is a formula used to convert a variable-length message into a fixed-length hash. It is also important to understand that this is a one-way function. A message can produce a hash, but a hash cannot produce the original message. After the hash is calculated, it is sent over the network together with the message. At the other end, the receiver performs the same action. It sends the message and the shared key through the hash algorithm and then compares the two hashes to verify whether they match.
Two HMAC algorithms are commonly used:

• HMAC-MD5 This protocol uses a 128-bit shared key. The key and the message are combined to a 128-bit hash.
• HMAC-SHA-1 This protocol uses a 160-bit shared key. The length of the hash is 160 bits. This protocol is considered stronger because of the longer key.

Origin Authentication
Another important function is origin authentication. Before the electronic era, a seal or a signature on a letter guaranteed its origin. In the electronic era, a document is signed with the sender’s private encryption key. This is also called a digital signature. This signature can be authenticated by decrypting it with the sender’s public key. When doing business over a long distance, it is important to know who is at the other side of the phone, fax, and so on. The same is true for VPNs. The devices at the other end of the tunnel must be authenticated before the path is considered secure. There are three peer authentication methods:

• Preshared keys A secret key is entered into each peer manually.
• RSA signatures The exchange of digital certificates authenticates the peers.
• RSA encryption nonces Nonces (a random number generated by the peers) are encrypted and then exchanged between peers. The two nonces are used during the peer authentication process.

Preshared Keys
If preshared keys are used, the same key is configured on each IPSec peer. At each end, the preshared keys are combined with other information (device-specific information) to form the authentication key. They are both sent through a hash algorithm to form a hash. Then the hash is sent to the other site, as you can see in Figure 2.

Figure 2. Preshared Keys

If the remote peer is able to independently create the same hash, the local peer is authenticated. After that, the authentication process continues in the opposite direction. The remote peer combines its specific information with the preshared key and sends the resulting hash to the local peer. If this peer can create the same hash from its stored information and the preshared key, the remote peer is authenticated. Each peer must authenticate its opposite peer before the tunnel is considered secure. This system with preshared keys is easy to configure manually but does not scale very well. Each IPSec peer must be configured with the preshared key of every other peer with which it wants to communicate.
RSA Signatures
With RSA signatures, both hashes are not only authenticated but also digitally signed. At the local end, the authentication key and identity information are sent through the hash algorithm to form the hash, a process similar to that used with preshared keys. But with RSA signatures, the hash is then encrypted using the local peer’s private key. The result of this procedure is a digital signature, as you can see in Figure 3. The digital signature and a digital certificate are both forwarded to the other site. The public encryption key that is also used to decrypt the signature is included in the digital certificate.

Figure 3. RSA Signatures

At the remote peer, the peer authentication is a two-step process. First, the remote site verifies the digital signature by decrypting it with the public key. The result should be the same hash that the local end made. Next, the remote peer independently creates a hash from its stored information and the authentication key, and this also results in a hash. If the hashes are equal, the local peer is authenticated.
After the local peer is authenticated, the process starts all over in the opposite direction. With this kind of authentication, both peers must authenticate their opposite peer before the tunnel is considered secure.
RSA-Encrypted Nonces
RSA-encrypted nonces require that each site generate a nonce. As stated previously, a nonce is a pseudorandom number. The generated nonces are then encrypted and exchanged. When the other side receives the nonces, it makes an authentication key from both nonces and some other information. That nonce-based key is then combined with device-specific information and run though the hash algorithm, as shown in Figure 4. After this, the process is similar to that used for RSA signatures.

Figure 4. RSA-Encrypted Nonces

NOTE
The word nonce comes from “number used once.”
Antireplay Protection
Antireplay protection verifies that each packet is unique and not duplicated. IPSec packets are protected by comparing the sequence number of the received packets and a sliding window on the destination host. Packets in which the sequence number is before the sliding window are considered late, or duplicate. These packets are dropped.
Protocol Framework
The previous sections discussed encryption, integrity, and authentication. Now let’s apply these three concepts to the IPSec protocol suite. IPSec is a framework of open standards. IPSec relies on existing technology, such as DES and 3DES, to secure the communication between two entities. There are two main IPSec framework protocols available:

• Authentication header (AH)
• Encapsulating security payload (ESP)

AH
AH is the protocol to use when confidentiality is not required. It provides data authentication and integrity for IP packets between two systems. It verifies that the origin of the packet is correct and that the packet is not modified during transport. It does not encrypt the data packet, so the text is transported in clear text.
Authentication is achieved by using a one-way hash function to create a message digest. The hash is then combined with the text and transmitted to the other site. When the packet reaches its destination, the receiver performs the same one-way hash function and compares the result with the message digest that the sender has supplied. Because the one-way hash uses a symmetric key between the two systems, the authenticity of the packet is guaranteed. The AH function is applied to the entire datagram, except for some header fields that change in transit, such as the Time-To-Live field. The workings of AH are shown in Figure 5 and are spelled out in the following steps:
Step 1. The IP header and data payload are hashed.
Step 2. The hash is used to build the AH, which is inserted into the original packet.
Step 3. The modified packet is send to the peer router.
Step 4. The peer router hashes the IP header and data payload.
Step 5. The router extracts the transmitted hash from the AH.
Step 6. The peer router compares the two hashes. The hashes have to match exactly to prove that the packet was not modified during transport.

Figure 5. AH

AH supports both HMAC-MD5 and HMAC-SHA-1 algorithms.
ESP
ESP can be used to provide encryption and authentication. It provides confidentiality by performing encryption at the IP packet layer. ESP provides authentication for the IP packet payload and the ESP header. As with AH, ESP verifies the following: that the packet originated from where it declares it did, that it is what it declares it is, and that the packet was not modified during transport.
ESP provides confidentiality by encrypting the payload. It supports several symmetric encryption algorithms. The default for IPSec is 56-bit DES, but Cisco products also support 3DES and AES for stronger encryption. ESP can be used alone or in combination with AH. Between two security gateways, the original data is well protected because the entire IP packet is encrypted. An ESP header and trailer are added to the encrypted payload, as shown in Figure 6.

Figure 6. ESP

With authentication, the encrypted IP datagram and the ESP header and trailer are included in the hashing process. A new IP header is appended to the front of the packet. This new IP header is used to route the packet through the Internet. When both ESP authentication and encryption are selected, encryption is performed before authentication. One of the main reasons for this order of processing is that it facilitates rapid detection and rejection of incorrect packets at the receiving side. Before decrypting the packet, the receiver can check the authentication of the packets. This requires less processing time and can reduce the impact of denial-of-service (DoS) attacks.
Tunnel or Transport Mode
Both ESP and AH can be applied to IP packets in two different ways:

• Transport mode
• Tunnel mode

These two different modes provide a further level of authentication or encryption support to IPSec. The sections that follow discuss these two IPSec modes in more detail.
Transport Mode
This mode is primarily used for end-to-end connections between hosts or devices acting as hosts. Transport mode protects the payload of the packet but leaves the original IP address readable. This address is used to route a packet through the Internet. Transport mode provides security to the higher layer protocols only. Figure 7 shows how transport mode affects AH IPSec connections.

Figure 7. AH Transport Mode

The Layer 3 and Layer 4 headers are pried apart, and the AH is added between them.
Figure 8 shows ESP transport mode. Again, the IP header is shifted to the left, and the ESP header is inserted. The ESP trailer and ESP authentication are then appended to the end of the packet.

Figure 8. ESP Transport Mode

Although the original header remains intact in both situations, the AH transport does not support Network Address Translation (NAT) because changing the source address in the IP header would cause the authentication to fail. If NAT is needed with AH transport mode, make sure that NAT happens before IPSec. ESP transport mode does not have this problem. The IP header remains outside the authentication and encryption area.