IPSec short

CHAPTER 18: Security at the Network Layer: IPSec

Introduction

• Security at the application and transport layers (like PGP, S/MIME, SSL/TLS) may not be sufficient because:

  • Not all client/server programs are protected at the application layer.
  • Not all programs use TCP (which SSL/TLS relies on); some use UDP.
  • Many applications (e.g., routing protocols) use IP directly and need security at the IP layer.

• IP Security (IPSec): A collection of protocols designed by the IETF to provide security for packets at the network (IP) layer.

• IPSec helps create authenticated and confidential packets for the IP layer.

• Usefulness of IPSec:

  • Enhances security for applications with their own security protocols (e.g., email).
  • Enhances security for applications using transport layer security (e.g., HTTP with TLS).
  • Provides security for applications not using transport layer security.
  • Provides security for node-to-node communication (e.g., routing protocols).

18.1 TWO MODES

IPSec operates in two different modes: transport mode or tunnel mode.

Transport Mode

• Protection Scope: Protects the payload delivered from the transport layer to the network layer (i.e., the network layer payload).

• Encapsulation: The IPSec header and trailer are inserted between the original IP header and the transport layer payload.

• IP Header: The original IP header is not protected by IPSec in transport mode; it is added after IPSec processing.

• Usage: Typically used for host-to-host (end-to-end) protection.

  • Sending host uses IPSec to authenticate/encrypt the transport layer payload.
  • Receiving host uses IPSec to check authentication/decrypt the packet before delivering to the transport layer.

Tunnel Mode

• Protection Scope: Protects the entire original IP packet, including the original IP header.
• Encapsulation: IPSec takes the entire IP packet, applies security, and then adds a new IP header.
• New IP Header: Contains different information (e.g., different source/destination addresses, often addresses of security gateways like routers) than the original IP header.
• Usage: Typically used when at least one endpoint is a security gateway (e.g., router).
  • Between two routers.
  • Between a host and a router.
  • Between a router and a host.
  • Used when the sender or receiver is not the final host. Protects the packet over an “imaginary tunnel”.

Comparison (Transport vs. Tunnel)

• Transport Mode:
  • IPSec layer is between the Transport and Network layers.
  • Protects Transport layer payload.
  • Original IP header is exposed.
• Tunnel Mode:
  • IPSec layer effectively encapsulates the Network layer packet and passes it back to the Network layer to add a new header.
  • Protects the entire original IP packet (including header).
  • Original IP header is hidden inside the new packet.
• 

18.2 TWO SECURITY PROTOCOLS

IPSec defines two main protocols:

1. Authentication Header (AH)
2. Encapsulating Security Payload (ESP)

Authentication Header (AH)

• Purpose:

  • Provides source host authentication.
  • Ensures data integrity of the payload.

• Mechanism: Uses a hash function and a symmetric key to create a message digest (Authentication Data).

• Placement: The AH header is placed in the packet based on the mode (transport or tunnel).

• Protocol Number: When AH is used, the Protocol field in the preceding IP header is set to $51$.

• AH Header Format (Figure 18.7):

  • Next Header ($8$ bits): Stores the original value of the Protocol field (e.g., TCP, UDP, ICMP). Identifies the type of payload following AH.
  • Payload Length ($8$ bits): Defines the length of the AH header itself in $4$-byte multiples, excluding the first $8$ bytes. (Misleading name).
  • Reserved ($16$ bits): Reserved for future use.
  • Security Parameter Index (SPI) ($32$ bits): Identifies the Security Association (SA) for this packet. Acts like a virtual circuit identifier.
  • Sequence Number ($32$ bits): Provides ordering information, prevents replay attacks. Starts at $0$, increments, does not wrap around (new SA needed after $2^{32}-1$). Not repeated for retransmissions.
  • Authentication Data (digest) (variable length): The message digest calculated over parts of the IP packet (including parts of the IP header that don’t change in transit, the AH header itself with this field zeroed, and the payload). Padding may be added before hashing.

• Process for Adding AH:

  1. Add AH header to payload (Authentication Data field = $0$).
  2. Add padding if required by the hash algorithm.
  3. Calculate hash (message digest) over the packet (excluding mutable IP header fields).
  4. Insert the calculated hash into the Authentication Data field.
  5. Add/Modify the IP header, setting the Protocol field to $51$ and storing the original protocol type in AH’s Next Header field.

• Security Provided: Source authentication and data integrity. Does not provide confidentiality (privacy).

Encapsulating Security Payload (ESP)

• Purpose:
  • Provides source authentication.
  • Provides data integrity.
  • Provides confidentiality (privacy) through encryption.
• Mechanism: Adds an ESP header and an ESP trailer. Authentication data is added at the end.
• Placement: Figure 18.8 shows ESP header and trailer placement (likely in transport mode, though not explicitly stated for this figure).
• Protocol Number: When ESP is used, the Protocol field in the preceding IP header is set to $50$.
• ESP Format (Figure 18.8):
  • ESP Header:
    • Security Parameter Index (SPI) ($32$ bits): Same function as in AH. Identifies the SA.
    • Sequence Number ($32$ bits): Same function as in AH. Prevents replay.
  • Payload Data (variable length): The original payload (e.g., transport layer segment). This part is encrypted.
  • ESP Trailer:
    • Padding (0-255 bytes): Used for block cipher padding requirements and to align the Pad Length and Next Header fields. Content is often $0$s. This part is encrypted.
    • Pad Length ($8$ bits): Specifies the number of padding bytes added. This part is encrypted.
    • Next Header ($8$ bits): Stores the original value of the Protocol field (identifies the type of payload data). This part is encrypted.
  • Authentication Data (variable length): A digest calculated over the ESP header, payload data, and ESP trailer. This part is not encrypted. Added after encryption.
• Process for Adding ESP:
  1. Add ESP trailer to the original payload.
  2. Encrypt the combined payload and ESP trailer.
  3. Add the ESP header before the encrypted data.
  4. Calculate the authentication data over the ESP header and the encrypted (payload+trailer).
  5. Append the authentication data.
  6. Add/Modify the IP header, setting the Protocol field to $50$ and storing the original protocol type in ESP’s Next Header field (inside the trailer).
• Security Provided: Source authentication, data integrity, and confidentiality.
• Authentication Scope Difference: AH authentication includes parts of the IP header, ESP authentication typically does not (it authenticates from the ESP header onwards).

IPv4 and IPv6

• IPSec supports both IPv4 and IPv6.
• In IPv6, AH and ESP are implemented as extension headers.

AH versus ESP

• ESP was designed after AH and provides all the services of AH plus confidentiality.
• Logically, AH might seem redundant.
• However, AH implementations exist in commercial products, so it remains part of the standard until phased out.

Services Provided by IPSec

Table 18.1 IPSec services

Service	AH	ESP	Description
Access control	yes	yes	Indirectly provided via Security Association Database (SAD). Packets without a matching SA are discarded.
Message integrity	yes	yes	Ensured by the keyed hash (Authentication Data field) created by the sender and verified by the receiver.
Entity authentication	yes	yes	Provided by the Security Association (SA) establishment and the keyed hash, authenticating the sender.
Confidentiality	no	yes	Provided by encryption in ESP. AH does not encrypt data. Use ESP if confidentiality is needed.
Replay attack protection	yes	yes	Prevented using sequence numbers and a fixed-size sliding receiver window (default size $W=64$). (See Figure 18.9).

Replay Attack Protection Details (Sliding Window)

• Each IPSec header (AH or ESP) has a unique sequence number per SA.
• Starts at $0$, increments, reset to $0$ when $2^{32}-1$ is reached (requires a new SA).
• Receiver maintains a sliding window of expected sequence numbers, size $W$ (default $64$). The window represents sequence numbers $[N,N+W-1]$.
• Packet Arrival Processing (Figure 18.9):
  1. Sequence Number $<N$: Packet is to the left of the window. Discarded (duplicate or too old).
  2. Sequence Number $\in [N,N+W-1]$: Packet is inside the window.
     • If packet is new (sequence number not marked as received) AND passes authentication: Mark sequence number as received, accept packet.
     • Otherwise (already received or fails authentication): Discard packet.
  3. Sequence Number $>N+W-1$: Packet is to the right of the window.
     • If packet passes authentication: Mark sequence number as received, slide window to the right so the new sequence number is the rightmost edge (new $N$ = received_seq_num - $W+1$). Accept packet.
     • Otherwise: Discard packet.
• Note: If a packet arrives with a sequence number much larger than $N+W-1$, sliding the window might cause unmarked sequence numbers to fall to the left, meaning those packets will be discarded if they arrive later.

18.3 SECURITY ASSOCIATION (SA)

• Concept: A logical relationship or contract between two communicating entities (hosts or gateways) required by IPSec. It defines the security parameters for communication in one direction.
• Unidirectional: For bidirectional communication, two SAs are needed (one inbound, one outbound).
• Simple SA Example (Figure 18.10): Alice sending confidentially to Bob.
  • Alice’s Outbound SA: Stores shared secret key, encryption algorithm (e.g., DES).
  • Bob’s Inbound SA: Stores the same shared secret key, decryption algorithm (e.g., DES).
• Complexity: SAs can be more complex, including parameters for integrity algorithms, keys, lifetimes, etc., especially when using AH or ESP.

Security Association Database (SAD)

• Hosts/gateways using IPSec need to store multiple SAs (for different peers, different protocols, inbound/outbound).
• This collection of SAs is stored in a Security Association Database (SAD).
• Typically, two SADs exist: one for inbound SAs and one for outbound SAs.
• Inbound SA Selection: An inbound SA is selected using a triple index:
  1. Security Parameter Index (SPI): A $32$-bit value carried in the AH/ESP header that uniquely identifies the SA at the destination. Determined during SA negotiation (e.g., via IKE).
  2. Destination Address: The IP address of the receiving host/gateway. Needed because a host might have multiple addresses (unicast, multicast).
  3. Protocol: The IPSec protocol being used (AH or ESP). Different SAs are needed for AH and ESP.
• Outbound SA Selection: (Not explicitly indexed in the same way by incoming packet fields, but selected based on policy matching, see SPD section).

SA Parameters

Each entry (row) in the SAD defines an SA and contains parameters. Typical parameters include (Table 18.2): Table 18.2 Typical SA Parameters

Parameter	Description
Sequence Number Counter	$32$-bit value used to generate the next sequence number for AH/ESP headers.
Sequence Number Overflow	Flag indicating action on sequence number counter overflow (e.g., trigger SA renegotiation).
Anti-Replay Window	Information for replay detection (e.g., window size $W$, received packet bitmap).
AH Information	Parameters specific to AH (Authentication algorithm, keys, key lifetime, etc.).
ESP Information	Parameters specific to ESP (Encryption algorithm, Authentication algorithm, keys, key lifetime, IVs, etc.).
SA Lifetime	Defines how long the SA is valid (time-based or byte-based).
IPSec Mode	Transport or Tunnel mode.
Path MTU	Maximum Transmission Unit for the path (relevant for fragmentation).

18.4 SECURITY POLICY (SP)

• Concept: Defines what type of security (if any) should be applied to IP packets based on their characteristics.
• Determines whether and how IPSec should be used before consulting the SAD.

Security Policy Database (SPD)

• Hosts/gateways using IPSec maintain a Security Policy Database (SPD).
• Separate inbound and outbound SPDs exist.
• SPD Entry Selection: Each policy entry is indexed by a sextuple index (selectors):
  1. Source Address
  2. Destination Address (can be unicast, multicast, wildcard)
  3. Name (e.g., DNS entity)
  4. Protocol (e.g., TCP, UDP, ICMP, or AH/ESP themselves if processing nested tunnels)
  5. Source Port
  6. Destination Port

Outbound SPD Processing

1. Packet selectors (source/dest IP, ports, protocol) are used to find a matching policy in the outbound SPD.
2. The output of the SPD lookup determines the action:
   • Drop: Policy dictates the packet should be discarded.
   • Bypass: Policy dictates IPSec is not needed for this packet. Packet is sent without IPSec processing.
   • Apply: Policy dictates IPSec must be applied.
     • If matching Outbound SA exists in SAD: Retrieve the SA parameters (identified by the policy, perhaps via an SA pointer in the SP entry). Apply AH/ESP processing using the SA. Transmit the packet.
     • If no matching Outbound SA exists: The Internet Key Exchange (IKE) protocol is invoked to negotiate and establish the required SA(s) (both outbound for the source and inbound for the destination). Once established, the new outbound SA is added to the SAD, and processing continues as above.

Inbound SPD Processing

1. Packet selectors are used to find a matching policy in the inbound SPD.
2. The output of the SPD lookup determines the action:
   • Discard: Policy dictates the packet should be dropped (even if it arrived with IPSec headers).
   • Bypass: Policy dictates IPSec is not expected for this packet. If the packet arrived with IPSec headers, they might be ignored (or it could be an error condition depending on specifics). Packet is delivered to the transport layer (or next processing step).
   • Apply: Policy dictates IPSec processing is required.
     • If packet arrived without IPSec headers: Discard the packet (policy requires IPSec, but it wasn’t applied).
     • If packet arrived with IPSec headers:
       • Use the triple index (SPI, Dest Addr, Protocol from IP header) to look up the corresponding inbound SA in the SAD.
       • If matching Inbound SA exists: Perform IPSec verification (decryption, authentication check) using the SA parameters. If successful, discard the AH/ESP header/trailer and deliver the inner packet to the transport layer. If verification fails, discard the packet.
       • If no matching Inbound SA exists: Discard the packet. (An SA should have been established previously).

18.5 INTERNET KEY EXCHANGE (IKE)

• Purpose: Protocol designed to negotiate, create, and manage Security Associations (SAs) for IPSec (both inbound and outbound).
• Trigger: Called when the SPD lookup indicates “Apply” but no suitable SA exists in the SAD.
• IKE creates SAs for IPSec.
• Complexity: IKE is complex and builds upon three other protocols
  • Oakley: A key agreement protocol based on Diffie-Hellman, with improvements (e.g., cookies, nonces, authentication). Defines mechanisms but not message formats.
  • SKEME: Another key exchange protocol, uses public-key encryption for entity authentication.
  • ISAKMP (Internet Security Association and Key Management Protocol): Defines packet formats, message exchanges, and parameters to implement the key exchange mechanisms specified by IKE. ISAKMP is the carrier protocol for IKE exchanges.

Improved Diffie-Hellman Key Exchange

• IKE uses Diffie-Hellman as the basis for key exchange.
• Basic Diffie-Hellman (Figure 18.16):
  • Agree on large prime $p$ and generator $g$ ($g$ generates group $<Z_p^{\ast },\times >$).
  • Alice chooses random secret $i$, computes $KE-I=g^i\mathrm{mod}p$, sends to Bob.
  • Bob chooses random secret $r$, computes $KE-R=g^r\mathrm{mod}p$, sends to Alice.
  • Shared symmetric key $K=(KE-R{)}^i\mathrm{mod}p=(KE-I{)}^r\mathrm{mod}p=g^{ir}\mathrm{mod}p$.
• Weaknesses of Basic Diffie-Hellman:
  1. Clogging Attack (DoS): Attacker sends many fake half-keys ($g^x\mathrm{mod}p$) to Bob. Bob wastes CPU resources calculating responses ($g^r\mathrm{mod}p$) and full keys ($g^{xr}\mathrm{mod}p$).
  2. Replay Attack: Attacker records messages from one session and replays them later.
  3. Man-in-the-Middle Attack: Eve intercepts messages, establishes separate keys with Alice ($K_{AE}$) and Bob ($K_{BE}$), and relays messages, potentially modifying them.
• IKE Improvements (based on Oakley/SKEME ideas):
  • Cookies for Clogging Attack Prevention (Figure 18.17):
    • Parties exchange cookies in initial messages before performing expensive Diffie-Hellman calculations.
    • Cookie = Hash(Peer Identifier (IP, port, etc.), Local Secret, Timestamp).
    • Initiator sends Cookie-I. Responder sends Cookie-R (and repeats Cookie-I). Initiator sends KE-I (and repeats Cookie-I, Cookie-R).
    • If a peer is fake (e.g., spoofed IP), it won’t receive the cookie response and cannot proceed. The legitimate peer avoids computation until cookies are verified (by receiving them back).
    • To protect against clogging attack, IKE uses cookies.
  • Nonces for Replay Attack Prevention:
    • Parties exchange unpredictable random numbers (nonces) in later messages (e.g., messages 3 & 4 in Main Mode).
    • Nonces ensure freshness and are included in cryptographic calculations (hashes, keys).
    • To protect against replay attack, IKE uses nonces.
  • Authentication for Man-in-the-Middle Prevention:
    • Parties must authenticate themselves and the integrity of messages exchanged.
    • Requires each party to prove possession of a secret.
    • To protect against man-in-the-middle attack, IKE requires that each party shows that it possesses a secret.
    • Types of Secrets in IKE: a. Preshared Secret Key: Symmetric key known only to the two parties beforehand. b. Public/Private Key Pair (Encryption): Party proves identity by decrypting a message encrypted with its public key. c. Public/Private Key Pair (Digital Signature): Party proves identity by signing a message with its private key, verifiable with its public key.

IKE Phases

• To avoid an infinite loop of needing SAs to create SAs, IKE is divided into two phases:
  • Phase I: Establishes a secure, authenticated channel between the two IKE peers. Creates the IKE SA (or ISAKMP SA). This phase is generic.
  • Phase II: Uses the secure channel (IKE SA) established in Phase I to negotiate SAs for other protocols, typically IPSec (AH/ESP). This phase is specific to the protocol needing the SA.
• Phase I Protection: Phase I messages themselves are initially unprotected but become progressively protected as keys are derived and authentication occurs during the exchange.

Phases and Modes

• Phase I Modes:
  • Main Mode: Provides identity protection for peers. Involves six messages.
  • Aggressive Mode: Faster, fewer messages (three messages). Does not provide identity protection during the exchange.
• Phase II Mode:
  • Quick Mode: Used to negotiate IPSec SAs under the protection of the Phase I IKE SA. Involves three messages.

Phase I Authentication Methods

Based on the type of pre-existing secret, Phase I (Main or Aggressive mode) can use one of four authentication methods:

1. Preshared Secret Key
2. Original Public Key (using public key encryption for nonce/ID)
3. Revised Public Key (using public key encryption only for nonce, derives temporary key)
4. Digital Signature

Phase I: Main Mode (6 Messages)

• Messages 1 & 2: Exchange cookies (in ISAKMP header), negotiate Phase I SA parameters (encryption/hash algorithms, Diffie-Hellman group, authentication method). Initiator proposes, Responder chooses. Establishes peer liveness (no clogging).
• Messages 3 & 4: Exchange Diffie-Hellman public values ($KE-I=g^i\mathrm{mod}p$, $KE-R=g^r\mathrm{mod}p$) and nonces ($N-I$, $N-R$). Other method-specific data may be included (e.g., IDs in some public key methods).
• Key Calculation: After message 4, both parties can calculate the shared secret Diffie-Hellman key $K=g^{ir}\mathrm{mod}p$. From this, several keys are derived using a pseudorandom function (prf) (e.g., HMAC):
  • $SKEYID$: Base keying material, calculation depends on auth method:
    • Preshared: $SKEYID=prf(\text{preshared-key},N-I|N-R)$
    • Public Key (Orig/Rev): $SKEYID=prf(N-I|N-R,g^{ir}\mathrm{mod}p)$
    • Signature: $SKEYID=prf(hash(N-I|N-R),\text{Cookie-I}|\text{Cookie-R})$
  • $SKEYID\_d$: Key used for deriving further keys. $SKEYID\_d=prf(SKEYID,g^{ir}\mathrm{mod}p|\text{Cookie-I}|\text{Cookie-R}|0)$
  • $SKEYID\_a$: Key used for authenticating Phase I messages. $SKEYID\_a=prf(SKEYID,SKEYID\_d|g^{ir}\mathrm{mod}p|\text{Cookie-I}|\text{Cookie-R}|1)$
  • $SKEYID\_e$: Key used for encrypting Phase I messages (after message 4). $SKEYID\_e=prf(SKEYID,SKEYID\_a|g^{ir}\mathrm{mod}p|\text{Cookie-I}|\text{Cookie-R}|2)$
• Hash Calculation (Authentication): Both parties calculate hashes to authenticate each other and the exchange. Hashes depend on the authentication method, but generally:
  • $HASH-I=prf(SKEYID,KE-I|KE-R|\text{Cookie-I}|\text{Cookie-R}|SA-I|ID-I)$
  • $HASH-R=prf(SKEYID,KE-I|KE-R|\text{Cookie-I}|\text{Cookie-R}|SA-I|ID-R)$
  • $SA-I$ is the initiator’s proposed SA payload. $ID-I$/$ID-R$ are initiator/responder identities. Using $SA-I$ protects the initiator’s proposal from modification.
• Messages 5 & 6: Exchange identities ($ID-I$, $ID-R$) and the calculated hashes ($HASH-I$, $HASH-R$) for authentication. These messages are encrypted using $SKEYID\_e$. Each party verifies the received hash using its own calculated version.

Main Mode: Preshared Secret-Key Method (Figure 18.20)

• Secret: Preshared symmetric key.
• Messages 1-4: Exchange cookies, SA params, nonces, DH values.
• Messages 5 & 6: Exchange encrypted IDs and Hashes ($HASH-I$, $HASH-R$). $SKEYID$ derived from preshared key and nonces.
• Issue: Responder needs Initiator’s ID to know which preshared key to use for $SKEYID$ calculation before decrypting message 5 which contains the ID. Often assumes ID = IP address (problematic for mobile hosts).

Main Mode: Original Public-Key Method (Figure 18.21)

• Secret: Pre-known public/private key pairs for encryption/decryption.
• Messages 1 & 2: Cookies, SA params.
• Messages 3 & 4: Exchange DH values. Also exchange Nonces and IDs, encrypted with the recipient’s public key.
• Messages 5 & 6: Exchange Hashes ($HASH-I$, $HASH-R$), encrypted with $SKEYID\_e$.
• $SKEYID$ derived from nonces and DH key $g^{ir}$. Authentication relies on the fact that only the legitimate owner of the private key could decrypt the nonce/ID needed to compute the correct $SKEYID$ and subsequent hashes. Uses double hashing (hash of nonces as key for HMAC).

Main Mode: Revised Public-Key Method (Figure 18.22)

• Secret: Public/private key pairs, but public key crypto used only to establish temporary secrets. Certificates (Cert-I, Cert-R) may be exchanged.
• Messages 1 & 2: Cookies, SA params.
• Messages 3 & 4: Exchange DH values, IDs, and Nonces. Initiator encrypts $N-I$ and $ID-I$ with Responder’s public key (R). Responder encrypts $N-R$ and $ID-R$ with Initiator’s public key (I). Optional certificates are sent encrypted with derived $SKEYID\_e$.
• Temporary Keys: $K-I=prf(N-I,\text{Cookie-I})$, $K-R=prf(N-R,\text{Cookie-R})$. These might be used for encrypting parts of messages 3/4 (details complex).
• Messages 5 & 6: Exchange Hashes ($HASH-I$, $HASH-R$), encrypted with $SKEYID\_e$.
• Reduces public key operations compared to original method.

Main Mode: Digital Signature Method (Figure 18.23)

• Secret: Public/private key pairs for signing. Certificates (Cert-I, Cert-R) often used.
• Messages 1-4: Exchange cookies, SA params, nonces, DH values.
• Messages 5 & 6: Exchange IDs, optional Certificates, and Signatures ($Sig-I$, $Sig-R$). These are encrypted with $SKEYID\_e$.
  • $Sig-I$ = Signature using Initiator’s private key over all data from messages 1-4.
  • $Sig-R$ = Signature using Responder’s private key over all data from messages 1-5.
• $SKEYID$ derived from hash of nonces and cookies. Authentication comes from verifying the signatures using the sender’s public key (obtained from certificate or pre-known).

Phase I: Aggressive Mode (3 Messages)

• Compressed version of Main Mode. Faster, but exposes identities.
• Message 1: Initiator sends Cookie-I, SA proposal, KE-I, N-I, ID-I.
• Message 2: Responder sends Cookie-R (and Cookie-I), chosen SA, KE-R, N-R, ID-R, and authentication payload (e.g., HASH-R or Sig-R).
• Message 3: Initiator sends authentication payload (e.g., HASH-I or Sig-I).
• Key calculation and authentication happen similarly, but timings differ. Responder can authenticate after message 1 (if using signatures/public keys) or message 3 (if using preshared keys). Initiator authenticates after message 2.

Aggressive Mode Methods (Figures 18.24-18.27)

• Preshared-Key (Fig 18.24): Similar flow, HASH-R in Msg 2, HASH-I (encrypted) in Msg 3.

• Original Public-Key (Fig 18.25): Nonces/IDs encrypted with public keys in Msgs 1 & 2. HASH-R in Msg 2, HASH-I (encrypted) in Msg 3.

• Revised Public-Key (Fig 18.26): Nonces/IDs encrypted with public keys, temporary keys derived. Certs exchanged. HASH-R in Msg 2, HASH-I (encrypted) in Msg 3.

• Digital Signature (Fig 18.27): Sig-R in Msg 2, Sig-I (encrypted) in Msg 3.

Phase II: Quick Mode (3 Messages)

• Purpose: Negotiate SAs for data protocols (like IPSec AH/ESP).

• Protection: Runs under the protection of the Phase I IKE SA (uses $SKEYID\_d$ and $SKEYID\_e$ from Phase I).

• Initiator: Either party from Phase I can initiate Phase II.

• Messages (Figure 18.28): All messages are encrypted using $SKEYID\_e$ from Phase I.

  1. Initiator → Responder: $HASH1$, SA proposal (for IPSec), $N-I$, [optional $KE-I$ for PFS ], [optional $ID-I^{\prime },ID-R^{\prime }$ for selectors].
  2. Responder → Initiator: $HASH2$, chosen SA, $N-R$, [optional $KE-R$ for PFS], [optional $ID-I^{\prime },ID-R^{\prime }$].
  3. Initiator → Responder: $HASH3$.

• Authentication: Uses keyed HMACs derived from Phase I keys: Quick Mode Hash 1: $HASH1=prf(SKEYID\_d,MsgID|SA|N\text{-}I)$ Quick Mode Hash 2: $HASH2=prf(SKEYID\_d,MsgID|SA|N\text{-}R)$ Quick Mode Hash 3: $HASH3=prf(SKEYID\_d,0|MsgID|SA|N\text{-}I|N\text{-}R)$

  • MsgID is a unique identifier for this Quick Mode exchange (from ISAKMP header), prevents collisions.

• Perfect Forward Security (PFS) - Optional:

  • If PFS is desired, an additional Diffie-Hellman exchange ($KE-I,KE-R$) occurs during Quick Mode.
  • The resulting new shared secret $g^{i^{\prime }{r}^{\prime }}\mathrm{mod}p$ is mixed into the key derivation for the IPSec keys.
  • Ensures that compromising the Phase I long-term keys ($SKEYID\_d$) does not compromise the keys for IPSec SAs negotiated with PFS enabled.
  • Requires deleting the temporary DH key ($g^{i^{\prime }{r}^{\prime }}$) immediately after use.

Key Materials (for IPSec SA)

• After Phase II Quick Mode completes, the key material $K$ for the new IPSec SA is derived:
  • Without PFS: $K=prf(SKEYID\_d,\text{protocol}|SPI|N-I|N-R)$
  • With PFS: $K=prf(SKEYID\_d,g^{i^{\prime }{r}^{\prime }}\mathrm{mod}p|\text{protocol}|SPI|N-I|N-R)$
  • protocol = AH or ESP, SPI = SPI for the new IPSec SA, $N-I,N-R$ are nonces from Quick Mode.
• Unidirectional: Key derivation is done independently by both peers. Since the SPI value will differ for each direction of the IPSec SA, the resulting keys $K$ will be different for inbound and outbound traffic.
• Key Length: If the required key length for the chosen cipher is longer than the output of prf, the prf is applied iteratively:
  • $K_1=prf(\text{key},\text{data})$
  • $K_2=prf(\text{key},K_1|\text{data})$
  • $K_3=prf(\text{key},K_2|\text{data})$
  • …
  • $K=K_1|K_2|K_3|...$

SA Algorithms (Negotiated in IKE)

• Diffie-Hellman Groups (Table 18.3): Defines the prime $p$ and generator $g$ (or elliptic curve parameters) for the DH exchange.
  • Group 1: $768$-bit modulus
  • Group 2: $1024$-bit modulus
  • Group 5: $1680$-bit modulus
  • Group 3, 4: Elliptic Curve groups ($155$-bit, $185$-bit fields)
• Hash Algorithms (Table 18.4): Used for authentication (PRF often based on these).
  • MD5 (Value $1$)
  • SHA-1 (Value $2$)
  • Tiger (Value $3$)
  • SHA2-256, SHA2-384, SHA2-512 (Values $4,5,6$)
• Encryption Algorithms (Table 18.5): Used for confidentiality (in ESP and IKE Phase 1 messages 5&6). Typically used in CBC mode.
  • DES (Value $1$)
  • IDEA (Value $2$)
  • Blowfish (Value $3$)
  • RC5 (Value $4$)
  • 3DES (Value $5$)
  • CAST (Value $6$)
  • AES (Value $7$)

18.6 ISAKMP (Internet Security Association and Key Management Protocol)

• Purpose: Defines the framework and packet formats for negotiating, establishing, modifying, and deleting SAs. It carries the IKE messages.

ISAKMP General Header

• Initiator Cookie ($32$ bits): Unique value generated by the initiator of the SA establishment/notification/deletion. Helps identify the IKE SA state.
• Responder Cookie ($32$ bits): Unique value generated by the responder. Value is $0$ in the first message from the initiator.
• Next Payload ($8$ bits): Identifies the type of the first payload immediately following this header (See Table 18.6).
• Major Version ($4$ bits): Current value is $1$.
• Minor Version ($4$ bits): Current value is $0$.
• Exchange Type ($8$ bits): Defines the type of IKE exchange being performed (e.g., Main Mode, Aggressive Mode, Quick Mode).
• Flags ($8$ bits): Indicate options.
  • Encryption bit (E): If set ($1$), payloads following the header are encrypted.
  • Commitment bit (C): If set ($1$), indicates notification about encryption material not being received before SA establishment.
  • Authentication bit (A): If set ($1$), indicates the rest of the payload is authenticated for integrity (but not encrypted).
• Message ID ($32$ bits): Unique ID used to identify messages within Phase II Quick Mode exchanges. Set to $0$ during Phase I.
• Message Length ($32$ bits): Total length of the ISAKMP message (header + all payloads) in bytes.

ISAKMP Payloads

• Messages are constructed by chaining payloads after the general header.
• Generic Payload Header (Figure 18.30): Precedes each specific payload type.
  • Next Payload ($8$ bits): Identifies the type of the next payload in the chain. Value $0$ means this is the last payload.
  • Reserved ($8$ bits): Must be $0$.
  • Payload Length ($16$ bits): Length of the current payload, including its generic header, in bytes.
• Payload Types (Table 18.6):

Type	Name	Brief Description
$0$	None	Marks end of payloads.
$1$	SA	Starts negotiation; contains proposals/transforms.
$2$	Proposal	Contains information for one protocol during negotiation.
$3$	Transform	Defines specific security attributes/algorithms.
$4$	Key Exchange (KE)	Carries data for key generation (e.g., DH half-key).
$5$	Identification (ID)	Carries peer identification information.
$6$	Certificate (CERT)	Carries a public-key certificate.
$7$	Certificate Req (CR)	Requests a certificate from the peer.
$8$	Hash (HASH)	Carries hash output for authentication/integrity.
$9$	Signature (SIG)	Carries digital signature output.
$10$	Nonce (NONCE)	Carries random data for replay protection/liveliness.
$11$	Notification (NOTIFY)	Carries error or status information.
$12$	Delete (DELETE)	Indicates specific SAs have been deleted by the sender.
$13$	Vendor ID (VENDOR)	Defines vendor-specific extensions.

Specific Payload Formats (Selected)

• SA Payload (Figure 18.31):

  • Generic Header
  • Domain of Interpretation (DOI) ($32$ bits): Specifies context (e.g., $1$ for IPSec).
  • Situation (variable length): Defines the context/situation for the negotiation.
  • Followed by one or more Proposal payloads.

• Proposal Payload (Figure 18.32):

  • Generic Header
  • Proposal # ($8$ bits): Number assigned by initiator.
  • Protocol ID ($8$ bits): Protocol for this proposal (e.g., $1$=ISAKMP, $2$=AH, $3$=ESP).
  • SPI Size ($8$ bits): Size of the SPI for this protocol (bytes).
  • Number of Transforms ($8$ bits): How many Transform payloads follow for this proposal.
  • SPI (variable length): The actual Security Parameter Index.
  • Followed by one or more Transform payloads.

• Transform Payload (Figure 18.33):

  • Generic Header
  • Transform # ($8$ bits): Number for this transform within the proposal.
  • Transform ID ($8$ bits): Identifies the specific transform (e.g., encryption algorithm, hash algorithm).
  • Reserved ($16$ bits): Must be $0$.
  • SA Attributes (variable length): Carries specific attributes (e.g., key length, algorithm mode) as Type/Length/Value triplets. Can be short form (Type/Value) or long form (Type/Length/Value).

• Key Exchange Payload (Figure 18.34):

  • Generic Header
  • KE Data (variable length): Data needed for key creation (e.g., Diffie-Hellman public value $g^i\mathrm{mod}p$).

• Identification Payload (Figure 18.35):

  • Generic Header
  • ID Type ($8$ bits): Type of identifier (e.g., IP address, FQDN). DOI specific.
  • ID Data ($24$ bits): Usually $0$.
  • Identification Data (variable length): The actual identity value.

• Certificate Payload (Figure 18.36):

  • Generic Header
  • Certificate Encoding ($8$ bits): Type of certificate
  • Certificate Data (variable length): The actual certificate content.

• Hash Payload (Figure 18.38):

  • Generic Header
  • Hash Data (variable length): The calculated hash value.

• Signature Payload (Figure 18.39):

  • Generic Header
  • Signature Data (variable length): The calculated digital signature value.

• Nonce Payload (Figure 18.40):

  • Generic Header
  • Nonce Data (variable length): The random nonce value.

• Notification Payload (Figure 18.41):

  • Generic Header
  • DOI ($32$ bits): Domain of Interpretation.
  • Protocol ID ($8$ bits): Protocol involved.
  • SPI Size ($8$ bits): Size of SPI.
  • Notify Message Type ($16$ bits): Code indicating status or error type (See Table 18.8 for errors, 18.9 for status).
  • SPI (variable length): Associated SPI.
  • Notification Data (variable length): Optional extra info.

• Delete Payload (Figure 18.42):

  • Generic Header
  • DOI ($32$ bits): Domain of Interpretation.
  • Protocol ID ($8$ bits): Protocol of SAs being deleted.
  • SPI Size ($8$ bits): Size of SPIs.
  • Number of SPIs ($16$ bits): How many SPIs follow.
  • SPIs (variable length): List of SPIs for the SAs that have been deleted.

• Vendor ID Payload (Figure 18.43):

  • Generic Header
  • Vendor ID (variable length): Constant identifying the vendor.