Okay, here are the concise study notes for Chapter 18: Security at the Network Layer: IPSec, formatted for exam preparation and quick review using MathJax for all mathematical notation.

Security at the Network Layer: IPSec

Objectives

Define IPSec architecture.
Discuss IPSec transport and tunnel modes.
Understand how IPSec provides authentication only (AH).
Understand how IPSec provides confidentiality and authentication (ESP).
Define Security Association (SA) and its implementation.
Define Internet Key Exchange (IKE) and its use by IPSec.

Introduction

Security at application and transport layers may be insufficient because:
- Not all applications are protected (e.g., PGP/S/MIME only for email).
- Not all applications use TCP (which can use SSL/TLS); some use UDP.
- Some applications (e.g., routing protocols) use IP directly.
IP Security (IPSec): A collection of protocols designed by IETF to provide security (authentication, confidentiality) for packets at the network (IP) layer.

18.1 TWO MODES

IPSec operates in two modes:

Transport Mode

Protection: Protects the payload delivered from the transport layer (TCP/UDP payload). It protects the IP payload.
IP Header: Does not protect the original IP header.
Mechanism: IPSec header/trailer are inserted between the original IP header and the transport layer payload. The original IP header is largely untouched (protocol field changed).
Usage: Typically used for host-to-host (end-to-end) protection.
Diagram Concept (Fig 18.2): IP-H | [IPSec-H | Transport Layer Payload | IPSec-T]

Tunnel Mode

Protection: Protects the entire original IP packet, including the original IP header.
Mechanism: The entire original IP packet is encapsulated with an IPSec header/trailer. A new IP header is added outside the IPSec protection.
Usage: Typically used when at least one endpoint is a gateway/router (e.g., router-to-router, host-to-router, router-to-host). Creates a “tunnel”.
Diagram Concept (Fig 18.4): New IP-H | [IPSec-H | Original IP-H | Original Payload | IPSec-T]

Comparison

Transport Mode: Protects payload only. Useful for end-to-end security between hosts. Layer flow: Transport → IPSec → Network.
Tunnel Mode: Protects entire original IP packet. Useful for VPNs, gateway protection. Layer flow: Network → IPSec → Network (new header).

18.2 TWO SECURITY PROTOCOLS

IPSec defines two main protocols:

Authentication Header (AH)

Purpose: Provides source authentication and data integrity for IP packets.
Mechanism: Uses a hash function and a shared secret key to create a message digest (Authentication Data).
Placement: AH header inserted into the IP packet (position depends on mode).
IP Protocol Field: Set to $51$ .
Header Fields (Fig 18.7):
- Next Header: 8 bits, indicates the type of payload following AH (original protocol type).
- Payload Length: 8 bits, length of AH header in 4-byte words, minus 2 (doesn’t include first 8 bytes).
- Reserved: 16 bits.
- Security Parameter Index (SPI): 32 bits, identifies the Security Association (SA).
- Sequence Number: 32 bits, anti-replay counter.
- Authentication Data: Variable length, contains the Integrity Check Value (ICV) or digest. Calculated over the entire IP packet (except mutable fields like TTL, checksum).
Limitation: Does NOT provide confidentiality (privacy).

Encapsulating Security Payload (ESP)

Purpose: Provides source authentication, data integrity, AND confidentiality (privacy).
Mechanism: Adds an ESP header and an ESP trailer. Payload and trailer are typically encrypted. Authentication covers ESP header, payload, and trailer.
IP Protocol Field: Set to $50$ .
Placement: ESP header before payload, ESP trailer after payload. Authentication data appended after trailer.
Header & Trailer Fields (Fig 18.8):
- ESP Header:
  - Security Parameter Index (SPI): 32 bits.
  - Sequence Number: 32 bits.
- ESP Trailer:
  - Padding: 0-255 bytes, used for block cipher alignment and traffic flow confidentiality.
  - Pad Length: 8 bits, number of padding bytes.
  - Next Header: 8 bits, indicates the type of payload that was encrypted.
- Authentication Data: Variable length, appended at the very end. Calculated over ESP header, encrypted payload, and ESP trailer (does not include IP header).
Advantage: Provides all services including privacy.

IPv4 and IPv6

IPSec supports both. In IPv6, AH and ESP are implemented as extension headers.

AH versus ESP

ESP can do everything AH does, plus confidentiality.
AH remains primarily due to existing implementations. ESP is generally preferred for new deployments needing confidentiality.

Services Provided by IPSec

Service	AH	ESP	How Provided
Access control	yes	yes	Indirectly via SAD lookup (drop if no SA)
Message integrity	yes	yes	Hash digest (Authentication Data field)
Entity authentication	yes	yes	SA establishment + keyed hash digest
*Confidentiality-	no	yes	Encryption of payload (ESP only)
Replay attack protection	yes	yes	Sequence numbers and receiver sliding window

Replay Attack Protection Details

Uses a 32-bit Sequence Number (starts at $0$ , increments, no wrap-around within one SA).
Receiver maintains a sliding window of fixed size $W$ (default $64$ ). Window spans $[N, N + W - 1]$ , where $N$ is the lowest sequence number accepted.
Packet Arrival:
1. Seq Num < N$: Discard (too old or duplicate).
2. Seq Num in $[N, N + W - 1]$ : If authenticated and new (not marked as received), mark as received, accept. Else discard.
3. Seq Num > N+W-1 $: I f a u t h e n t i c a t e d, ma r ka srece i v e d, s l i d e w in d o w ba se$ N$ up to (received Seq Num - W + 1), accept. Else discard.

18.3 SECURITY ASSOCIATION

Idea of Security Association (SA)

Definition: A Security Association (SA) is a unidirectional logical connection (a “contract”) between two communicating entities that describes how they will use security services (algorithms, keys, modes) for a particular traffic flow.
Unidirectional: For bidirectional communication, two SAs are needed (one inbound, one outbound).
Content: Contains parameters like keys, algorithms, sequence numbers, lifetimes, mode, etc. (Fig 18.10).

Security Association Database (SAD)

Definition: A database (SAD) that stores all active SAs on a host or gateway.
Structure: Usually maintains separate inbound and outbound SADs. Each row represents one SA.
Lookup:
- Outbound: When sending, IPSec consults the outbound SAD to find the correct SA based on destination, protocol etc., to apply security.
- Inbound: When receiving, IPSec uses information from the packet (SPI, destination address, protocol) to look up the corresponding SA in the inbound SAD to verify security.
Inbound SA Index: Uniquely identified by a triple:
1. Security Parameter Index (SPI): 32-bit value in AH/ESP header.
2. Destination IP Address: Receiver’s IP address.
3. Protocol: AH ( $51$ ) or ESP ( $50$ ).
SA Parameters (Table 18.2): Key parameters stored within an SA entry include:
- Sequence Number Counter
- Sequence Number Overflow flag
- Anti-Replay Window state
- AH/ESP algorithm info (authentication, encryption)
- Keys (authentication, encryption)
- Key Lifetimes
- SA Lifetime
- IPSec Mode (Transport/Tunnel)
- Path MTU

18.4 SECURITY POLICY

Security Policy Database (SPD)

Definition: A database (SPD) on each IPSec host/gateway that defines what security (if any) should be applied to IP traffic. Consulted before the SAD.
Structure: Separate inbound and outbound SPDs. Entries define policies for specific traffic flows.
Index (Selector): Each entry is indexed by a set of fields derived from the packet, often a sextuple:
- Source IP Address
- Destination IP Address
- Name (e.g., DNS)
- Protocol (from transport layer or higher)
- Source Port
- Destination Port
- (Addresses can be specific, ranges, or wildcards).

Outbound SPD Processing (Fig 18.13)

Packet details form the index to query the outbound SPD.
SPD lookup results in one of three actions:
- Drop: Discard the packet. Security policy forbids transmission.
- Bypass: Send the packet without applying IPSec. No security needed for this traffic.
- Apply: Apply IPSec security.
  - The SPD entry points to an SA (or indicates one is needed).
  - If a suitable SA exists in the SAD: Retrieve parameters, apply AH/ESP, send.
  - If no suitable SA exists: Trigger Internet Key Exchange (IKE) to negotiate SAs with the peer. Once SAs are created and stored in SAD, apply security and send.

Inbound SPD Processing (Fig 18.14)

Packet details form the index to query the inbound SPD.
SPD lookup results in one of three actions:
- Discard: Drop the packet. Policy requires IPSec protection, but packet arrived without it (or with wrong type).
- Bypass: Process the packet normally, ignoring any IPSec headers. Policy indicates no security required.
- Apply: Packet should have IPSec protection.
  - Use SPI, destination addr, protocol from the packet to look up SA in the inbound SAD.
  - If SA exists: Verify security (authentication, decryption). If valid, strip IPSec headers, pass payload up. If invalid, discard.
  - If no SA exists: Discard packet (required SA is missing).

18.5 INTERNET KEY EXCHANGE (IKE)

Purpose: A protocol used to dynamically negotiate and establish Security Associations (SAs) and cryptographic keys for IPSec. IKE creates SAs for IPSec.
Trigger: Called when the SPD requires IPSec (‘Apply’) but no suitable SA exists in the SAD.
Complexity: Based on three other protocols (Fig 18.15):
- Oakley: Key exchange protocol framework (based on Diffie-Hellman).
- SKEME: Key exchange techniques, providing features like public-key based authentication, nonces.
- ISAKMP (Internet Security Association and Key Management Protocol): Defines procedures and packet formats for negotiating, establishing, modifying, and deleting SAs. Provides the framework/carrier for IKE messages.

Improved Diffie-Hellman Key Exchange

IKE uses Diffie-Hellman (DH) but enhances it to address weaknesses:

Basic DH: Exchange $K E - I = g^{i} (mod p)$ and $K E - R = g^{r} (mod p)$ to compute shared key $K = g^{i r} (mod p)$ .
Clogging Attack (DoS): Malicious actor floods responder with DH requests, forcing expensive modular exponentiation ( $g^{r} (mod p)$ , $K = (g^{i})^{r} (mod p)$ ).
- Mitigation: Cookies. Parties exchange lightweight cookies (hash of IP, port, secret, timestamp) in initial messages. Heavy computation ( $g^{r}, K$ ) is postponed until cookies are exchanged and validated, proving the initiator is reachable. IKE uses cookies.
Replay Attack: Attacker replays messages from a previous session.
- Mitigation: Nonces (random numbers). Included in messages to ensure freshness. IKE uses nonces.
Man-in-the-Middle (MitM) Attack: Attacker intercepts DH exchange, establishes separate keys with each party.
- Mitigation: Authentication. Parties must prove their identity by demonstrating knowledge of a secret. IKE requires authentication.

Authentication Secrets in IKE

Authentication can be based on:

Preshared Secret Key: Both parties possess a shared symmetric key.
Public Key Encryption: Parties possess public/private key pairs; demonstrate identity by encrypting/decrypting with them.
Digital Signatures: Parties possess public/private key pairs; demonstrate identity by signing messages.

IKE Phases

IKE operates in two phases to manage SA creation:

Phase I: Establishes an IKE SA. This SA is secure (authenticated and optionally encrypted) and is used to protect Phase II negotiations. Phase I is generic.
- Modes:
  - Main Mode: 6 messages. Provides identity protection for peers.
  - Aggressive Mode: 3 messages. Faster, but exposes peer identities.
Phase II: Establishes SAs for other protocols, typically IPSec SAs. Uses the IKE SA created in Phase I for protection. Phase II is specific to the protocol needing the SA (e.g., IPSec AH/ESP).
- Mode:
  - Quick Mode: 3 messages.

Phase I Authentication Methods

Main Mode and Aggressive Mode can use one of four methods (Fig 18.19):

Preshared secret key
Original public-key (encryption)
Revised public-key (encryption, optimized)
Digital signature

Phase I: Main Mode Details (6 messages)

Messages 1 & 2: Initiator proposes SA parameters (encryption, hash, auth method, DH group). Responder selects parameters, sends cookie. Initiator sends cookie. Protects against clogging.
Messages 3 & 4: Exchange DH public values ( $g^{i}, g^{r}$ ) and nonces ( $N - I, N - R$ ). Establishes shared secret, ensures freshness.
Messages 5 & 6: Exchange identity information and authentication proof (hashes/signatures), usually encrypted using keys derived from the DH exchange. Authenticates peers.

Key Derivation:
- A base key SKEYID is derived, depending on the auth method:
  - Preshared: $S K E Y I D = p r f (preshared-key, N - I ∣ N - R)$
  - Public-key: $S K E Y I D = p r f (N - I ∣ N - R, g^{i r})$
  - Digital Sig: $S K E Y I D = p r f (hash (N - I ∣ N - R), Cookie-I ∣ Cookie-R)$
- Further keys are derived from SKEYID using prf (Pseudo-Random Function, often HMAC):
  - $S K E Y I D_d$ : For deriving Phase II keys. $S K E Y I D_d = p r f (S K E Y I D, g^{i r} ∣ C K Y - I ∣ C K Y - R ∣0)$
  - $S K E Y I D_a$ : For authenticating Phase I messages. $S K E Y I D_a = p r f (S K E Y I D, S K E Y I D_d ∣ g^{i r} ∣ C K Y - I ∣ C K Y - R ∣1)$
  - $S K E Y I D_e$ : For encrypting Phase I messages. $S K E Y I D_e = p r f (S K E Y I D, S K E Y I D_a ∣ g^{i r} ∣ C K Y - I ∣ C K Y - R ∣2)$ (Note: $C K Y - I, C K Y - R$ are cookies; $g^{i r}$ is the DH shared secret)
Authentication Hashes:
- $H A S H - I = p r f (S K E Y I D, K E - I ∣ K E - R ∣ C K Y - I ∣ C K Y - R ∣ S A - I ∣ I D - I)$
- $H A S H - R = p r f (S K E Y I D, K E - R ∣ K E - I ∣ C K Y - R ∣ C K Y - I ∣ S A - I ∣ I D - R)$ (Note: $S A - I$ is the initiator’s SA proposal payload; $I D - I, I D - R$ are identities)
- HASH-I sent in msg 5, HASH-R sent in msg 6. Encrypted with $S K E Y I D_e$ .

Phase I: Aggressive Mode Details (3 messages)

Combines messages: Msgs 1+3 → Msg 1; Msgs 2+4+6 → Msg 2; Msg 5 → Msg 3.
Faster negotiation.
Does not provide identity protection in early messages.
Calculations of keys and hashes are similar, but inputs are available at different times.

Phase II: Quick Mode Details (3 messages)

Purpose: Negotiate IPSec SAs (or SAs for other protocols).
Protection: Protected by the IKE SA established in Phase I (using $S K E Y I D_d, S K E Y I D_e, S K E Y I D_a$ ).
Messages (Fig 18.28):
1. Msg 1: HDR, HASH(1), SA, Ni, [KEi], [IDi, IDr]
2. Msg 2: HDR, HASH(2), SA, Nr, [KEr], [IDi, IDr]
3. Msg 3: HDR, HASH(3)
- HDR: ISAKMP header with Phase II message ID.
- HASH(x): Provides authentication for Phase II messages. Derived using $S K E Y I D_d$ .
  - $H A S H 1 = p r f (S K E Y I D_d, M s g I D ∣ S A ∣ N - I ∣ [K E - I] ∣ [I D - I ∣ I D - R])$
  - $H A S H 2 = p r f (S K E Y I D_d, M s g I D ∣ N - I ∣ S A ∣ N - R ∣ [K E - R] ∣ [I D - I ∣ I D - R])$
  - $H A S H 3 = p r f (S K E Y I D_d, 0∣ M s g I D ∣ N - I ∣ N - R)$ (Note: Simplified formulas from text used here for consistency, actual calculation involves all payloads)
- SA: Proposed IPSec SA parameters (AH/ESP, algorithms, mode).
- Ni, Nr: Fresh nonces for this Phase II exchange (replay protection, key generation).
- [KEi], [KEr]: Optional DH public values for Perfect Forward Secrecy (PFS).
- [IDi], [IDr]: Optional identities (e.g., for selecting policy if traffic selectors needed).
All messages are encrypted using $S K E Y I D_e$ .

Perfect Forward Secrecy (PFS)

Concept: Ensures that compromising the Phase I key ( $S K E Y I D_d$ ) does not compromise keys generated for IPSec SAs in Phase II.
Mechanism: If PFS is enabled (optional), Quick Mode includes an additional DH exchange ([KEi], [KEr]). The resulting new DH shared secret ( $g^{i r^{'}}$ ) is used in the derivation of the IPSec key material $K$ . The DH private keys ( $i^{'}, r^{'}$ ) are discarded immediately after use.

Key Materials for IPSec SA

The final output of Phase II is key material $K$ used by IPSec (e.g., for ESP encryption/authentication).
Derivation depends on PFS:
- Without PFS: $K = p r f (S K E Y I D_d, protocol ∣ SP I ∣ N - I ∣ N - R)$
- With PFS: $K = p r f (S K E Y I D_d, g^{i r^{'}} ∣ protocol ∣ SP I ∣ N - I ∣ N - R)$
If $K$ is too short, additional keys $K_{1}, K_{2}, ...$ are derived iteratively and concatenated:
- $K_{1} = p r f (key, data)$
- $K_{2} = p r f (key, K_{1} ∣ data)$
- $K_{3} = p r f (key, K_{2} ∣ data)$ … where key is $S K E Y I D_d$ or $S K E Y I D_d ∣ g^{i r^{'}}$ and data is protocol | SPI | N-I | N-R.
Key material is unidirectional.

SA Algorithms (Negotiated in IKE Phase I)

Diffie-Hellman Groups (Table 18.3): Determine the strength of the DH exchange (e.g., Group 1: 768-bit, Group 2: 1024-bit, Group 5: 1680-bit modulus; Group 14: 2048-bit; ECC groups also exist).
Hash Algorithms (Table 18.4): For authentication/PRF (e.g., MD5, SHA-1, SHA2-256, SHA2-384, SHA2-512).
Encryption Algorithms (Table 18.5): For confidentiality in ESP and IKE (e.g., DES, 3DES, AES-128, AES-192, AES-256). Usually used in CBC mode.

18.6 ISAKMP

Purpose: Defines the framework, packet formats, and state machine for negotiating, establishing, and managing SAs. ISAKMP carries IKE messages.

General Header (Fig 18.29)

Initiator Cookie: 64 bits, chosen by initiator.
Responder Cookie: 64 bits, chosen by responder (0 in first message). Cookies identify the IKE SA state.
Next Payload: 8 bits, type of the first payload following the header.
Major Ver / Minor Ver: 4 bits each (Currently $1.0$ ).
Exchange Type: 8 bits (e.g., Main Mode, Aggressive Mode, Quick Mode).
Flags: 8 bits (Encryption, Commit, Authentication).
Message ID: 32 bits, used to sequence messages, especially in Phase II and for notifications/deletions. $0$ in Phase I.
Message Length: 32 bits, total length of message including header and all payloads.

Payloads (Table 18.6)

ISAKMP messages consist of the general header followed by one or more payloads.
Generic Payload Header (Fig 18.30): Each payload starts with:
- Next Payload: 8 bits, type of the next payload (0 if last).
- Reserved: 8 bits.
- Payload Length: 16 bits, length of the current payload including its generic header.
Payload Types:
- SA (1): Initiates negotiation, contains proposals. Defines DOI, Situation.
- Proposal (2): Defines Protocol ID (IKE, AH, ESP), SPI, number of Transforms.
- Transform (3): Specifies a cryptographic algorithm (e.g., AES, SHA1) and its attributes.
- Key Exchange (KE) (4): Carries DH public values or other keying material.
- Identification (ID) (5): Carries identity information (IP address, FQDN, etc.).
- Certificate (CERT) (6): Carries a public key certificate.
- Certificate Request (CR) (7): Requests a certificate.
- Hash (HASH) (8): Carries hash output for authentication.
- Signature (SIG) (9): Carries digital signature output.
- Nonce (NONCE) (10): Carries a random nonce value.
- Notification (NOTIFY) (11): Reports status or error conditions (Table 18.8, 18.9).
- Delete (DELETE) (12): Informs peer that an SA has been deleted.
- Vendor ID (VID) (13): Carries vendor-specific information.

18.9 SUMMARY

IPSec provides network-layer security.
Operates in Transport (payload protection) or Tunnel (full packet protection) mode.
Uses AH (authentication, integrity) or ESP (authentication, integrity, confidentiality).
Requires Security Associations (SAs) stored in SAD, governed by policies in SPD.
IKE is used to negotiate SAs for IPSec.
IKE has Phase I (Main/Aggressive modes, creates IKE SA) and Phase II (Quick mode, creates IPSec SA).
IKE uses cookies (anti-clogging), nonces (anti-replay), and authentication (preshared key, public key crypto, signatures) to secure the exchange.
PFS provides enhanced security for IPSec keys.
ISAKMP defines the message formats and framework carrying IKE negotiations.

Quartz 4

Explorer

ipsec