IPsec Overhead Calculator
Calculate a safe IPsec inner MTU and TCP MSS from the ESP profile and path budget, including NAT-T, padding and optional reserve bytes.{{ summaryTitle }} {{ summaryValue }} {{ summaryLine }} {{ badge.label }}{{ badge.value }}
| Measure | Value | Decision use | Copy |
|---|---|---|---|
| {{ row.measure }} | {{ row.value }} | {{ row.note }} |
| Component | Bytes | Scope | Copy |
|---|---|---|---|
| {{ row.component }} | {{ row.bytes }} | {{ row.scope }} |
Recommended starting point
{{ formatInteger(model.recommended_inner_mtu_bytes) }} B inner MTU and {{ formatInteger(model.tcp_mss_clamp_bytes) }} B conservative TCP clamp. Verify the negotiated transform and packet path before production rollout.
- Selected transform
- {{ selectedProfile.label }}
- Outer packet
- {{ formatInteger(model.outer_packet_bytes) }} B of {{ formatInteger(normalizedInputs.underlay_mtu) }} B
- IPv6 floor
- {{ floorText }}
Calculation record
{{ formulaText }}
- Allow Path MTU Discovery feedback where policy permits; the TCP clamp does not resize UDP, QUIC, or other non-TCP traffic.
- Do not add an access or carrier encapsulation twice when the entered underlay MTU already reflects it.
- Capture the actual proposal and platform-specific transform details in the change record.
A VPN can pass small packets and still fail under ordinary application traffic. File transfers, TLS sessions, database connections, and large responses may stall when the original packet plus IPsec encapsulation exceeds the maximum transmission unit (MTU) of the outside path.
IPsec tunnel mode protects a complete inner IP packet and carries it inside a new outer IP packet. Encapsulating Security Payload (ESP) adds its own header, initialization vector or nonce, padding, trailer, and integrity check or authentication tag. NAT traversal adds a UDP header, while PPPoE, VLAN, GRE, IP-in-IP, or another outer encapsulation may consume more of the path budget.
| Quantity | Meaning | Planning use |
|---|---|---|
| Underlay MTU | Maximum packet size accepted by the outside path between VPN peers. | Sets the total byte budget. |
| Inner MTU | Largest protected IP packet that fits after the selected overhead and reserve. | Configures the tunnel interface or routed path. |
| TCP MSS | Largest planned TCP payload after inner IP, TCP, and option-reserve bytes are removed. | Provides a conservative clamp starting point. |
| On-wire overhead | Outer IP, NAT-T, path extras, and transmitted ESP bytes at the fitting packet size. | Explains where the byte budget went. |
The negotiated transform matters because transmitted nonce, tag, and block-alignment lengths differ. AES-GCM and ChaCha20-Poly1305 have different profiles from AES-CBC with HMAC. Padding also depends on the candidate inner packet size, so a block cipher does not always reduce MTU by one fixed overhead number.
Outside-path accounting needs one consistent boundary. If a measured 1492-byte MTU already reflects PPPoE, subtracting PPPoE again produces an unnecessarily small inner MTU. Conversely, entering 1500 while omitting a known carrier or nested-tunnel header can produce packets that fragment or disappear.
A TCP MSS clamp helps TCP endpoints avoid oversize segments, but it does not solve MTU problems for UDP, QUIC, ICMP, or other traffic. Path MTU discovery and correctly handled ICMP or ICMPv6 errors remain important, and protected IPv6 traffic needs special attention when the inner MTU falls below 1280 bytes.
How to Use This Tool:
Start from the smallest verified outside-path MTU and add only encapsulation that is not already reflected in that number.
- Enter the Underlay MTU. Use a measured lower path value when available rather than assuming every link accepts 1500 bytes.
- Choose the negotiated ESP profile and the outer and inner IP versions. Use the custom profile only when the transmitted IV or nonce, ICV or tag, and alignment lengths are known.
- Select an Outside path extra and NAT traversal only when they apply outside the entered MTU boundary. Do not count PPPoE, VLAN, GRE, or IP-in-IP twice.
- Add a safety reserve, TCP option reserve, traffic-flow-confidentiality padding, or additional path bytes only for a known requirement. Safety reserve changes the inner MTU; TCP option reserve changes only the MSS.
- Review Tunnel fit and the byte ledger, then compare profiles. Stop when IPv6 falls below its 1280-byte floor or the result leaves no TCP payload room, and verify the negotiated tunnel with packet capture or platform diagnostics before rollout.
Interpreting Results:
Recommended inner MTU is the largest tested inner packet that fits within the underlay MTU after the safety reserve. TCP MSS clamp subtracts the selected inner IP header, the 20-byte base TCP header, and any TCP option reserve from that inner MTU.
- Fits path means the modeled packet stays within budget, the TCP payload remains positive, and any IPv6 inner packet clears 1280 bytes.
- IPv6 floor missed means the fitted inner MTU is below 1280 bytes. Treat that as a deployment warning rather than silently using the smaller value.
- No TCP room means headers and the option reserve consume the entire fitted inner packet budget.
- Total MTU reduction may exceed on-wire overhead because it also includes the safety reserve and any unused alignment headroom.
- Packet expansion is modeled on-wire overhead divided by the fitted inner MTU. It describes the selected packet size, not average tunnel bandwidth efficiency.
Technical Details:
ESP tunnel sizing is a constrained byte-budget problem. A candidate inner MTU must leave room for every transmitted outer and ESP field, while the ESP padding count must satisfy both the transform block size and the four-byte alignment boundary that precedes the integrity data.
Formula Core
The fitted value is the largest nonnegative candidate whose complete outer packet does not exceed the safety-adjusted budget.
Outer IPv4 and IPv6 headers are budgeted at 20 and 40 bytes. NAT-T contributes 8 bytes when enabled. The fixed ESP header is 8 bytes for the Security Parameters Index and sequence number, and the trailer contributes 2 bytes for Pad Length and Next Header. Profile-specific IV or nonce and ICV or tag bytes complete the fixed portion.
Mechanism Core
For each candidate, plaintext length is the inner MTU plus optional TFC padding plus the two ESP trailer bytes. The smallest padding from 0 through 255 is accepted only when the encrypted plaintext is divisible by the profile block size and the IV plus plaintext is divisible by four. Candidates are reduced until the complete packet fits.
| ESP profile | IV or nonce | ICV or tag | Block alignment |
|---|---|---|---|
| AES-GCM, 16-byte tag | 8 B | 16 B | 1 B |
| AES-GCM, 12-byte tag | 8 B | 12 B | 1 B |
| AES-CBC with HMAC-SHA1-96 | 16 B | 12 B | 16 B |
| AES-CBC with HMAC-SHA-256-128 | 16 B | 16 B | 16 B |
| ChaCha20-Poly1305, 16-byte tag | 8 B | 16 B | 1 B |
| NULL encryption with HMAC-SHA1-96 | 0 B | 12 B | 1 B |
Outside-path presets add 4 bytes for a VLAN tag, 8 for PPPoE, 12 for PPPoE plus VLAN, 24 for GRE over IPv4, or 20 for IP-in-IP over IPv4. Additional path bytes are added to the selected preset. The underlay input accepts whole values from 576 through 65,535 bytes; custom profile fields and all reserves also use whole bytes.
Limitations:
The result models the declared packet envelope. It does not inspect a security association, discover the live path MTU, or confirm how a vendor applies every negotiated option.
- Extended sequence numbers, unusual tag truncation, nested tunnels, carrier encapsulation, and platform-specific headers may require a custom allowance.
- TCP option reserve is a planning deduction, not another transmitted header in the ESP ledger.
- MSS clamping affects TCP only. PMTUD behavior and ICMP filtering must be tested for other traffic.
- A fitted size does not prove that intermediate devices, firewalls, or cloud gateways accept the same packet size.
Worked Examples:
AES-GCM with NAT-T on a 1500-byte IPv4 path
An IPv4 outer header contributes 20 bytes, NAT-T adds 8, the ESP header adds 8, the transmitted nonce adds 8, the trailer adds 2, and the authentication tag adds 16. With no path extras, TFC padding, or reserve, the 62-byte envelope leaves a 1438-byte inner MTU. An IPv4 inner header and 20-byte TCP header reduce the conservative MSS to 1398 bytes.
References:
- RFC 4303: IP Encapsulating Security Payload, RFC Editor, December 2005.
- RFC 3948: UDP Encapsulation of IPsec ESP Packets, RFC Editor, January 2005.
- RFC 4106: The Use of Galois/Counter Mode in IPsec ESP, RFC Editor, June 2005.
- RFC 3602: The AES-CBC Cipher Algorithm and Its Use with IPsec, RFC Editor, September 2003.
- RFC 7634: ChaCha20, Poly1305, and Their Use in IPsec, RFC Editor, August 2015.
- RFC 8200: Internet Protocol, Version 6 Specification, RFC Editor, July 2017.
- RFC 8201: Path MTU Discovery for IPv6, RFC Editor, July 2017.