Types of Firewalls

In order to gain a thorough understanding of firewall technology, it is important to understand the various types of firewalls. These various types of firewalls provide more or less the same functions that were outlined earlier. However, their methods of doing so provide differentiation in terms of performance and level of security offered.

The firewalls discussed in this section are divided into five categories based on the mechanism that each uses to provide firewall functionality:

• Circuit-level firewalls
• Proxy server firewalls
• Nonstateful packet filters
• Stateful packet filters
• Personal firewalls

These various types of firewalls gather different types of information from the data flowing through them to keep track of legitimate and illegitimate traffic and to protect against unauthorized access. The type of information they use often also determines the level of security they provide.

Circuit-Level Firewalls

These firewalls act as relays for TCP connections. They intercept TCP connections being made to a host behind them and complete the handshake on behalf of that host. Only after the connection is established is the traffic allowed to flow to the client. Also, the firewall makes sure that as soon as the connection is established, only data packets belonging to the connection are allowed to go through.

Circuit-level firewalls do not validate the payload or any other information in the packet, so they are fairly fast. These firewalls essentially are interested only in making sure that the TCP handshake is properly completed before a connection is allowed. Consequently, these firewalls do not allow access restrictions to be placed on protocols other than TCP and do not allow the use of payload information in the higher-layer protocols to restrict access.

Proxy Server Firewalls

Proxy server firewalls work by examining packets at the application layer. Essentially a proxy server intercepts the requests being made by the applications sitting behind it and performs the requested functions on behalf of the requesting application. It then forwards the results to the application. In this way it can provide a fairly high level of security to the applications, which do not have to interact directly with outside applications and servers.

Proxy servers are advantageous in the sense that they are aware of application-level protocols and they can restrict or allow access based on these protocols. They also can look into the data portions of the packets and use that information to restrict access. However, this very capability of processing the packets at a higher layer of the stack can contribute to the slowness of proxy servers. Also, because the inbound traffic has to be processed by the proxy server as well as the end-user application, further degradation in speed can occur. Proxy servers often are not transparent to end users who have to make modifications to their applications in order to use the proxy server. For each new application that must go through a proxy firewall, modifications need to be made to the firewall's protocol stack to handle that type of application.

Nonstateful Packet Filters

Nonstateful packet filters are fairly simple devices that sit on the periphery of a network and, based on a set of rules, allow some packets through while blocking others. The decisions are made based on the addressing information contained in network layer protocols such as IP and, in some cases, information contained in transport layer protocols such as TCP or UDP headers as well.

Nonstateful packet filters are fairly simple devices, but to function properly they require a thorough understanding of the usage of services required by a network to be protected. Although these filters can be fast because they do not proxy any traffic but only inspect it as it passes through, they do not have any knowledge of the application-level protocols or the data elements in the packet. Consequently, their usefulness is limited. These filters also do not retain any knowledge of the sessions established through them. Instead, they just keep tabs on what is immediately passing through.. The use of simple and extended access lists (without the established keyword) on routers are examples of such firewalls.

Stateful Packet Filters

Stateful packet filters are more intelligent than simple packet filters in that they can block pretty much all incoming traffic and still can allow return traffic for the traffic generated by machines sitting behind them. They do so by keeping a record of the transport layer connections that are established through them by the hosts behind them.

Stateful packet filters are the mechanism for implementing firewalls in most modern networks. Stateful packet filters can keep track of a variety of information regarding the packets that are traversing them, including the following:

• Source and destination TCP and UDP port numbers
• TCP sequence numbering
• TCP flags
• TCP session state based on the RFCed TCP state machine
• UDP traffic tracking based on timers

Stateful firewalls often have built-in advanced IP layer handling features such as fragment reassembly and clearing or rejecting of IP options.

Many modern stateful packet filters are aware of application layer protocols such as FTP and HTTP and can perform access-control functions based on these protocols' specific needs.

Personal Firewalls

Personal firewalls are firewalls installed on personal computers. They are designed to protect against network attacks. These firewalls are generally aware of the applications running on the machine and allow only connections established by these applications to operate on the machine.

A personal firewall is a useful addition to any PC because it increases the level of security already offered by a network firewall. However, because many of the attacks on today's networks originate from inside the protected network, a PC firewall is an even more useful tool, because network firewalls cannot protect against these attacks. Personal firewalls come in a variety of flavors. Most are implemented to be aware of the applications running on the PC. However, they are designed to not require any changes from the user applications running on the PC, as is required in the case of proxy servers.

Troubleshooting IPsec VPNs

 Because IPsec is a combination of multiple protocols, it is important to have a very strong understanding of how these protocols work together to troubleshoot IPsec. A thorough understanding of ISAKMP is also very useful in identifying negotiation problems in IPsec. This section looks at some of the tools available to troubleshoot IPsec, as well as some of the common issues surrounding its implementation.

IPsec's Order of Events

IPsec's order of events is important to know because IPsec interacts with a wide variety of protocols running on a router. Because IPsec adds an additional header on top of the original header with a changed IP address in the case of ESP tunnel mode, it is important to understand how the various other routines affect this new header. Tablebelow outlines the sequence of events. "Inside" is generally the private network behind the router, and "outside" is the network on the public side of the router.

Table. Order in Which Various Operations Are Performed for Packets Passing Through a Router

Inside to Outside	Outside to Inside
If IPsec, check input access list Decryption for CET or IPsec Check input access list Check input rate limits Input accounting Inspect Policy routing Routing Redirect to web cache NAT inside to outside (local-to-global translation) Crypto (check map and mark for encryption) Check output access list Inspect TCP intercept Encryption for CET or IPsec	If IPsec, check input access list Decryption for CET or IPsec Check input access list Check input rate limits Input accounting Inspect NAT outside to inside (global-to-local translation) Policy routing Routing Redirect to web cache Crypto (check map and mark for encryption) Check output access list Inspect TCP intercept Encryption for CET or IPsec

It is interesting to note that for IPsec traffic, the incoming access list is processed twice-once before decryption and once after decryption. Therefore, not only does ESP traffic need to be allowed through the incoming access list, but a hole needs to be opened for the subnets that are considered interesting traffic for IPsec and are being tunneled. However, because the router drops any packets arriving on its outside interface that match the IPsec interesting traffic access list and are not IPsec-encapsulated, the risk of an attacker's using this hole to gain access to the network behind the router is minimized.

IPsec Debugs

IPsec debug messages are a very important source for understanding any issues that might creep into the implementation of IPsec VPNs. The three most commonly useddebug commands to troubleshoot IPsec are

• debug crypto isakmp
• debug crypto ipsec
• debug crypto engine

The first two debug commands are valid for the PIX Firewall as well. PIX has similar debugs as the ones shown for a router.

Example below shows the configuration of the router on one end of the IPsec tunnel. The router on the other end has a corresponding configuration . This configuration is used to generate the debugs shown here.

Example . IPsec Configuration on a Router Used to Generate Sample Debugs

Router#write terminal



crypto isakmp policy 1

 encr 3des

 authentication pre-share

crypto isakmp key jw4ep9846804ijl address 172.16.172.20

!

crypto ipsec transform-set myset esp-3des esp-md5-hmac

!

crypto map vpn 10 ipsec-isakmp

 set peer 172.16.172.20

 set transform-set myset

 match address 101



interface Ethernet0/0

 ip address 10.1.1.1 255.255.255.0

!

interface Ethernet1/0

 ip address 172.16.172.10 255.255.255.240

 crypto map vpn

!



access-list 101 permit ip 10.1.1.0 0.0.0.255 10.1.2.0 0.0.0.255

Example below shows the debug messages generated by turning on the three debugs just mentioned.

NOTE

In the sample debugs shown in Example below, the explanation of the debug is followed by the actual debugs for that explanation.

Example . IPsec Debugs Generated by Issuing the Commands debug crypto isakmp, debug crypto ipsec, and debug crypto engine

Router#debug crypto ISAKMP

Router#debug crypto engine

Router#debug crypto ipsec



!Ping source and destination addresses matched the address access list for the

!crypto map VPN



   00:04:10: IPSEC(sa_request): ,

   (key eng. msg.) OUTBOUND local= 172.16.172.10, remote= 172.16.172.20,

   local_proxy = 10.1.1.0/255.255.255.0/0/0 (type=4),

   remote_proxy = 10.1.2.0/255.255.255.0/0/0 (type=4),



!The 'local' is the local tunnel endpoint, and the 'remote' is the remote crypto

!endpoint as configured in the map. The local proxy is the src interesting traffic

!as defined by the match address access list. The remote proxy is the destination

!interesting traffic as defined by the match address access list.



         protocol= ESP, transform= esp-3des esp-md5-hmac ,

    lifedur= 3600s and 4608000kb,

    spi= 0x4A10F22E(1242624558), conn_id= 0, keysize= 0, flags= 0x400C



!The protocol and the transforms are specified by the crypto map that has been

!hit, as are the lifetimes

!negotiate phase I SA parameters.



  ISAKMP: received ke message (1/1)

  ISAKMP: local port 500, remote port 500

  ISAKMP (0:1): Input = IKE_MESG_FROM_IPSEC, IKE_SA_REQ_MM Old State =

        IKE_READY  New State = IKE_I_MM1

  ISAKMP (0:1): beginning Main Mode exchange

  00:04:10: ISAKMP (0:1): sending packet to 172.16.172.20 (I) MM_NO_STATE

  00:04:10: ISAKMP (0:1): received packet from 172.16.172.20 (I) MM_NO_STATE

  00:04:10: ISAKMP (0:1): Input = IKE_MESG_FROM_PEER, IKE_MM_EXCH

  Old State = IKE_I_MM1  New State = IKE_I_MM2



00:04:10: ISAKMP (0:1): processing SA payload. message ID = 0

00:04:10: ISAKMP (0:1): found peer pre-shared key matching 172.16.172.20

00:04:10: ISAKMP (0:1): Checking ISAKMP transform 1 against priority 1 policy

00:04:10: ISAKMP:      encryption 3DES-CBC

00:04:10: ISAKMP:      hash SHA

00:04:10: ISAKMP:      default group 1

00:04:10: ISAKMP:      auth pre-share

00:04:10: ISAKMP:      life type in seconds

00:04:10: ISAKMP:      life duration (VPI) of  0x0 0x1 0x51 0x80

00:04:10: ISAKMP (0:1): atts are acceptable. Next payload is 0

00:04:10: ISAKMP (0:1): Input = IKE_MESG_INTERNAL, IKE_PROCESS_MAIN_MODE

Old State = IKE_I_MM2  New State = IKE_I_MM2



!Policy 1 on this router and the atts offered by the other side matched.

!The third and fourth packets complete the  Diffie-Hellman Exchange.



ISAKMP (0:1): sending packet to

172.16.172.20 (I) MM_SA_SETUP

ISAKMP (0:1): Input = IKE_MESG_INTERNAL,

  IKE_PROCESS_COMPLETE Old State = IKE_I_MM2  New State = IKE_I_MM3

ISAKMP (0:1): received packet from 172.16.172.20 (I) MM_SA_SETUP

ISAKMP (0:1): Input = IKE_MESG_FROM_PEER, IKE_MM_EXCH

Old State = IKE_I_MM3  New State = IKE_I_MM4



ISAKMP (0:1): processing KE payload. message ID = 0

ISAKMP (0:1): processing NONCE payload. message ID = 0

ISAKMP (0:1): found peer pre-shared key matching 172.16.172.20

ISAKMP (0:1): SKEYID state generated

ISAKMP (0:1): processing vendor id payload



!The fifth and sixth packets complete IKE authentication. Phase I SA established.



ISAKMP (0:1): SA is doing pre-shared key

authentication using id type ID_IPV4_ADDR

...

ISAKMP (0:1): sending packet to 172.16.172.20

(I) MM_KEY_EXCH

ISAKMP (0:1): Input = IKE_MESG_INTERNAL,

IKE_PROCESS_COMPLETEOld State = IKE_I_MM4  New State = IKE_I_MM5



ISAKMP (0:1): received packet from 172.16.172.20 (I) MM_KEY_EXCH

ISAKMP (0:1): Input = IKE_MESG_FROM_PEER, IKE_MM_EXCH

Old State = IKE_I_MM5  New State = IKE_I_MM6

ISAKMP (0:1): processing ID payload. message ID = 0

ISAKMP (0:1): processing HASH payload. message ID = 0

ISAKMP (0:1): SA has been authenticated with 172.16.172.20

ISAKMP (0:1): Input = IKE_MESG_INTERNAL, IKE_PROCESS_COMPLETE

Old State = IKE_I_MM6  New State = IKE_P1_COMPLETE



!Begin Quick Mode exchange. IPsec SA is negotiated in QM.



ISAKMP (0:1): beginning Quick Mode exchange, M-ID of 965273472

ISAKMP (0:1): sending packet to 172.16.172.20 (I) QM_IDLE

ISAKMP (0:1): Node 965273472, Input = IKE_MESG_INTERNAL, IKE_INIT_QM Old State =

  IKE_QM_READY  New State = IKE_QM_I_QM1

ISAKMP (0:1): received packet from 172.16.172.20 (I) QM_IDLE



!The IPsec SA proposal offered by the far end is checked against the local crypto

!map configuration



ISAKMP (0:1): processing HASH payload. message ID = 965273472

ISAKMP (0:1): processing SA payload. message ID = 965273472



ISAKMP (0:1): Checking IPsec proposal 1

ISAKMP: transform 1, ESP_3DES

ISAKMP:   attributes in transform:

ISAKMP:      encaps is 1

ISAKMP:      SA life type in seconds

ISAKMP:      SA life duration (basic) of 3600

ISAKMP:      SA life type in kilobytes

ISAKMP:      SA life duration (VPI) of  0x0 0x46 0x50 0x0

ISAKMP:      authenticator is HMAC-MD5

ISAKMP (0:1): atts are acceptable.

IPSEC(validate_proposal_request): proposal part #1,

  (key eng. msg.) INBOUND local= 172.16.172.10, remote= 172.16.172.20,

  local_proxy= 10.1.1.0/255.255.255.0/0/0 (type=4),

  remote_proxy= 10.1.2.0/255.255.255.0/0/0 (type=4),

  protocol= ESP, transform= esp-3des esp-md5-hmac ,

  lifedur= 0s and 0kb,

  spi= 0x0(0), conn_id= 0, keysize= 0, flags= 0x4



!Two IPsec SAs have been negotiated--an incoming SA with the SPI generated by the

!local machine and an outbound SA with the SPIs proposed by the remote end.





ISAKMP (0:1): Creating IPsec SAs

inbound SA from 172.16.172.20 to 172.16.172.10(proxy 10.1.2.0 to 10.1.1.0)

has spi 0x8EAB0B22 and conn_id 2029 and flags 4

 lifetime of 3600 seconds lifetime of 4608000 kilobytes

outbound SA from 172.16.172.10   to 172.16.172.20 (proxy 10.1.1.0 to 10.1.2.0)

has spi -343614331 and conn_id 2030 and flags C

  lifetime of 3600 seconds lifetime of 4608000 kilobytes



!The IPsec SA info negotiated by IKE will be populated into the router's SADB.



00:04:10: IPSEC(key_engine): got a queue event...

00:04:10: IPSEC(initialize_sas): ,

  (key eng. msg.) INBOUND local= 172.16.172.10, remote= 172.16.172.20,

  local_proxy= 10.1.1.0/255.255.255.0/0/0 (type=4),

  remote_proxy= 10.1.2.0/255.255.255.0/0/0 (type=4),

  protocol= ESP, transform= esp-3des esp-md5-hmac ,

  lifedur= 3600s and 4608000kb,

  spi= 0x8EAB0B22(2393574178), conn_id= 2029, keysize= 0, flags= 0x4

00:04:10: IPSEC(initialize_sas): ,

  (key eng. msg.) OUTBOUND local= 172.16.172.10, remote= 172.16.172.20,

  local_proxy= 10.1.1.0/255.255.255.0/0/0 (type=4),

  remote_proxy= 10.1.2.0/255.255.255.0/0/0 (type=4),

  protocol= ESP, transform= esp-3des esp-md5-hmac ,

  lifedur= 3600s and 4608000kb,

  spi= 0xEB84DC85(3951352965), conn_id= 2030, keysize= 0, flags= 0xC



!IPsec SA created in SADB, sent out last packet with commit bit set. IPsec

!tunnel established.

IPSEC(create_sa): sa created,

 (sa) sa_dest= 172.16.172.10,

  sa_prot= 50,

  sa_spi= 0x8EAB0B22(2393574178),

  sa_trans= esp-3des esp-md5-hmac ,

  sa_conn_id= 2029

IPSEC(create_sa): sa created,

  (sa) sa_dest= 172.16.172.20, sa_prot= 50, sa_spi= 0xEB84DC85(3951352965),

  sa_trans= esp-3des esp-md5-hmac , sa_conn_id= 2030

ISAKMP (0:1): sending packet to 172.16.172.20 (I) QM_IDLE

ISAKMP (0:1): Node 965273472, Input = IKE_MESG_FROM_PEER, IKE_QM_EXCH

Old State = IKE_QM_I_QM1  New State = IKE_QM_PHASE2_COMPLETE

IPsec show Commands

IPsec show commands are particularly useful in debugging IPsec. Having established basic connectivity, the network administrator can use show commands to see the issues related to the operation of IPsec on top of the basic network connectivity. Three show commands are most commonly used to view the status of an IPsec connection:

• show crypto isakmp sa
• show crypto ipsec sa
• show crypto engine connection active

The show command output in Example below shows one ISAKMP SA and two IPsec SAs established. The packet count for each of the IPsec SAs, along with the negotiated parameters, are also shown.

Example . Sample Output for the show crypto engine connection active Command

Router#show crypto engine connection active

ID   Interface       IP-Address      State  Algorithm           Encrypt   Decrypt

 1   <none>          <none>          set    HMAC_SHA+3DES_56_C        0        0

!Shown above is the ISAKMP SA. It shows the hashing algorithm, SHA-HMAC, which is

!being used for authentication, as well as the encryption algorithm, 3DES, used to

!encrypt the IKE negotiation messages. The encrypt and decrypt counts are 0

!because the ISAKMP SA is not used to encrypt and decrypt data.

2029 Ethernet1/0     172.16.172.10   set    HMAC_MD5+3DES_56_C        0        4

2030 Ethernet1/0     172.16.172.10   set    HMAC_MD5+3DES_56_C        4        0

!Shown above are the two IPsec SAs. They show the hashing algorithm, SHA-HMAC,

!which is used for message integrity checking, as well as the encryption

!algorithm, 3DES, used to encrypt the ESP packets. The first SA is the incoming

!SA, because the packet count shows decrypts in it. The other one is the outgoing

!SA, showing only encrypts.

The show command in Example below shows the ISAKMP SA. It is in QM_IDLE state, meaning that quick mode has been successfully completed.

Example . Sample Output for the show crypto isakmp sa Command

Router#show crypto isakmp sa

dst             src             state           conn-id    slot

172.16.172.20   172.16.172.10   QM_IDLE               1       0

The ISAKMP SA can be in a number of states, depending on which state the negotiation is in. The following tables list these states. They are an excellent reference if you are trying to use show command output to find out the point at which a negotiation is. Table below shows the states displayed in the show crypto isakmp sa command when main mode is being negotiated. important.

Table . States Displayed in the show crypto isakmp sa Command When Main Mode Is Being Negotiated

State	Description
OAK_MM_NO_STATE	The ISAKMP SA has been created, but nothing else has happened yet. It is "larval" at this stage-there is no state.
OAK_MM_SA_SETUP	The peers have agreed on parameters for the ISAKMP SA.
OAK_MM_KEY_EXCH	The peers have exchanged Diffie-Hellman public keys and have generated a shared secret. The ISAKMP SA remains unauthenticated.
OAK_MM_KEY_AUTH	The ISAKMP SA has been authenticated. If the router initiated this exchange, this state transitions immediately to OAK_QM_IDLE, and a quick mode exchange begins.

Table below shows the states displayed in the show crypto isakmp sa command when aggressive mode is being negotiated. 

Table . States Displayed in the show crypto isakmp sa Command When Aggressive Mode Is Being Negotiated

State	Description
OAK_AG_NO_STATE	The ISAKMP SA has been created, but nothing else has happened yet. It is "larval" at this stage-there is no state.
OAK_AG_INIT_EXCH	The peers have done the first exchange in aggressive mode, but the SA is not authenticated.
OAK_AG_AUTH	The ISAKMP SA has been authenticated. If the router initiated this exchange, this state transitions immediately to OAK_QM_IDLE, and a quick mode exchange begins.

Table below shows the state displayed in the show crypto isakmp sa command when quick mode is being negotiated or has been negotiated. In general, the states shown in these tables are the most commonly seen states. Some other states might also be seen, but these shown here are by far the most 

Table . State Displayed in the show crypto isakmp sa Command When Quick Mode Is Being Negotiated or Has Been Negotiated

State	Description
OAK_QM_IDLE	The ISAKMP SA is idle. It remains authenticated with its peer and may be used for subsequent quick mode exchanges. It is in a quiescent state.

The show commands in Example below show the packets, counts, SPI, and various other parameters for both of the IPsec SAs in place between the two routers.

Example . Sample Output for the show crypto IPsec sa Command

Router#show crypto IPsec sa

interface: Ethernet1/0

    Crypto map tag: vpn, local addr. 172.16.172.10



   local  ident (addr/mask/prot/port): (10.1.1.0/255.255.255.0/0/0)

   remote ident (addr/mask/prot/port): (10.1.2.0/255.255.255.0/0/0)

   current_peer: 172.16.172.20

     PERMIT, flags={origin_is_acl,}

    #pkts encaps: 4, #pkts encrypt: 4, #pkts digest 4

    #pkts decaps: 4, #pkts decrypt: 4, #pkts verify 4

    #pkts compressed: 0, #pkts decompressed: 0

    #pkts not compressed: 0, #pkts compr. failed: 0, #pkts decompress failed: 0

    #send errors 6, #recv errors 0



     local crypto endpt.: 172.16.172.10, remote crypto endpt.: 172.16.172.20

     path mtu 1500, media mtu 1500

     current outbound spi: EB84DC85





inbound esp sas:

      spi: 0x8EAB0B22(2393574178)

        transform: esp-3des esp-md5-hmac ,

        in use settings ={Tunnel, }

        slot: 0, conn id: 2029, flow_id: 1, crypto map: vpn

        sa timing: remaining key lifetime (k/sec): (4607998/3347)

        IV size: 8 bytes

        replay detection support: Y



outbound esp sas:

      spi: 0xEB84DC85(3951352965)

        transform: esp-3des esp-md5-hmac ,

        in use settings ={Tunnel, }

        slot: 0, conn id: 2030, flow_id: 2, crypto map: vpn

        sa timing: remaining key lifetime (k/sec): (4607999/3347)

        IV size: 8 bytes

        replay detection support: Y

Commonly Seen IPsec Problems and Resolutions

IPsec issues can arise from a wide variety of reasons. This section discusses some of the more common problems with setting up IPsec tunnels and their functioning.

Incompatible ISAKMP Policy or Preshared Key

It is important for the ISAKMP preshared key, if that is the method of authentication being used, to match on the two IPsec peers. If this doesn't happen, the tunnel won't come up. The failure occurs in phase one of ISAKMP.

Example below shows the debugs seen on a Cisco IOS router when the preshared keys do not match.

Example . Sample Debug Output Seen on a Cisco IOS Router When the Preshared Keys Configured on It and Its Peer Do Not Match

Router#debug crypto ISAKMP

Router#debug crypto engine

Router#debug crypto ipsec



ISAKMP: reserved no zero on payload 5!

%CRYPTO-4-IKMP_BAD_MESSAGE: IKE message from 155.0.0.1   failed its

sanity check or is malformed

It is also important for at least one ISAKMP policy on the two peers to match. If this does not happen, phase 1 of ISAKMP fails.

Example below contains sample debugs seen on a router when ISAKMP policies fail to match.

Example . Sample Debug Output Seen on a Router When ISAKMP Policies Fail to Match

Router#debug crypto ISAKMP

Router#debug crypto engine

Router#debug crypto ipsec



ISAKMP (0:1): Encryption algorithm offered does not match policy!

ISAKMP (0:1): atts are not acceptable. Next payload is 0

ISAKMP (0:1): no offers accepted!

ISAKMP (0:1): phase 1 SA not acceptable!

Incorrect Access Lists for Interesting Traffic

For IPsec to work correctly, the access control lists on the two peers must define interesting traffic such that the two peers can agree that they want to encrypt the same traffic. In most cases, the access lists configured on the two peers are exact reflections of each other. However, it is permissible for the traffic specified by an access list defined on one peer to be a subset of the traffic specified by the access list configured on the other peer. In the case of VPN clients, dynamic crypto maps need to be used to get over the problem of the VPN clients coming from an unknown IP address.

Example below shows the debugs that appear on the routers when the access lists configured on two peers negotiating IPsec are neither the same nor the subsets of each other.

Example . Sample Debug Output When the Access Lists Configured on Two Peers Negotiating IPsec Are Not the Same or Are Subsets of Each Other

Router#debug crypto ISAKMP

Router#debug crypto engine

Router#debug crypto ipsec



3d00h: IPsec(validate_transform_proposal): proxy identities not supported

3d00h: ISAKMP (0:3): IPsec policy invalidated proposal

3d00h: ISAKMP (0:3): phase 2 SA not acceptable!

Crypto Map Is on the Wrong Interface

Generally, the crypto map needs to be applied to the outgoing interface of the router or the PIX. Not applying the map to the egress interface stops IPsec from kicking in at all, and no ISAKMP debugs are generated. The crypto map needs to be applied to tunnel interfaces as well if you are using GRE with IPsec. In general, if a logical interface is used in conjunction with a physical interface to pass IPsec traffic, the crypto map needs to be applied to both of them.

Routing Issues

Routing can play an important role in bringing up an IPsec tunnel successfully. Keep in mind the following points when examining routing issues in IPsec implementations:

• A packet needs to be routed to the interface that has the crypto map configured on it before IPsec kicks in.
• Routes need to be there not only for the router to reach its peer's address but also for the IP subnet addresses in the IP packets' heads after they have been decrypted.
• Use the debug ip packet acl detailed command to see if the routing is occurring correctly.

Bypassing NAT

We have already discussed how NAT can be bypassed on a router for traffic that is to be encapsulated in an IPsec tunnel.

On the PIX Firewall, the nat (inside) 0 access-list command can be used to bypass NAT in a fashion similar to the routers.

Time Settings for Certificates

Digital certificates are issued for a time frame. They become invalid after that and are invalid before the time frame starts. The clock and calendar of the machine on which the certificates are installed are used to determine the certificates' validity. It is important to have the correct time and date configured on the router or the PIX using certificates so that they can function properly.

Firewall or Access List Is Blocking IPsec Negotiations or Traffic

A firewall in the middle of an IPsec tunnel must allow the following protocols to go through it for IPsec to function properly:

• ESP and/or AH
• UDP port 500

The IPsec router, if it has an access list configured on the egress interface, must have holes in that access list not only for these two protocols but also to accommodate the fact that the incoming access list is applied twice to the IPsec traffic. See the discussion at the start of this section for details.

IPsec MTU Issues

IPsec adds significant overhead to the original IP packet being encapsulated. Therefore, it is possible that the new encapsulated packet's size might be more than the MTU on certain segments over which the IPsec packets must pass. Normal Path MTU mechanisms using ICMP packets can take care of fixing the MTU size, but when such ICMPs are blocked by a firewall or an access list somewhere along the path that the IPsec packet takes, issues can arise. The symptoms of these issues are often the inability of users to run applications such as e-mail and large file FTPs across the IPsec tunnel. Figure below shows how Path MTU discovery with IPsec works. If a router in the path of the packet being sent by the end host is unable to transmit the frame using the size of the frame it has received, it sends back an ICMP packet to the end host, asking for the packet size to be reduced to the maximum size the router can support. The end host then transmits the packets in frames of reduced size, which the router can transmit. This process is repeated as necessary by all the routers in the path of the packets transmitted by the end host.

Figure . IPsec Path MTU

However, there are situations in which the ICMP packets generated in the fashion previously described are blocked (perhaps by a firewall) from reaching the end host. Therefore, the end host never finds out that there is a problem with the MTU of the packets it is sending and continues sending frames with the large MTU size resulting in these large packets being dropped by the device which does have an MTU size large enough to accommodate these packets. Another situation arises when the end host does receive the ICMP packets but due to a bug or faulty implementation of the TCP/IP stack does not reduce its MTU size. The result is the same as described for ICMPs being blocked from reaching the end host.

In both of these cases, two options remain for the network administrator to fix the problem:

• Setting up the TCP MSS on an edge router, forcing a small TCP MSS to be negotiated
• Manually reducing the MTU on the end hosts that are not receiving the ICMP packets or are choosing to ignore them

To use either of these methods, you must know how to calculate the size of the packet after the IPsec header has been added. The following formula is used to calculate the size to which the original IP packet must be restricted to remain within the smallest MTU available on the path that is traversed by the IPsec tunnel:

maximum original packet size allowed = floor((IPsec packet size allowed),8) - (IPsec header+SPI+sequence number+HMAC) - (IV+pad+pad length+next header)

where

IPsec header = 20 bytes in ESP tunnel mode

SPI (Security Parameter Index) = 4 bytes

Sequence number = 4 bytes

ESP-HMAC MD5/SHA 96 digest = 12 bytes

IV (Initialization Vector) = 8 bytes

Pad = 1 byte

Pad length = 1 byte

Next header = 1 byte

The floor function here is calculated by finding the largest whole number (a number such as 1200.0 or 1300.0, not 1200.345 or 1300.897) that is fully divisible by 8 and that is less than "IPsec packet size allowed." For example, if the IPsec packet size allowed is 876, the floor to 8 would be 872, because 872 is fully divisible by 8 (872 / 8 = 109.0) and 872 is the largest such number that is less than 876.

A sample calculation is shown next. In this case, the network administrator has found out that the smallest MTU on the IPsec tunnel path is 1330 bytes:

maximum original packet size allowed = floor((1330),8) - (20+4+4+12) - (8-1-1-1) = 1328 - 40 - 11 = 1277

Therefore, the network administrator can manually configure the end host not to send a packet larger than 1277 bytes.

If the network administrator does not want to change the MTU on the end host, such as when there are too many of them, he or she can use the tcp mss command on the router's ingress interface. The router for the default gateway for the end hosts is:

ip tcp adjust-mss number

This command forces the router to sniff on the incoming TCP SYN packets and tweak the TCP MSS field to the number configured in this command. With the MSS value tweaked, the two end hosts sitting behind the two IPsec peers must agree to this tweaked, negotiated TCP MSS value.

The number is calculated by taking into account the size of the TCP header in the packet. The calculation is as follows:

Maximum TCP MSS value allowed = maximum original packet size allowed - 40

So, in the case of our example, the TCP MSS number would be 1277 - 40 = 1237 bytes.