Understanding IPSec tunnels (L2L)

In my current role, I deal with L2L (LAN to LAN) tunnels quite frequently.  They are the ‘cheap’ way to connect two locations in comparison to dedicated access circuits.  The only requirement is having the correct hardware and a ‘always on’ static IP at each location.  I frequently joke that configuring a L2L tunnel on a ASA is easy and only takes 20 config lines in the CLI.  That being said like anything else in the network world, if you don’t understand how the technology works, you are going to get stuck one day.  Typing 20 config lines into a console and actually understanding what’s happening are two entirely different things.  In this post I’ll talk about IPSec basics, how tunnels form, and some troubleshooting steps you can take when you run into issues.

Terminology
IPSec (Internet Protocol Security) – What we are talking about

IKE (Internet Key Exchange) – Think of IKE as the toolset required to establish and maintain a secure connection between two endpoints.  Basically IKE allows the connection to form and maintain SAs (Security Associations) which are the key part of an IPSec tunnel

ISAKMP (Internet Security Association and Key Management Protocol) – ISAKMP is the standard that defines how to work with SAs.  In other words, ISAKMP handles how SAs are created, maintained, modified, and torn down.  Have you noticed the overlap?  IKE and ISAKMP sound awfully similar, don’t they?  That’s because they are.  In fact, Cisco even notes this in their documentation.

“The Internet Security Association and Key Management Protocol, also called IKE, is the negotiation protocol that lets two hosts agree on how to build an IPsec security association. Each ISAKMP negotiation is divided into two sections called Phase1 and Phase 2.”

Peer Address – Depending on context, it’s either the public address at one end of the tunnel or at the other end of the tunnel.

DH (Diffie-Hellman) – DH allows two parties to share a secret key over an insecure channel.  DH is used during IKE phase 1 to exchange keys an establish a secure secret key.

ISAKMP Policy sets – A policy set specifying the IKE encryption algorithm, the IKE authentication algorithm, the IKE authentication type, DH version, and the IKE tunnel lifetime.  The ISAKMP Policy set is used during IKE Phase 1 negotiations.  To make this even more confusing ISAKMP policy sets are also referred to as either ISAKMP transform sets, IKE transform sets, or just plain old transform sets

IPSec Transform Sets – A transform set consists of settings relevant to the IPSec tunnel creation.  The actual transform set is used during IKE Phase 2 and generally just consists of the IKE encryption and authentication type.  Again the confusing part is that often times some people just call these transform sets.

SAs (Security Associations) – Basically it’s the set of IPSec parameters that both sides of the tunnel agree upon.  I’ll make a critical distinction here for those of us in Cisco land.  There will be a ISAKMP SA and a IPSec SA.  There are several tunnels created while the IPSec tunnel is being created.  Just remember that the ISAKMP SA is considered the result of the IKE phase 1 and the IPSec SA is considered the result of IKE phase 2.  Also, a SA is only a one way connection.  Since most traffic between endpoints require two way communication, each ASA will create its own SA to talk to the other peer.

Nonce – A number, usually used for a secure key, that is only used once.  If it’s used a second time it’s usually a sign of a fraudulent connection attempt.

Terminology Summary – VPN terminology is really, really, really confusing.  Just try to remember that ISAKMP and IKE are sometimes used interchangeably.  Additionally, transform-sets are a general term for a group of settings.  I try to use the phrase only when referring to what I type after ‘transform-set’ in the CLI which is used to specify the authentication and encryption type for the IKE Phase 2.

The Process
Now that we know some of the key terms we can start talking about the how the IPSec tunnel forms.  In my opinion, there are 4 phases to the life of a IPSec tunnel.  I’ll lay them out below and then we’ll talk about them

Phase 1 – Interesting traffic generates the creation of the tunnel
Phase 2 – IKE Phase 1
Phase 3 – IKE Phase 2
Phase 4 – Tunnel Termination

Some people throw a phase between my phase 3 and 4 and list it as ‘IPSec tunnel created’ which in my view point isn’t actually a phase.  Listing the product of phases 1 through 3 doesn’t seem to justify its own phase in my mind.  At any rate, let’s go into detail on each one.

Phase 1 – Interesting traffic generates the creation of the tunnel
What many people don’t know about IPSec is that it’s what I call an ‘on demand’ service.  That is, once you create a tunnel its not up and running all of the time.  What we call ‘interesting traffic’, triggers the creation of a tunnel.  Creating the tunnel doesn’t take very long but can, in most cases, be noticed when using ping to test.  In most cases when I try to ping from a client machine across the tunnel the first ICMP request will fail because the tunnel is being loaded.  Subsequent ICMPs go through correctly.  Interesting traffic is defined on the ASA with an ACL.  Then the ACL is specified within the tunnels crypto map with a ‘match address’ command.  An example ACL might look like this…

access-list L2LCryptoMap extended permit ip <Local subnet> <Local mask> <Destination subnet>  <Destination mask>

You tell the ASA to match traffic coming from your subnet to the subnet on the other side of the tunnel.  This brings up the question of duplicate private subnets.  If your subnet on inside interface of both ASAs is the same, you are going to have issues.  Think of it this way, when a client on one side tries to go to a resource on the other subnet it’s not going to go anywhere.  By default the client will ARP for a response because it thinks the remote client you are trying to access is on its local subnet.  Since it doesn’t try to go off subnet the client will never contact the default gateway (the ASA’s inside interface) and you’ll never form a tunnel.  Even if you did, traffic still wouldn’t flow.  There are some ‘tricky’ things we can do to fix this issue but they don’t fall within the realm of this blog entry.  Best situation is to have unique private networks in each location.

Phase 2 – IKE Phase 1
Once the ASA gets a request for a remote subnet, which it matches to a crypto map, IKE Phase 1 begins.  The goal of IKE phase 1 is to setup the connection for the IKE phase 2.  Basically, IKE phase 1 lays the ground work for the actual connection to occur.  It actually creates its own SA which is sometimes referred to as the management SA.  This management SA is used by IKE phase 2 to create the actual data tunnels.  From a technical standpoint there are three steps to IKE phase 1.  The first step is called policy negotiation.  Policy negotiation boils down to finding matching transform sets on each endpoint.  Before the ISAKMP SA forms each endpoint needs to agree on a matching IKE policy set.  The following items need to match within the policy set on each side of the tunnel.

-The IKE Encryption algorithm
-The IKE Authentication algorithm
-The IKE Key definition
-The Diffie Hellman Version
-The IKE Lifetime

Remember, IKE policy sets are associated with IKE phase 1 completion.  An endpoint can have any number of policy sets which are evaluated in a order. For instance, you might have policy set 10, 20, and 30.  The endpoints will cycle through the policy sets in order, starting with the lowest first until they find a match.  Hence, it makes sense to define stronger policy sets first.  If they are defined after weaker policy sets you might end up finding a weaker match first.

Now, before we go too much further into IKE phase 1 its pertinent to explain that there are two different types of IKE phase 1.  The one used most often is called ‘main mode’ and consists of 6 messages being exchanged between peers.  The other type is called ‘aggressive’ mode and condenses the messages sent between peers into 3 messages.  Main mode is slower in setup but more secure.  Aggressive mode is faster in setup but less secure.  Don’t quote me on this, but I believe that main mode is used by default.  I’m not going to dive into specifics here, they both accomplish the same thing and the details I listed above should clue you in on the big differences.

Our first step to IKE phase 1 was policy negotiation – the peers need to agree upon a matching policy set.  They do this by sending policy sets to each other for comparison and finding the lowest matching policy set.  Once that’s completed, we can move onto the next step which is the DH key exchange.  During this process the peers exchange public keys that are used to establish the shared secret.  After this process is done, the Phase 1 SA (ISAKMP SA) is formed and we move onto the last step which is peer authentication.

During peer authentication, the authentication method defined is used to ensure that rogue devices aren’t forming secure connections into your network.  Most of the time this is done with pre-shared keys.  This step is pretty simple, the peers check to make sure that the keys match.  If they don’t, authentication fails.  If they do match we move onto IKE phase 2.

Phase 3 – IKE Phase 2
I like to think of IKE phase 2 as the actual building of the data tunnel.  The work up to this point has been mainly to ensure that we have a secure communications channel in place so that we can build the actual IPsec SAs.  Compared to IKE phase 1, IKE phase 2 only has one type of mode which is called ‘quick mode’.  Quick mode uses the existing ISAKMP SA created in IKE phase 1 and creates the IPSec SAs and manages the key exchanges.  The first step of quick mode is to negotiate the IPSec transform sets (not the same as the ISAKMP policy sets!!).  The following items need to match on both ends for the IPSec SAs to form.

-The IPSec Protocol
-The IPsec Encryption type
-The IPSec Authentication type
-The IPSec mode
-The IPSec SA lifetime

Now lets take a brief moment to compare the transform-sets from IKE phase 1 and phase 2.  The following is a typical configuration off of a ASA.

ISAKMP Policy Set
crypto isakmp policy 10
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 86400

IPSec transform
crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac

See anything odd?  In the ISAKMP policy set you can clearly see the 5 required pieces of the transform-set that need to be agreed upon.  In the IPSec transform it’s not so clear.  This is because the transform really only defines 3 of the 5 required pieces.  The example above defines the IPSec protocol (ESP), the IPSec authentication type (SHA), and the IPSec encryption type (3DES).  IPSec mode and IPSec SA lifetime don’t need to be defined.  When they aren’t, the ASA simply assumes the following values of 28,800 for the SA lifetime and ‘tunnel’ mode for all transform sets.  I usually don’t change these values unless the other side changed them.

So once the transform sets have been negotiated and matched we can create the IPSec SAs.  As stated above, a SA is a unidirectional connection.  So for traffic to flow each way two IPSec SAs need to form.  The ASA does this work for you of course so there isn’t much detail to share here.  Quick mode uses nonces to generate new key material which prevents replay attacks.  The third step is the process of periodically renegotiating the connection.  The SAs need to be regenerated prior to the lifetime expiration of the tunnel.  Quick mode monitors this and generates new SAs before the old ones expire out.

Phase 4 – Tunnel Termination
At this point we have a fully functional VPN tunnel!  Traffic should be passing in both directions.  The only thing left to do is tear down the tunnel if there isn’t any interesting traffic.  The SAs are only regenerated if interesting traffic continues to flow.  If interesting traffic stops coming in the SA is allowed to expire out after its lifetime has been reached.  If the SA is allowed to expire ALL data regarding the SAS is flushed out of the SA.  Next time interesting traffic is generated the whole process will start from scratch.

Troubleshooting
The most common issue in IPSec connectivity is not finding transform-set matches for IKE phase 1 and 2.  If you run into issues, the first thing you should always do is double check the settings on both sides to ensure that they match 100%.  The next thing you can do is take a look at the SAs on the ASA and determine if they are created and if they are, what status they have.

To see the IKE Phase 1 SA issue this command
ASA# show crypto isakmp sa

To see the IKE Phase 2 SA issue this command
ASA# show crypto ipsec sa

If you don’t have a phase 1 SA then you aren’t going to get very far.  The state of the SA tells you a couple of things.  It tells you whether or not the SA was created with main or aggressive mode, which side brought the tunnel up, and also tells you the status of negotiations.  A look at a typical phase 1 SA is shown below.

ASA# show crypto isakmp sa
Active SA: 1
Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1

1   IKE Peer: <Peer IP Address>
     Type    : L2L             Role    : responder
     Rekey   : no              State   : MM_ACTIVE

As you can see from the above output, we have one IKE phase 1 SA.  The Type is ‘L2L’ (which indicates it’s a site to site IPSec tunnel), the role is ‘responder’ (meaning that the peer brought up the tunnel), and the state is MM_Active (meaning that it’s using main mode (MM) and the IKE phase 1 is working (ACTIVE)).  MM_ACTIVE is what we want to see on a phase 1 IKE SA.  That means that everything is ready for IKE phase 2 to occur.  So what should you do if you don’t have the ACTIVE state?  Or no SA at all?

There are several other states that can clue you into what’s going on with the ISAKMP SA.

Note: These status codes are only for the newer ASA version, the older IPSec code used different status messages.
The first two letters tell you whether the connection was made with main mode (MM) or aggressive mode (AM).  Since it can be either main or aggressive mode I will list is simply as ‘XX’ below.  Granted you’ll only see some of them with either main or aggressive mode.  The definitions I list to each state are what I have found to be the issue in most cases.

XX_Active – Connected
XX_KEY_EXCH – Authentication error, check your authentication method
XX_INIT_EXCH – Authentication error, check your authentication method
XX_NO_STATE – Unable to match IKE Phase 1 policies, verify on each side
XX_WAIT_MSG2 – Waiting for response from the peer

Moreover if you don’t have a SA at all, start by checking the obvious.  Start at layer 1 and work your way up ensuring both units have connectivity and that the peer addresses are correct.

99% of the time an issue is with the IKE phase 1 SA.  If you are getting a state of ACTIVE on phase 1 you are probably in good shape.  If you are still having issues, take a look at your IKE phase 2 SA.  I’m not going to show an example of one since its mostly just statistics.  The same concept applies though, if you don’t have a IPSec SA you don’t have a VPN tunnel.

Debugging
If you are still having issues, like anything else on the ASA, try debugging.  The two commands you are looking for would be…

ASA# debug crypto isakmp
ASA# debug crypto ipsec
ASA# terminal monitor

Make sure you enable terminal monitor if you aren’t on the console port

The Big Fix
I cant tell you how many times I’ve spent hours trying to get a tunnel to load with absolutely no success.  In the end, two commands always fixed my issues.  When all else fails, clear all of your ISAKMP and IPSEC SAs.  This will of course blow away ANY IPSec connections to the box.  Something about just blowing all the other connections away and starting from scratch seems to fix it.  To do this….

ASA# clear crypto ipsec sa
ASA# clear crypto isakmp sa

Conclusion
I hope you have seen, through this article, that having a base understanding of how IPSec works makes a world of difference.  Knowing where to look for errors and how to interpret them is key to troubleshooting IPSec L2L tunnels.