Das Blinken Lichten

You could add this to my ‘Commands I always forget’ post, but since I’m going to turn this into a little bit of a walk through I decided to make it into it’s own post.  Most admins use two commands to verify IPSec VPN security associations.  Those, of course, are…

ASA# show crypto isakmp sa

and

ASA# show crypto ipsec sa

Both of these commands provide you with a wealth of information about the IPSec connection.  However, what about if you start talking about SSL VPN sessions?  Or WebVPN sessions?  Since these technically aren’t IPSec connections, they don’t show up in the ‘show crpypto’ commands.  Below I’ll walk through a couple of commands which show you some more information about all types of VPN connections.

How to see current WebVPN Sessions
ASA# show vpn-sessiondb webvpn
Session Type: WebVPN
Username     : langemakj              Index        : 13
Public IP    : 10.20.30.78
Protocol     : Clientless
License      : SSL VPN
Encryption   : RC4                    Hashing      : SHA1
Bytes Tx     : 147092                 Bytes Rx     : 31993
Group Policy : GP_SSLVPN              Tunnel Group : TG_SSLVPN
Login Time   : 14:04:01 CST Thu Jul 29 2010
Duration     : 0h:00m:11s
NAC Result   : Unknown
VLAN Mapping : N/A                    VLAN         : none
Notes: So as you can see, this gives you a ton of info on the connection including the users group policy, tunnel group, and their public IP (Note: I’m testing off of the internal ASA interface hence the RFC 1918 addressing).

How to log off current WebVPN Sessions
ASA# vpn-sessiondb logoff name langemakj
Do you want to logoff the VPN session(s)? [confirm]
INFO: Number of sessions with name "langemakj" logged off : 1
Notes: What’s interesting about the log off procedure is that its done by tunnel group or username.  Note in this instance, I don’t even have to note that it’s a WebVPN session that I want to log off.  Conversely, if I wanted to log off all of the WebVPN sessions I could just input ‘vpn-sessiondb logoff webvpn’ which would log off all users connected to WebVPN.

Wrap up
So now that we have an idea of how it works with WebVPN connections, let’s use the trusty ‘?’ to see what else we can do with the ‘vpn-sessiondb’.

ASA# show vpn-sessiondb ?
  detail       Show detailed output
  email-proxy  Email-Proxy sessions
  full         Output formatted for data management programs
  index        Index of session
  l2l          IPsec LAN-to-LAN sessions
  ratio        Show VPN Session protocol or encryption ratios
  remote       IPsec Remote Access sessions
  summary      Show VPN Session summary
  svc          SSL VPN Client sessions
  vpn-lb       VPN Load Balancing Mgmt sessions
  webvpn       WebVPN sessions
  |            Output modifiers
  <cr>

As you can see, you can use the vpn-sessiondb command to look at each type of VPN connection.  While I usually still use the ‘show crypto’ commands for IPSec connections, you HAVE to use the vpn-sessiondb for AnyConnect and WebVPN.  Play around with it, remember, the ‘?’ is your best friend!

I had a bit of misfortune last night with VMWare and seeing as I’m big into learning from my mistakes, I thought I’d share my mistake with everyone.  I had been working on an issue on a VMWare ESX 3.5 server and after speaking with VMWare support they recommended that I rebuild the guest’s VMX file.  For those of you who don’t know, a VMX is essentially the guest configuration file.  This isn’t a big deal to do and I have done it several times in the past successfully.  However this time was different.  I waited until 6pm, shut down the VM, removed it from inventory, copied the current VMX file out of the guest’s VM folder for backup, renamed it, and then began creating the new virtual machine.  Creating the new virtual machine is much like how you create any other sort of VMWare guest, the only difference is that you exactly replicate the original guest’s configuration settings and you chose the ‘use existing disks option’ pointing the new guest configuration to the servers existing disks.  Pretty easy right?  I thought so.  So I ensured the new VMX saved and booted up the guest machine.  Immediately I found a problem.  I couldn’t log in and I was getting domain controller errors which forced me to log in as the local admin.  Upon logging in I realized that things looked drastically different.  Starting to freak out (since this was a production exchange server) I checked the mail databases to find the last edited date to be over 3 months ago.  What had gone wrong?!?!?

After I managed to calm down a bit, I shut down the guest, and put my original VMX file back in place hoping I could get the server to boot with its initial configuration.  After moving the new VMX file out, and putting the old one back in, I tried to boot the server and I was greeted with an error message that looked like this….
 

At this point I immediately realized what had happened and was in absolute horror.  Someone had created a snapshot of the server 3 months ago, and never merged it back into the VM.  When I created the new guest, I had pointed to the disks to original VMDK files (which is your only option) and when I booted the server with them the file became modified which broke the snapshot chain.  I thought I had lost the system.

Before I go any further, I want to point out how I got to this point and things that should have been done to avoid such a catastrophic problem.

Issue - When you create the new VMX file and are asked to select the existing disk, the wizard only shows you the base VMDK file rather than all of the snapshots (even though they end in VMDK as well). 
Lesson Learned – Check the snapshot manager before making ANY changes.  A more thorough look in the data store browser would have point out the snapshot files to me as well

Issue – I had wanted to make a local copy of the guests folder on the data store prior to making ANY changes but was unable due to lack of space. 
Lesson Learned – If you are using local storage, ALWAYS have enough space to make a complete backup of any guest on that system.  If its a SAN, you might be able to get away with just taking a snapshot at the storage level.

Issue – The snapshots in general
Lesson Learned – As a rule, I don’t usually take snapshots in VMWare.  And if I do, I don’t let them hang out there for more than a day.  Keep an eye on the snapshots!

Green – Console output
Blue – My values, you’ll need to insert your own
Normal text – What I entered
Bold – Parts I’m trying to point out in the output

How I got out of this mess
After doing some research, I determined that the issue was with what VMWare calls the CID chain.  The CID value is used to link snapshots to parent VMDK files and when you load the parent file when there are snapshots present, you screw up this chain.  I didn’t have a backup of the system (at the VMWare level) and even though I had found steps online on how to try to fix this.  I let the experts at VMWare do it for me.  Afterwards, I recreated the scenario at home for the purpose of making this blog entry.  The steps below are offered at your own risk and there is no guarantee that they will work.

Step 1
BACKUP EVERYTHING.  Backup as much as you can, these steps will outline making direct edits to VMDK files which ,if done incorrectly, can cause corruption.  Also, try not to make ANY changes on the VMWare guest when you boot the parent VMDK.  I realized my mistake, and shut it down as soon as I could.

Step 2 
If you haven’t done so already, put the original VMX file back in place.  We’ll be using it to try to boot the system.  If you deleted it (for some silly reason) you may be able to make a new one but I haven’t tried that.

Step 3
Enable SSH on the ESX box if you haven’t already.  I’m running ESXi 3.5 in this test but you can very easily google how to enable SSH on your ESX version.  Using SSH just makes this a lot easier (especially if you are remote as I was).  In this example I’ll actually be using telnet.

Step 4
Figure out what the CID values are for each of the snapshots.  To do this we need to log in to the console (via SSH) and find the value on each of the files.  To do this, I entered the following commands.  Note the server I am running this test on is called UberServ01. 

Change to the guest directory
~ # cd /vmfs/volumes
/vmfs/volumes # ls
0016047d-c4d39b6a-ec38-631130484fa9  Hypervisor1
3abb47ef-875ea67c-c948-7bf6ff8d3c38  Hypervisor2
4af76f09-2611a4b1-ea7e-000f1ff86fb0  Hypervisor3
4af76f0b-0ba64946-17d8-000f1ff86fb0  datastore1
931ac070-8437760b-9dcc-b0a7dbce2d74
/vmfs/volumes # cd ./datastore1/
/vmfs/volumes/4af76f0b-0ba64946-17d8-000f1ff86fb0 # ls
ISO                    UberServ01 
/vmfs/volumes/4af76f0b-0ba64946-17d8-000f1ff86fb0 # cd ./UberServ01/
/vmfs/volumes/4af76f0b-0ba64946-17d8-000f1ff86fb0/UberServ01 # ls -ltr *.vmdk
-rw——-    1 root     root          255 Jul 15 21:34 UberServ01-000001.vmdk
-rw——-    1 root     root     67131904 Jul 15 21:36 UberServ01-000001-delta.vmdk
-rw——-    1 root     root          262 Jul 15 21:42 UberServ01-000002.vmdk
-rw——-    1 root     root     67131904 Jul 15 21:44 UberServ01-000002-delta.vmdk
-rw——-    1 root     root          532 Jul 15 21:48 UberServ01.vmdk
-rw——-    1 root     root    10758666240 Jul 15 21:52 UberServ01-flat.vmdk

The last command outputs the name of all of the files in the directory with the VMDK file extension.  As you can see, we have two snapshot files here.  One called UberServer01-000001.vmdk and the other UberServ02-000002.vmdk.  To return the CID values enter the following commands.

/vmfs/volumes/4af76f0b-0ba64946-17d8-000f1ff86fb0/UberServ01 # grep CID ./UberServ01-000001.vmdk
CID=986a79c0
parentCID=4fc239f6

/vmfs/volumes/4af76f0b-0ba64946-17d8-000f1ff86fb0/UberServ01 # grep CID ./UberServ01-000002.vmdk
CID=1df04fbb
parentCID=986a79c0

/vmfs/volumes/4af76f0b-0ba64946-17d8-000f1ff86fb0/UberServ01 # grep CID ./UberServ01.vmdk
CID=0db4eee2
parentCID=ffffffff

Note that the parent file (the original VMDK) will always have the parentCID value of ‘ffffffff’ since its the parent.

So as of right now here is the info we have….
Correct Parent CID – 0db4eee2
Snapshot 1 Parent CID – 4fc239f6
Snapshot 1 CID -986a79c0
Snapshot 2 Parent CID – 986a79c0
Snapshot 2 CID - 1df04fbb

The issue should be apparent at this point.  The CIDs need to reference each other in the correct order.  So this is what we currently have…..

Do you see the issue?  The 1st snapshot references the wrong CID value for the parent.  To fix this, we need to edit the parent VMDK file to reference the new CID value of 4fc239f6.  To to do this, enter the following commands….

/vmfs/volumes/4af76f0b-0ba64946-17d8-000f1ff86fb0/UberServ01 # vi ./UberServ01.vmdk

Now this is the tricky part.  I absolutely hate vi, I just can’t seem to get the hang of it.  But here’s what I did to edit the file.  Keep in mind that if the file is large, it can take some time to load, so if you get a black screen for a period of time, just hang tight.

-Use the arrow keys to arrow down to the beginning of the line that starts with ‘CID=’.  This line should list the incorrect SID which you pulled earlier.  Rather than deleting the value we are just going to comment it out.
-Press the insert key once
-Press enter to insert a new line
-Type a # to comment out the original CID line
-Arrow up one to start entering text on your new blank line
-Type in the new CID value prefaced by ‘CID=’
-When you are done, the lines should look something like this….

-Press the escape key once. 
-Type in ‘:wq’ (minus the single quotes)
-This should kick you back out to the command line

At this point I would verify the chain one more time to ensure that you have the CIDs correct.

/vmfs/volumes/4af76f0b-0ba64946-17d8-000f1ff86fb0/UberServ01 # grep CID ./UberServ01.vmdk
CID=4fc239f6
#CID=0db4eee2
parentCID=ffffffff
/vmfs/volumes/4af76f0b-0ba64946-17d8-000f1ff86fb0/UberServ01 # grep CID ./UberServ01-000001.vmdk
CID=986a79c0
parentCID=4fc239f6
/vmfs/volumes/4af76f0b-0ba64946-17d8-000f1ff86fb0/UberServ01 # grep CID ./UberServ01-000002.vmdk
CID=1df04fbb
parentCID=986a79c0

So now it looks like….

 
If you are confident with your changes, try to fire up the VM now.  With any luck, it will boot correctly and your current data will be back where it should be.  After you make sure that it boots correctly, I would shut it down and delete the snapshots to merge everything back into one file.  If you booted up the parent file and made changes the chances that some sort of corruption occurred is high.  In my case, I had to rebuild the server WMI namespace which wasn’t that big of a deal.  But all I did was boot the server up!

I had the opportunity to configure ISP failover on an ASA the other day and I thought I’d share the configuration as well as a couple of tips on using it.  I recall that when I started working on ASA’s I would always read that ‘dual ISP’ support was a feature of the Security Plus (Sec+) licensing set.  To me, that always meant that it was its own feature and had its own configuration commands.  As it turns out, that’s not the case.  In fact, the only reason you really need Sec+ to accomplish this is so that you have can have an additional ‘full’ interface.  The rest of the configuration is really just a SLA monitor, tracked default routes, and a extra global NAT pool.  Let’s walk through the configuration on a 5505.  I’m going to only touch the major parts of this so please don’t consider this a full build.

Notes
-Insert your relevant information between <>
-Console prompts are show in green
-Text in blue are variable names I made up, feel free to change them

Configure your interfaces
Inside Interface
ASA(config)# interface Vlan1
ASA(config-if)#description Inside Interface
ASA(config-if)# nameif inside
ASA(config-if)# ip address <Inside IP> <Inside Mask>

Outside Interface
ASA(config)# interface Vlan2
ASA(config-if)# description Primary ISP
ASA(config-if)# nameif outside
ASA(config-if)# ip address <ISP 1 IP> <ISP 1 Mask>

Backup ISP Interface
ASA(config)# interface Vlan3
ASA(config-if)# description Backup ISP
ASA(config-if)# nameif backupisp
ASA(config-if)# security-level 0
ASA(config-if)# ip address <ISP 2 IP> <ISP 2 Mask>

Assign them to switchports
ASA(config)# interface Ethernet0/0
ASA(config-if)# switchport access vlan 2
ASA(config)# interface Ethernet0/1
ASA(config-if)# switchport access vlan 3
Notes: VLAN 1 is the default so I’m not assigning it, just use one of the other ports for it.

Configure NAT
Define your Global pools
ASA(config)# global (outside) 1 <An IP in your Primary ISPs pool that you want to use for NAT/PAT>
ASA(config)# global (backupisp) 1 <An IP in your Backup ISPs pool that you want to use for NAT/PAT>
Notes: You need to define both the primary and backup address as global pools to match up against the NAT pool.  I totally forgot about that during the install and couldn’t figure out why I wasn’t passing traffic.

Define your inside NAT
ASA(config)# nat (inside) 1 0.0.0.0 0.0.0.0
Notes: Some people use a specific network here, I always just use 0 0 if its a small setup

Configure the SLA Monitor
ASA(config)# sla monitor 10
ASA(config-sla-monitor)# type echo protocol ipIcmpEcho 4.2.2.2 interface outside
ASA(config-sla-monitor-echo)# num-packets 3
ASA(config-sla-monitor-echo)# timeout 1000
ASA(config-sla-monitor-echo)# frequency 3
ASA(config)# sla monitor schedule 10 life forever start-time now
Notes: Ok, so here is the actual ‘failover’ piece of all of this. So I’ll break it down piece by piece. 
Line 1 – Configures a SLA monitor with the ID of 10
Line 2 – Configures the monitoring protocol and the target of the monitoring probe.  In this case I chose 4.2.2.2 since I have been able to ping that magical IP address since the beginning of time.  You also need to tell it which interface to source the ICMP traffic from.  In this case, it would be the outside interface. 
Line 3 – Sets the number of packets to be sent in each probe.
Line 4 – Configures the timeout period in milliseconds. 
Line 5 – Configures the frequency of the probe in seconds. 
Line 6 – Instructs the ASA to start SLA monitor 10 now and let it run for forever. 

Configure the Route tracking
ASA(config)#route outside 0.0.0.0 0.0.0.0 <ISP 1 Default route> 1 track 1
ASA(config)#route backupisp 0.0.0.0 0.0.0.0 <ISP 2 Default route> 254
Notes: Here we define the default network routes out to the internet.  Notice that we define out normal default route with an administrative distance of 1.  However, we also add the ‘track 1’ statement at the end.  This means that this route being in the routing table is dependant on tracked item 1 (If you don’t know what that means hold on, we’ll get there soon enough).  We also install a second route for the backup ISP which HAS to have a higher administrative distance than the primary ISP’s default route. 

ASA(config)# track 1 rtr 10 reachability
Notes: This is where the magic happens.  The above statement reads like this in plain English.  “Keep an eye on SLA monitor 10 and when it fails any routes associated with me also fail”.  So what happens is when the SLA monitor fails, the tracked route gets removed from the routing table, and the route with the higher administrative distance comes in and takes its place since its the best available route. 

Summary
So it’s a pretty cool setup if you are ONLY looking for outbound internet failover.  Keep in mind that all of your static NATs, external DNS entries, VPNs, etc won’t work when your primary ISP fails (assuming that’s the IP they are all assigned on).  This particular client had me make a primary and a backup PCF file for their Cisco VPN clients so that they could access the VPN when they were in a failover state.  Then I just added the backup ISP interface to the crypto map for their client VPN and turned on ISAKMP on the backup ISP interface.  Keep in mind though that the backup VPN will only work when the backup ISP circuit is live and the primary VPN will only work when the primary ISP circuit is live.  Both will never work at the same time.

I had the glorious opportunity today of making a Microsoft support call.  Im not sure how many of you out there have made one before, but I have to say its getting WAY better.  I had a call back in less than an hour, and the tech was very knowledgeable.  All said and done, they had the issue fixed in 30 minutes.  Whenever I watch another tech work (this time remotely) I like watching what shortcuts they use when they are doing their work.  I picked up a couple from this support call that I thought were worth sharing.  The first one, in my opinion is enough by itself.

Display your network adapters (Without going to the network and sharing center first!!!)
My biggest pet peeve with server 08 is that it takes more than 2 clicks to see your actual adapters. Not anymore, this shortcut loads the adapter screen in one command!
Command: ncpa.cpl
Usage: Start Menu,  Run, Type  ‘ncpa.cpl’ – Press <Enter>

Display Add/Remove Program (program and features)
Not a huge breakthrough, but it saves time all the same
Command: appwiz.cpl
Usage: Start Menu,  Run, Type  ‘appwiz.cpl’ – Press <Enter>

Show IIS (7)
Another time savor
Command: inetmgr
Usage: Start Menu,  Run, Type  ‘inetmgr’ – Press <Enter>

I’m mostly on the network side, so the first command is most useful to me.  Nonetheless its useful to know some shortcuts even with a GUI OS like Windows.  As a side note you can also run all of these commands out of command prompt but for shortcuts that don’t have the ‘.cpl’ extension you need to preface the command with the word ‘start’ (EG: ‘start inetmgr’)