I’ve been wanting to write this blog for awhile, but I’ve been behind on blogs (as you can likely see). At any rate, I had the task to move a port-channel a couple of weeks ago and I developed a procedure for doing so in a ‘hitless’ fashion. Actually, it tied in nicely to my CCIE studies which are still ,unfortunately, stuck on spanning tree.
So here’s what the network looks like to start with…
The environment is a internet facing VPN DMZ. It’s totally redundant, but the firewalls are in a active standby pair. That means, that the VPN clusters that are hung off the right switch, need to traverse the trunk between the switches to talk to the internet (depicted by red dotted line below)
Normally, I’d just schedule an outage to shut down the old port-channel and turn the new one up. However, this environment supports a lot (I mean a ton) of IPSec VPN sessions worldwide so I didn’t really want to kick anyone off. As an additional challenge ,the existing port-channel was set to ‘mode on’ which needed to be changed over to LACP. So I started doing some lab work to see what I could come up with. In the end, I came up with the following procedure.
Verify which switch is the spanning tree root
Since I hadn’t done any ‘fancy’ spanning-tree config, the same switch was the root for all the VLANs that spanned the port-channel trunk. In my case, the left switch was the root. This is important because all of the ports on the root switch are in forwarding mode.
Configure the port-channel on the non-root switch
I started by configuring the first end of the new port-channel on the non-root switch. My config looked something like this…
interface range gi3/23 – 24
no shut
switchport
channel-group 10 mode on
int po10
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1,10,20,30
switchport mode trunk
spanning-tree cost 50
no shut
spanning-tree uplinkfast
In addition to configuring the port-channel, I did a couple of other steps. I set the spanning-tree cost of the new port-channel (po10) to 50. This guarantees that spanning-tree will see it as being less desirable than the existing 2x Gig port-channel. This step is critical so that you can control when the switch decides to use the new link. I also configured uplinkfast on the switch. Uplinkfast is a Cisco feature that allows an alternate port to immediately transition to forwarding when the existing forwarding link fails. If this feature isn’t on, the switch can detect that the forwarding interface failed, but then has to transition the alternate link through the spanning tree stages (listening and learning) before the alternate can forward traffic. Cisco documentation on the uplinkfast.
Configure the port-channel on the root switch
This is the configuration I used on the root-switch.
interface range gi3/22 – 24
no shut
switchport
channel-group 10 mode on
int po10
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1,10,20,20
switchport mode trunk
spanning-tree cost 50
no shut
Verify Spanning Tree is acting the way it should be
If you’ve done everything right at this point, things should look like this…
The legacy port-channel is still forwarding traffic and non-root switch see’s the new port-channel, but it’s in blocking mode. It’s blocking the new port-channel since the port-cost is significantly higher than the existing port-channel. However, even though it’s blocking, the switch knows that this is a safe alternate to the existing path. Make sure you wait for the new port to go through the spanning-tree port states before you move on!
Shutdown the legacy port-channel
On the non-root switch shutdown the legacy port-channel. In my case, that was port-channel 1.
int po1
shutdown
The instant that the switch detects that po1 is down, it immediately starts forwarding over the alternate link which happens to be our new port-channel.
When it was all said and done monitoring hadn’t detected a single drop during the cutover. I confirmed that I hadn’t dropped any traffic between the right VPN cluster and the outside world.
Cleanup
At this point, I cleaned up the config for the legacy port-channel and removed the spanning-tree config I put in to facilitate the change.
Summary
Studying for the CCIE has really opened my eyes to a lot of interesting things that I never even thought about before. It’s pretty easy to spend so much time reading about new things (DC fabrics, SDN, etc) and forget about the basic.
I’ve got another design that I’m about to use that leans on spanning-tree again. I’ll be sure to post that once I get done with the implementation.
