You are currently browsing articles tagged TISHK.

I’ve been wanting to write this blog for awhile, but I’ve been behind on blogs (as you can likely see).  At any rate, I had the task to move a port-channel a couple of weeks ago and I developed a procedure for doing so in a ‘hitless’ fashion.  Actually, it tied in nicely to my CCIE studies which are still ,unfortunately, stuck on spanning tree.  

So here’s what the network looks like to start with…


The environment is a internet facing VPN DMZ.  It’s totally redundant, but the firewalls are in a active standby pair.  That means, that the VPN clusters that are hung off the right switch, need to traverse the trunk between the switches to talk to the internet (depicted by red dotted line below)


Normally, I’d just schedule an outage to shut down the old port-channel and turn the new one up.  However, this environment supports a lot (I mean a ton) of IPSec VPN sessions worldwide so I didn’t really want to kick anyone off.  As an additional challenge ,the existing port-channel was set to ‘mode on’ which needed to be changed over to LACP.  So I started doing some lab work to see what I could come up with.  In the end, I came up with the following procedure. 

Verify which switch is the spanning tree root
Since I hadn’t done any ‘fancy’ spanning-tree config, the same switch was the root for all the VLANs that spanned the port-channel trunk.  In my case, the left switch was the root.  This is important because all of the ports on the root switch are in forwarding mode.

Configure the port-channel on the non-root switch
I started by configuring the first end of the new port-channel on the non-root switch.  My config looked something like this…

interface range gi3/23 – 24
no shut
channel-group 10 mode on

int po10
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1,10,20,30
switchport mode trunk
spanning-tree cost 50
no shut

spanning-tree uplinkfast

In addition to configuring the port-channel, I did a couple of other steps.  I set the spanning-tree cost of the new port-channel (po10) to 50.  This guarantees that spanning-tree will see it as being less desirable than the existing 2x Gig port-channel.  This step is critical so that you can control when the switch decides to use the new link.  I also configured uplinkfast on the switch.  Uplinkfast is a Cisco feature that allows an alternate port to immediately transition to forwarding when the existing forwarding link fails.  If this feature isn’t on, the switch can detect that the forwarding interface failed, but then has to transition the alternate link through the spanning tree stages (listening and learning) before the alternate can forward traffic.  Cisco documentation on the uplinkfast.

Configure the port-channel on the root switch
This is the configuration I used on the root-switch.

interface range gi3/22 – 24
no shut
channel-group 10 mode on

int po10
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1,10,20,20
switchport mode trunk
spanning-tree cost 50
no shut

Verify Spanning Tree is acting the way it should be
If you’ve done everything right at this point, things should look like this…


The legacy port-channel is still forwarding traffic and non-root switch see’s the new port-channel, but it’s in blocking mode.  It’s blocking the new port-channel since the port-cost is significantly higher than the existing port-channel.  However, even though it’s blocking, the switch knows that this is a safe alternate to the existing path.  Make sure you wait for the new port to go through the spanning-tree port states before you move on!

Shutdown the legacy port-channel
On the non-root switch shutdown the legacy port-channel.  In my case, that was port-channel 1.

int po1

The instant that the switch detects that po1 is down, it immediately starts forwarding over the alternate link which happens to be our new port-channel.


When it was all said and done monitoring hadn’t detected a single drop during the cutover.  I confirmed that I hadn’t dropped any traffic between the right VPN cluster and the outside world.

At this point, I cleaned up the config for the legacy port-channel and removed the spanning-tree config I put in to facilitate the change.

Studying for the CCIE has really opened my eyes to a lot of interesting things that I never even thought about before.  It’s pretty easy to spend so much time reading about new things (DC fabrics, SDN, etc) and forget about the basic. 

I’ve got another design that I’m about to use that leans on spanning-tree again.  I’ll be sure to post that once I get done with the implementation.

Tags: ,

So I’m one of those ‘why’ guys.  If you tell me I can’t do something I have to know why.  So when someone was telling me about a recent loop they encountered because of a port-channel configuration issue, I had to know the how, what, why and where.

As we all know, we should always configuring switch to switch port-channels with ‘mode active’.  It’s better to allow the switch to determine what links should be in the bundle rather than force it.  However, there are still many ‘mode on’ configurations left out there and I run across them once in awhile. 

I’m not going to dive into the details here on how LACP (802.3ad) works.  However, you should know that there are only 3 valid combinations of settings that work to form a port-channel.  They are…

Active – Active

Active – Passive

On – On

Active – The switch will openly try and negotiate a LACP channel
Passive – The switch will participate in a LACP channel, but won’t start the conversation
On – On.  Always on.
So the risk here is that one side of a port-channel could be set to on, and the other side could be set to active.  If that’s the case, the active side won’t ever negotiate the link.  So is that a problem?  Yep, it sure is.  Let’s look at an example and see what happens…

So in this case, we have switch 1 configured to start LACP negotiations.  Unfortunately, switch 2 doesn’t care.  He’s in ‘mode on’ and is going to form a port-channel regardless of what happens.  So let’s configure each side and see what happens.  Switch 2’s port-channel loads right away and shows up.  Switch 1’s port channel attempts to negotiate but fails since switch 2 doesn’t care.  At this point, the ports on Switch 1 become ‘independent’ which effectively means that they are just physical ports connected to switch 2. 

Now, let’s go back and talk about spanning-tree.  Let’s run through the two possibilities here…

Switch 1 is the root
Let’s say that switch 1 happens to be the root of our spanning tree.  After we configure the port-channels, we notice the following errors on each switch.



So it looks like both switches shut down their interfaces that were supposed to be part of the port-channel.  You can see that switch 2 detected a channel-mis-configuration (STP) right before the ports went down.  If switch 1 is the root, it will send hello’s out all of it’s ports.  In this case spanning-tree doesn’t see a single port-channel, it sees two physical ports.  So spanning-tree sends the hello out both ports facing switch 2.  image

Switch 2 thinks that it should have received the hello over only one of the links in what it perceives to be a port-channel interface.  Since it received it over more than one, it knows that there is an issue and it put’s it’s ports in errDisable mode. At this point, the link between the switches it completely down.  

Switch 2 is the root
So now let’s see what happens when switch 2 starts as the root switch.  In this case, switch 2 forwards it’s hellos out all of it’s ports.  However, spanning-tree only see’s one port (po1) going to switch 1.  So switch 2 is forwarding hellos out of port-channel interface as well as the interface towards the workstation.  See below…



So at this point, spanning-tree doesn’t see a problem.  Switch 1 is receiving the hellos and forwarding them out all of it’s interfaces with the exception of the one it received it on.  So it would look something like this…

Now that doesn’t look so bad does it?  Seems like fairly normal operation.  But that only shows the initial hellos the root bridge sends.  Take a look at what happens when host starts sending broadcast traffic…


In step 1, the host sends a packet with a local broadcast destination (  Switch 2 receives the broadcast and forwards it out all of it’s remaining interfaces (PO1) in step 2.  At this point, switch 1 receives the broadcast on one of the two links that switch 2 believes to be part of it’s port-channel interface.  In step 3, switch 1 forwards it out all of the interfaces except the one that it heard the broadcast on.  This would include the link to the host as well as the second link that switch 2 thinks is in it’s port-channel.  The end result?


Switch 2 is suffering from CAM table stability issues.  It’s hearing a MAC address from out of the port facing the host (Gi1/0/1) as well as the port-channel (Po1).  Now imagine this on a large scale with more than one host.  That switch could easily become overwhelmed and tip over in such a scenario. 

So as you can see, it’s best to have both sides negotiate.  If you don’t you can run into some pretty serious issues. 


So it’s not that I didn’t know spanning-tree, it’s more that I don’t know it well enough for where I’m at in my career.  To be frank, I stay away from it.  I design networks specifically to avoid having to deal with it.  Layer 3 to the edge is how I like things done.  At any rate, here’s a crack at getting a better understanding of it. 

Spanning-tree Protocols
802.1d (Standard Spanning-tree)

So the entire goal of spanning-tree is to create a loop free layer 2 domain.  There is no TTL in a layer 2 frame so if you don’t have spanning-tree, a frame can loop forever.  So the original 802.1d standard set out to fix this.  There are a few main pieces to the 802.1d process.  They are…

1. Elect a root bridge. 
This bridge is the ‘root’ of the spanning-tree.  In order to elect a root bridge, all of the switches send out BPDU (Bridge Protocol Data Units).  The BPDU has a bridge priority in it which the switches use to determine which switch should be the root.  The lowest ID wins.  The original standard specified a bridge ID as…


As time progressed there became a need to create multiple spanning-trees for multiple VLANs (we’ll get to that later).  So, the bridge ID format had to be changed.  What they came up with was..


So, now you know why you need to have a bridge priority that’s in multiples of 4096 (if you don’t.. A total of 4 bits gives you a total of 16 values, 16 * 4096 gives you 65,536 which is the old bridge priority max value – 1).

So at this point, we have a mess of switches swarming around with BPDUs.  If a switch receives a BPDU with a lower bridge priority it knows that it isn’t the root.  At that point, it stops sending out it’s own bridge ID and starts sending out BPDUs with the better (lower) priority that it heard of.  In the end, all of the switches will be forwarding BPDUs with the lowest bridge ID.  At that point, the switch originating the best(lowest) bridge ID knows that it is the root bridge. 

2. Each switch selects a root port
So now that we know which switch is the root, every non-root switch needs to select it’s root port.  That is, the port with the lowest cost to the root switch.  To determine this, the root port sends ‘Hellos’ out of all of it’s port every 2 seconds.  When a non-root switch receives the hello, it does a couple of things.  First, it reads the ‘cost’ from the hello message and updates it by adding the port cost.  So if a hello came in a fast Ethernet port with a cost of 4, the switch would add 19 to it giving you a new cost of 23.  After all of the hellos are sent, the switch picks it’s root port by selecting the port which had the lowest calculated cost.  Now, a bit about port costs.  See the table below…

Interface Speed Original IEEE Port Cost New IEEE port Cost
10 Mbps 100 100
100 Mbps 10 19
1000 Mbps 1 4
10000 Mbps 1 2

So as you can see, with the increase in speed came a upgrade to the port costs.  Now that we have 40 gig interfaces I’m wondering if they will redo that again.  At any rate, if there is a tie, say two ports that have a calculated cost of 23.  The switch breaks the tie in the following fashion..

1. Pick the lowest bridge ID of switch that sent the hellos
2. Pick the lowest port priority of the switch that sent the hellos
3. Use the lowest port number of the switch that sent the hellos

(We’ll talk about port priorities in a bit)  Now that we have a root port we can move onto step 3.

3.  Pick a designated port
This part is pretty easy.  Basically, each segment can only have one designated port.  The switch that forwards the lowest cost hello onto a particular segment becomes the designated switch and the port that it uses to do that is the designated port.  So, that would mean that each port on the root bridge would be a designated port.  Then, ports that are neither root ports or designated ports (non-designated ports) go into blocking state.  If a tie occurs, the same tiebreaker process occurs as in step 2. 

At this point, we have a fully converged spanning-tree!

Normal Operation
Under normal operation the root sends hellos out of all it’s active ports.  Each connected switch receives the hellos on their root ports, updates it, and forwards it out of it’s designated port (if it has one).  Blocked ports receive the hellos, but never forward them. 

Topology Changes
When a switch notices a topology change, it’s responsible for telling all other connected switches about the change.  The most effective way to do this, is to tell the root switch so that it can tell all of the other switches.  When a switch notices a topology change, it sends a TCN (topology change notification) out it’s root port.  The switch will send the TCN every hello time until the upstream switch acknowledges it.  The upstream switch acknowledges by sending a hello with a TCA (topology change acknowledgement).  This process continues until the root becomes notified.  The root will then set the TC flag on it’s hellos.  When switches in the tree see the TC set in the hello from the root, they know that there has been a topology change and that they need to age out their CAM tables.  Switches aging out their CAM tables is an important part of a topology change and reconvergence. 

802.1D Port States
Blocking – The port is blocking all traffic with the exception of receiving STP BPDUs.  The port will not forward any frames in this state.

Listening – Same as blocking but will now begin to send BPDUs. 

Learning – The switch will begin to learn MAC information in this state.

Forwarding – Normal full up and up port state.  Forwarding normal traffic.

There are a couple of main timers in the STP protocol.  These are..
Forward Delay Timer – Default of 15 seconds
Hello – Default of 2 seconds
MaxAge – Default of 20 seconds

Spanning-Tree enhancements (Cisco Proprietary)
– Immediately puts a port into forwarding mode.  Essentially disables the STP process.  Should only be used for connecting to end hosts.
UplinkFast – Should be used on access layer switches connecting to distribution.  Used to fail over the root port in the case of the primary root port failing.  CAM entries are timed out by the access layer generating multicast frames with attached devices MACs as the source for the frames.  This is different than the normal TCN process as described earlier.  UplinkFast also causes the switch to increase the root priority to 49152 and set all of the ports costs to 3000.
BackboneFast – Used to detect indirect STP failures.  This way the switch doesn’t have to wait MaxAge to reconverge.  The feature needs to be configured on all switches in order for it to work.  The switch queries it’s upstream switches when it sops receiving hellos with a RLQ (Root Link Query).  If the upstream switch had a failure it can reply to the local switch so that it can converge to another port without waiting for the MaxAge to expire. 

802.1w (Rapid Spanning-Tree)
Rapid spanning-tree takes 802.1d and makes it faster.  In addition, they take some of the Cisco proprietary features and standardize them.  Here are some of the notable changes that 802.1w makes.

-Switches only wait to miss 3 hellos on their root port prior to reconverging.  This number in 802.1d was 10 (MaxAge, or 10 times hello).

-Fewer port states.  802.1w takes the number of port states from 5 (Im counting disabled) down to 3.  The new states are discarding, learning, and forwarding.

-Concept of a backup DP when a switch has multiple ports connected to the same segment.

-Standardization of the Cisco proprietary PortFast, UplinkFast, and BackboneFast.

802.1w Link Types
Point to Point – Connects a switch to another switch in full duplex mode. 
Shared – Connects a switch to a hub using half duplex
Edge – A user access port

802.1w Port roles
Root Port – The same as in 802.1d
Designated Port – The same as in 802.1d
Alternate Port – Same as the uplink fast feature, backup RP connection
Backup Port – Alternate DP port, can take over if the existing DP fails

802.1s (Multiple Spanning-Tree)
Multiple spanning-tree (MST) lets you map VLANs into a particular spanning tree.  These VLANs are then considered to be part of the same MST region.  MST uses the same features as RSTP for convergence, so if you are running MST, you are by default also running RSTP.  Much like any other ‘group’ technology, there are several parameters that must be met before switches/vlans can become part of the same region.

-MST must be globally enabled
-The MST region name must be configured (and the same on each switch)
-Define the MST revision number (and make it the same on each switch)
-Map the same VLANs into each region (or instance)

MST can con-exist with other switches that don’t talk MST.  In this case, the entire MST region appears to be a single switch to the other ‘external’ spanning-tree.  The spanning-tree that connects the region to the ‘outside’ is considered to be the IST, or Internal Spanning Tree. 

Spanning-tree Protection
There are several ‘protection’ mechanisms available that can be implemented in conjunction with spanning-tree to protect the spanning-tree from failure or loops.

BPDU Guard – Should be enabled on all ports that will never connect to anything but an end user port.  The configuration will err-disable a port if a BPDU is received on that port.  To recover from this condition the port must be shut/no shut.

Root Guard – Protects the switch from choosing the wrong RP.  If a superior BPDU is heard on this port the port is placed into root-inconsistent state until the BPDUs are no longer heard. 

UDLD – Unidirectional link detection is used to detect when one side (transmit or receive) is lost.  States like this can cause loops and loss of connectivity.  UDLD functions in two modes, aggressive and normal.  Normal mode uses layer 2 messaging to determine if a switches transmission capabilities have failed.  If this is detected, the switch with the failed transmit side goes into err-disable.  In aggressive mode the switch tries to reconnect with the other side 8 times.  If this fails, both sides go into err-disable.

Loop Guard – When a port configured with loop guard stops hearing BPDUs it goes into loop-inconsistent state rather than transitioning into forwarding. 

Tags: ,

« Older entries