Why does my traceroute look like that?

So here’s another common question I see from other engineers.  Goes something like this…

"I ran a traceroute from the switch and I’m getting all sorts of goofy responses.  Lines that show multiple hops and hops that definitely shouldn’t appear where they do.  What’s going on?"

What they are referring to is the dreaded multiple response per hop output…

MLS# traceroute 192.168.5.1
Type escape sequence to abort.
Tracing the route to 192.168.5.1
1 172.172.172.26 464 msec
  172.172.172.22 360 msec
   172.172.172.26 80 msec
2 172.172.172.2 960 msec
   172.172.172.6 1004 msec
   172.172.172.2 1216 msec
3 172.172.172.14 1216 msec 1680 msec 1852 msec

So you might look at this and be entirely confused.  Or you might look at it and know that you probably should have ran the trace from a workstation rather than a MLS but didn’t know why you should have (The boat I was in a few years ago).  Let’s dig in and see why we get output like this. 

First off, you need to know traceroute works.  If you don’t (you should) I’ll explain it in quickly.  The machine generating the traceroute sends packets towards the destination incrementing the packets TTL with each new packet.  Basically, it starts with a TTL of 1.  The next hop router (the next layer 3 hop) get’s the packet, decrements the TTL by one and sends it on it’s way.  If the resulting TTL is 0, it can’t send it on it’s way so it instead returns a ‘TTL Exceeded’ message back to the host that sent the packet.  When the machine generating the traces get’s the response it sends another packet towards the destination with a TTL of 1 larger than the last packet.  This way, the host generating the traceroute get’s a response back from each layer 3 hop on it’s way to the destination.  Make sense?

So now that we know that, what’s going on with the output we are getting?  Basically, the host has two equal cost paths to the destination you are trying to reach.  At this point, a smart network engineer would say "Ok, but doesn’t the MLS use the CEF default of per destination load balancing?".  Ah ha!  You’ve hit the nail on the head here!  CEF defaults to per destination load balancing.  This ensures that traffic from the same source, headed to the same destination always takes the same path (saving us from out of order packets).  So that being said, this still doesn’t make sense.  If each probe takes the same path, then why are we getting two different paths at the first two hops? 

The answer is so simple it initially baffled me.  I can’t help myself here, so let me give you one more clue before I give you the answer.  If the same traceroute was ran from a standalone device plugged into this same MLS, we would get a traceroute with the same hop for each probe per TTL (in other words, a normal looking traceroute).  Did that help?

Ok, ok.  So the difference is that in this case, we are running the trace from the MLS itself.  That being said, we aren’t technically using the CEF table to do destination lookups.  Since the MLS originates the packets, it process switches  them.  When the traceroute probes are process switched, they default to the default process switching load balancing method which is per-packet.  Since the host we are generating the packets from has two equal cost paths, per packet load balancing sends one probe one way, the second the second way, and the third the first way again.  The same goes on for each subsequent hop.  If we generated the trace from a PC plugged into this MLS we wouldn’t have two equal cost paths.  The MLS does, but it’s just going to do per-destination load balancing on the trace probes that I send.  So I wont see different results for each probe.

Does that make sense?  The bottom line is that there is significant difference between a switch forwarding a packet, and a switch generating a packet.