Simulating latency and packet loss on a Linux host

Every once and a great while there is a need to simulate bad network behavior.  Simulating things like packet loss and high latency are often harder to do than you’d expect.  However – if you’re dealing with a Linux host – there’s a simple solution.  The tc command which comes along with the iproute2 toolset can help you simulate symptoms of a bad network quite easily.

The tc command offers a lot of functionality but in this post we’re just going to walk through a couple of quick examples of using it in conjunction with the netem (network emulator) included in the Linux kernel .  To do this, we’ll use just use two hosts…

To start with – let’s make sure that tc is installed and that it’s working…

user@ubuntu-1:~$ sudo tc qdisc show dev ens32
qdisc pfifo_fast 0: root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
user@ubuntu-1:~$

So what did we just do here? Well we used the tc command to return the current qdisc configuration for our servers physical network interface named ‘ens32’.  So what’s a qdisc?  Qdisc is shorthand for ‘Queue discipline’ and defines the queuing method assigned to the interface. In our case we can see that this interface is using the ‘pfifo_fast’ queuing methodology. This is a default configuration and you should see something similar on your Linux host.

Side Note: Im doing all of this in a lab.  Make sure you know what you’re changing before you change anything.  For instance, configuring a high rate of packet loss on an interface you’re using to SSH into the host is likely not a great idea.  Common sense rules apply.

Let’s first start a constant ping on our host ubuntu-5 heading toward ubuntu-1. Now let’s head back to ubuntu-1 and configure the following command…

sudo tc qdisc add dev ens32 root netem delay 200ms

So let’s break this command down. We’re telling tc that we want to work with the interface ens32 and add a delay of 200ms to the root qdisc. The root qdisc is an egress queue and where all the packets will inherently get queued by default. If we go and check the ping on the other server we should see the latency has increased…

64 bytes from ubuntu1604-1.interubernet.local (10.20.30.71): icmp_seq=14 ttl=63 time=0.426 ms
64 bytes from ubuntu1604-1.interubernet.local (10.20.30.71): icmp_seq=15 ttl=63 time=0.356 ms
64 bytes from ubuntu1604-1.interubernet.local (10.20.30.71): icmp_seq=16 ttl=63 time=0.399 ms
64 bytes from ubuntu1604-1.interubernet.local (10.20.30.71): icmp_seq=17 ttl=63 time=0.578 ms
64 bytes from ubuntu1604-1.interubernet.local (10.20.30.71): icmp_seq=18 ttl=63 time=0.619 ms
64 bytes from ubuntu1604-1.interubernet.local (10.20.30.71): icmp_seq=19 ttl=63 time=200 ms
64 bytes from ubuntu1604-1.interubernet.local (10.20.30.71): icmp_seq=20 ttl=63 time=200 ms
64 bytes from ubuntu1604-1.interubernet.local (10.20.30.71): icmp_seq=21 ttl=63 time=200 ms
64 bytes from ubuntu1604-1.interubernet.local (10.20.30.71): icmp_seq=22 ttl=63 time=200 ms

Great! Now let’s modify the rule with this command…

sudo tc qdisc change dev ens32 root netem delay 200ms 50ms

This command tells the rule to include a random deviation of up to 50ms. Once the rule is in, you should start seeing latency number in between 150ms and 250ms…

64 bytes from ubuntu1604-1.interubernet.local (10.20.30.71): icmp_seq=23 ttl=63 time=200 ms
64 bytes from ubuntu1604-1.interubernet.local (10.20.30.71): icmp_seq=24 ttl=63 time=200 ms
64 bytes from ubuntu1604-1.interubernet.local (10.20.30.71): icmp_seq=25 ttl=63 time=211 ms
64 bytes from ubuntu1604-1.interubernet.local (10.20.30.71): icmp_seq=26 ttl=63 time=219 ms
64 bytes from ubuntu1604-1.interubernet.local (10.20.30.71): icmp_seq=27 ttl=63 time=191 ms
64 bytes from ubuntu1604-1.interubernet.local (10.20.30.71): icmp_seq=28 ttl=63 time=189 ms
64 bytes from ubuntu1604-1.interubernet.local (10.20.30.71): icmp_seq=29 ttl=63 time=205 ms
64 bytes from ubuntu1604-1.interubernet.local (10.20.30.71): icmp_seq=30 ttl=63 time=157 ms
64 bytes from ubuntu1604-1.interubernet.local (10.20.30.71): icmp_seq=31 ttl=63 time=195 ms
64 bytes from ubuntu1604-1.interubernet.local (10.20.30.71): icmp_seq=32 ttl=63 time=167 ms
64 bytes from ubuntu1604-1.interubernet.local (10.20.30.71): icmp_seq=33 ttl=63 time=222 ms

Perfect! Having that random deviation helps make it look like ‘real’ latency. Now let’s delete the rule like so…

sudo tc qdisc del dev ens32 root netem delay 200ms 50ms

Its important to keep track of the add|change|del keywords. If you try to change a rule that doesnt exist or something similar you’re going to start getting weird errors when you try to work with the rules.

Next up – let’s introduce some packet loss!

sudo tc qdisc add dev ens32 root netem loss 20%

The command is similar to the latency commands but now we’re specifying ‘loss’ instead of ‘delay’. If we send 10 ICMP pings to ubuntu-1 from ubuntu-5 we should see some packet loss…

user@ubuntu-5:~$ ping ubuntu-1 -c 10
PING ubuntu-1.interubernet.local (10.20.30.71) 56(84) bytes of data.
64 bytes from ubuntu1604-1.interubernet.local (10.20.30.71): icmp_seq=1 ttl=63 time=0.222 ms
64 bytes from ubuntu1604-1.interubernet.local (10.20.30.71): icmp_seq=2 ttl=63 time=0.663 ms
64 bytes from ubuntu1604-1.interubernet.local (10.20.30.71): icmp_seq=4 ttl=63 time=0.496 ms
64 bytes from ubuntu1604-1.interubernet.local (10.20.30.71): icmp_seq=5 ttl=63 time=0.303 ms
64 bytes from ubuntu1604-1.interubernet.local (10.20.30.71): icmp_seq=6 ttl=63 time=0.538 ms
64 bytes from ubuntu1604-1.interubernet.local (10.20.30.71): icmp_seq=7 ttl=63 time=0.585 ms
64 bytes from ubuntu1604-1.interubernet.local (10.20.30.71): icmp_seq=8 ttl=63 time=0.456 ms
64 bytes from ubuntu1604-1.interubernet.local (10.20.30.71): icmp_seq=9 ttl=63 time=0.354 ms

--- ubuntu-1.interubernet.local ping statistics ---
10 packets transmitted, 8 received, 20% packet loss, time 8998ms
rtt min/avg/max/mdev = 0.222/0.452/0.663/0.140 ms
user@ubuntu-5:~$

Yep, so 20% packet loss as we expected. When you’re done testing, dont forget to delete the rule…

sudo tc qdisc delete dev ens32 root netem loss 20%

One interesting thought to point out here is that you could quite easily build a server that acted as a sort of ‘WAN simulator’ to use in your lab.  In that case, if the server was inline with two interface you could enact policy in both flow directions by applying specific policy to each interface.  Lots of possibilities there!

This is literally just the tip of the iceberg. There are so many more things you can do with tc and netem.  If you’re looking for more info here’s a good starting point.  Be aware that the in-document hyperlinks don’t appear to work but you can just scroll down to see all of the examples.

Update: Heres another link with better info about TC and NetEm.  Thanks John!