An introduction to network namespaces

Network namespaces allow you to provide unique views of the network to different processes running on a Linux host.  If you’re coming from a traditional networking background, the closest relative to network namespaces would be VRF (Virtual Routing and Forwarding) instances.  In both cases the constructs allow us to provide a different network experience to different processes or interfaces.  For the sake of starting the conversation, let’s quickly look at an example of both VRFs and network namespaces so you get an idea of how they work.

The easiest scenario to illustrate either of these technologies is out of band management.  Take for instance this very simple network diagram…

     
Note: I’m being purposefully vague here about the network layout and addressing.  Bear with me for a moment while I get to the point. 

As you can see, we have two users that live on the same segment (forgive me for not drawing an Ethernet segment connecting the two).  Let’s assume that the user on the left has to traverse northbound to get to resources that hang off the top network cloud.  Let’s also assume the user on the right has to manage the router through the management interface.  Standard practice these days dictate that most network infrastructure be managed through a separate ‘out of band’ network.  So in the diagram above, we have two ‘data’ interfaces and one ‘mgmt’ interface.  We also have a default route pointing north and a 192 route pointing south.  Do you see the problem with the two different traffic flows we have?

The user on the left can route northbound following the default route to get to resources in the top network cloud.  When the resources send return traffic, the router routes them south as expected.  The user on the right routes directly to the router’s management interface, but then heads out the routers southbound data interface following the 192.168.1.0/24 route.  This is what we refer to as asymmetric routing.  While this isn’t always a bad thing in networking, there could be stateful network devices in path that won’t approve of the asymmetry.  Additionally, if we’re relying on the ‘data’ interface to return management traffic it sort of defeats the purpose of out of band management.  The problem is we can’t have a single routing table with the 192.168.1.0/24 route pointing out two different interfaces (you actually can but that would load balance the traffic and not fix the issue). 

So how is this solved?  We solve this by implementing a management VRF.  Each VRF is given it’s own routing table instance and IP interfaces can be joined to any available VRF.  So in this case, we can leave the two ‘data’ interfaces in the default VRF along with the route’s shown on the diagram.  In addition, we can create a management VRF, add the ‘mgmt’ interface to it, and add a duplicate 192.168.1.0/24 route pointing out of the management interface…

 
This solves our asymmetric problem and allows the router to be partitioned from a layer 3 (routing) point of view.  The VRF construct has been around networking for a long time and is widely used in the networking space.   

Note: I didn’t show an example config of VRFs because they’re so widely used.  If you’d like to see one take a look at this walk through on Jeremy Stretch’s blog.

So how do VRFs compare to network namespaces? Let’s use a similar example…

Here we have a server that has a data interface and a management interface (not ILO, that’s different hardware with a different routing table).  Lets say that the user on the left wants to access a web page hosted off of the data interface while the user on the right wants to manage the server off of the ‘mgmt’ interface.  So we have the same problem as we did with the router.  If we only have one routing table instance, we can’t solve this without asymmetrically routing the traffic.  Enter network name spaces.  Since we want to get our feet wet with network namespaces let’s take a look at a more concrete example…

  
Much like the router example from above, we’re going to simulate a server that has a data and a management interface.  This time however, we’re going to take things a step further to fully illustrate the power of network namespaces.  You’ll notice in this example that both the left and the right side of the diagram implement the exact same IP addressing.  We’ll assume that the server on the left wishes to access a web site hosted on the netnstest server and that the server on the right wishes to manage the netnstest server through SSH.  With the duplicate IP addressing on both sides of the diagram (and server) the only way we can make this work is with network namespaces. 

So let’s assume that the data connection is built in the default namespace.  Nothing special, just a standard interface configuration.  In my case, the configuration is stored in ‘/etc/sysconfig/network-scripts/ifcfg-ens18’ and looks like this…

Again – nothing special, a standard Linux network config file like you’re used to seeing.  At this point, the left server should be able to ping the netnstest server’s data interface as well as access the super cool webpage it is hosting…

Alright – so the normal network stuff is working.  So now let’s add the mgmt namespace.  To do that, let’s think about how we’d build a normal IP interface in Linux.  We’d probably do something like this…

#Turn up the link
ip link set ens19 up

#Configure the mgmt IP address
ip addr add 10.20.30.1/24 dev ens19

#Configure the default route
ip route add default via 10.20.30.20

Pretty straightforward right?  So how do we accomplish the same thing with network namespaces?  We do this…

#Create the mgmt network namespace
ip netns add mgmt

#Add the physical NIC to the namespace
ip link set ens19 netns mgmt

#Turn up the link
ip netns exec mgmt ip link set ens19 up

#Configure the mgmt IP address
ip netns exec mgmt ip addr add 10.20.30.1/24 dev ens19

#Configure the default route for the mgmt namespace
ip netns exec mgmt ip route add default via 10.20.30.20

The difference really boils down to the first two steps.  First, we have to create the mgmt network namespace.  Second, we have to add the physical interface that we want to use (ens19) for management to the network namespace.  After that, the only configuration difference is that we’re executing the Linux network commands within the namespace using the ‘ip netns exec mgmt’ syntax.

As you might have guessed, the ‘ip netns’ command is what we use to interact with Linux network namespaces from the shell.  If we look at the command help, we can see we have a few basic options…

The big hitters are add, delete, list, and exec.  The first three should speak for themselves.  Exec is the more interesting option and allows us to execute commands from within a specified network namespace.  We’ll see a few more examples of using the exec syntax in just a moment.  So let’s enter the above commands to build our management interface and then make sure it’s working.  First let’s make sure that our data interface still looks good…

Awesome, so now let’s try some tests from the right server to see what we get…

Looking good!  Now let’s try to manage the server by SSHing into it…

 
Huh.  So that didn’t work.  If we think about this for a second, the SSH daemon wasn’t told to run in the management namespace.  It was already running in the default namespace.  So for this to work we have to run a copy of the SSH daemon in the ‘mgmt’ namespace.  We can do this by using the following command…

ip netns exec mgmt /usr/sbin/sshd -o PidFile=/run/sshd-mgmt.pid

If we run that on the netnstest host, we should now be able to SSH into the server from the server on the right…

 
As one last piece of evidence that this is working let’s telnet to the left and right router from the netnstest server itself…

 
As you can see, when I telnet to 10.20.30.1 I get connected to the left router.  When I telnet to 10.20.30.1 using the ‘netns exec’ command from the ‘mgmt’ network namespace, I get connected to the right router.

We should make note at this point that network namespaces are not persistent.  That is, if I reboot the netnstest server I’ll lose all of this configuration.  The fix for that would be to create a startup script which builds the management interface each time the server boots.  As part of that script, you’ll need to start a copy of any services (SSH) that you wish to use in that network namespace. 

Interestingly enough, using network namespace for management interfaces isn’t the reason I wrote this blog.  The examples we went through above were designed to give you a base level understanding of how a network namespace can function.  While I’m not to saying that the management use case is invalid, the really interesting use cases for network namespaces lie in the virtual and container spaces.  In the next post on this topic we’ll be talking about how Docker uses network namespaces for container network isolation and connectivity. 

In the mean time – take a look at Matt Oswalt’s recent blog where he describes how namespaces are the new network access layer.