You are currently browsing the archive for the Linux category.

If you’ve been paying attention to the discussions around container networking you’ve likely heard the acronym CNI being used.  CNI stands for Container Networking Interface and it’s goal is to create a generic plugin-based networking solution for containers.  CNI is defined by a spec (read it now, its not very long) that has some interesting language in it.  Here are a couple of points I found interesting during my first read through…

  • The spec defines a container as being a Linux network namespace.  We should be comfortable with that definition as container runtimes like Docker create a new network namespace for each container.
  • Network definitions for CNI are stored as JSON files.
  • The network definitions are streamed to the plugin through STDIN.  That is – there are no configuration files sitting on the host for the network configuration.
  • Other arguments are passed to the plugin via environmental variables
  • A CNI plugin is implemented as an executable.
  • The CNI plugin is responsible wiring up the container.  That is – it needs to do all the work to get the container on the network.  In Docker, this would include connecting the container network namespace back to the host somehow.
  • The CNI plugin is responsible for IPAM which includes IP address assignment and installing any required routes.

If you’re used to dealing with Docker this doesn’t quite seem to fit the mold.  It’s apparent to me that the CNI plugin is responsible for the network end of the container, but it wasn’t initially clear to me how that was actually implemented.  So the next question might be, can I use CNI with Docker?  The answer is yes, but not as an all in one solution.  Docker has it’s own network plugin system called CNM.  CNM allows plugins to interact directly with Docker.  A CNM plugin can be registered to Docker and used directly from it.  That is, you can use Docker to run containers and directly assign their network to the CNM registered plugin.  This works well, but because Docker has CNM, they dont directly integrate with CNI (as far as I can tell).  That does not mean however, that you can’t use CNI with Docker.  Recall from the sixth bullet above that the plugin is responsible for wiring up the container.  So it seems possible that Docker could be the container runtime – but not handle the networking end of things (more on this in a future post).

At this point – I think its fair to start looking at what CNI actually does to try to get a better feel for how it fits into the picture.  Let’s look at a quick example of using one of the plugins.

Let’s start by downloading the pre-built CNI binaries…

Ok – let’s make sure we understand what we just did there.  We first created a directory called ‘cni’ to store the binaries in.  We then used the curl command to download the CNI release bundle.  When using curl to download a file we need to pass the ‘O’ parameter to tell curl to output to a file.  We also need to pass the ‘L’ parameter in this case to allow curl to follow redirects since the URL we’re downloading from is actually redirecting us elsewhere.  Once downloaded, we unpack the archive using the tar command.

After all of that we can see that we have a few new files.  For right now, let’s focus on the ‘bridge’ file which is the bridge plugin.  Bridge is one of the included plugins that ships with CNI.  It’s job, as you might have guessed, is to attach a container to a bridge interface.  So now that we have the plugins, how do we actually use them?  One of the earlier bullet points mentioned that network configuration is streamed into the plugin through STDIN.  So we know we need to use STDIN to get information about the network into the plugin but that’s not all the info the plugin needs.  The plugin also needs more information such as the action you wish to perform, the namespace you wish to work with, and other various information.  This information is passed to the plugin via environmental variables.  Confused?  No worries, let’s walk through an example.  Let’s first define a network configuration file we wish to use for our bridge…

Above we create a JSON definition for our bridge network.  There are some CNI generic definitions listed above as well as some specific to the bridge plugin itself.  Let’s walk through them one at a time.

CNI generic parameters

  • cniVersion: The version of the CNI spec in which the definition works with
  • name: The network name
  • type: The name of the plugin you wish to use.  In this case, the actual name of the plugin executable
  • args: Optional additional parameters
  • ipMasq: Configure outbound masquerade (source NAT) for this network
  • ipam:
    • type: The name of the IPAM plugin executable
    • subnet: The subnet to allocate out of (this is actually part of the IPAM plugin)
    • routes:
      • dst: The subnet you wish to reach
      • gw: The IP address of the next hop to reach the dst.  If not specified the default gateway for the subnet is assumed
  • dns:
    • nameservers: A list of nameservers you wish to use with this network
    • domain: The search domain to use for DNS requests
    • search: A list of search domains
    • options: A list of options to be passed to the receiver

Plugin (bridge) specific parameters

  • isgateway: If true, assigns an IP address to the bridge so containers connected to it may use it as a gateway.
  • isdefaultgateway: If true, sets the assigned IP address as the default route.
  • forceAddress: Tells the plugin to allocate a new IP address if the previous value has changed.
  • mtu: Define the MTU of the bridge.
  • hairpinMode: Set hairpin mode for the interfaces on the bridge

The items that are in bold above are the ones we’re using in this example.  You should play around with the others to get a feeling for how they work but most are fairly straight forward.  You’ll also note that one of the items is part of the IPAM plugin.  We arent going to cover those in this post (we will later!) but for now just know that we’re using multiple CNI plugins to make this work.

Ok – so now that we have our network definition, we want to run it.  However – at this point we’ve only defined characteristics of the bridge.  The point of CNI is to network containers so we need to tell the plugin about the container we want to work with as well.  These variables are passed to the plugin via environmental variables.  So our command might look like this…

Let’s walk through this.  I think most of you are probably familiar with using environmental variables on systems by setting them at the shell or system level.  In addition to that, you can also pass them directly to a command.  When you do this, they will be used only by the executable you are calling and only during that execution.  So in this case, the following variables will be passed to the bridge executable…

  • CNI_COMMAND=ADD – We are telling CNI that we want to add a connection
  • CNI_CONTAINER=1234567890 – We’re telling CNI that the network namespace we want to work is called ‘1234567890’ (more on this below)
  • CNI_NETNS=/var/run/netns/1234567890 – The path to the namespace in question
  • CNI_IFNAME=eth12 – The name of the interface we wish to use on the container side of the connection
  • CNI_PATH=pwd – We always need to tell CNI where the plugin executables live.  In this case, since we’re already in the ‘cni’ directory we just have the variable reference pwd (present working directory). You need the ticks around the command pwd for it to evaluate correctly. Formatting here seems to be removing them but they are in the command above correctly

Once the variables you wish to pass to the executable are defined, we then pick the plugin we want to use which in this case is bridge.  Lastly – we feed the network configuration file into the plugin using STDIN.  To do this just use the left facing bracket ‘<‘.  Before we run the command, we need to create the network namespace that the plugin is going to work with.  Tpically the container runtime would handle this but since we’re keeping things simple this first go around we’ll just create one ourselves…

Once that’s created let’s run the plugin…

Running the command returns a couple of things.  First – it returns an error since the IPAM driver can’t find the file it uses to store IP information locally.  If we ran this again for a different namespace, we wouldn’t get this error since the file is created the first time we run the plugin.  The second thing we get is a JSON return indicating the relevant IP configuration that was configured by the plugin.  In this case, the bridge itself should have received the IP address of and the namespace interface would have received  It also added the default route and the route that we defined in the network configuration JSON.  So let’s look and see what it did…

Notice we now have a bridge interface called ‘cni_bridge0’ which has the IP interface we expected to see.  Also note at the bottom we have one side of a veth pair.  Recall that we also asked it to enable masquerading.  If we look at our hosts iptables rules we’ll see the masquerade and accept rule…

Let’s now look in the network namespace…

Our namespace is also configured as we expected.  The namespace has an interface named ‘eth12’ with an IP address of and the routes we defined are also there.  So it worked!

This was a simple example but I think it highlights how CNI is implemented and works.  Next week we’ll dig further into the CNI plugins as we examine an example of how to use CNI with a container runtime.

Before I wrap up – I do want to comment briefly on one item that I initially got hung up on and that’s how the plugin is actually called.  In our example – we’re calling a specific plugin directly.  As such – I was initially confused as to why you needed to specify the location of the plugins with the ‘CNI_PATH’.  After all – we’re calling a plugin directly so obviously we already know where it is.  The reason for this is that this is not how CNI is typically used.  Typically – you have a another application or system that is reading the CNI network definitions and running them.  In those cases, the CNI_PATH will already be defined within the system.  Since the network configuration file defines what plugin to use (in our case bridge) all the system would need to know is where to find the plugins.  To find them, it references the CNI_PATH variable.  We’ll talk more about this in future posts where we discuss what other applications use CNI (cough, Kubernetes, cough) so for now just know that the example above shows how CNI works, but does not show a typical use case outside of testing.


imageAs many of you have noticed I’ve been neglecting the blog for past few months.  The main reason for this is that the majority of my free time was being spent generating content for a new book.  I’m pleased to announce that the book, Docker Networking Cookbook, has now been released! 

Here’s a brief description of the book…

“Networking functionality in Docker has changed considerably since its first release, evolving to offer a rich set of built-in networking features, as well as an extensible plugin model allowing for a wide variety of networking functionality. This book explores Docker networking capabilities from end to end. Begin by examining the building blocks used by Docker to implement fundamental containing networking before learning how to consume built-in networking constructs as well as custom networks you create on your own. Next, explore common third-party networking plugins, including detailed information on how these plugins inter-operate with the Docker engine. Consider available options for securing container networks, as well as a process for troubleshooting container connectivity.  Finally, examine advanced Docker networking functions and their relevant use cases, tying together everything you need to succeed with your own projects.”

The book is available from Packt and I believe Amazon has it as well.  If you happen to buy a copy I would greatly appreciate it if you would send me any and all feedback you have.  This is my first attempt at writing a book so any feedback and critiques you can share would be really great.

A big thank you to all of the folks at Packt that made this possible and worked with me through the editing and publishing process.  I’d also like to thank the technical reviewer Francisco Souza for his review. 

Now that the book is published I look forward to spending my free time blogging again.  Thanks for hanging in there!


Ansible up and running

After much delay – I’ve finally found time to take a look at Ansible.  I’ve spent some time looking at possible platforms to automate network deployment and Ansible seems to be a favorite in this arena.  One of the primary reasons for this is that Ansible is ‘clientless’ (I’m putting that in quotes for a reason, more on that in a later post).  So unlike Chef, Puppet, and Salt (Yes – there are proxy modes available in some products) Ansible does not require an installed client on the remote endpoints.  So let’s get right into a basic lab setup.

While the end goal will be to use Ansible to automate network appliances, we’re going to start with the a more standard use case – Linux servers.  The base lab we will start with is two servers, one acting as the Ansible server and the second being a Ansible client or remote server.  Both hosts are CentOS 7 based Linux hosts.  So our base lab looks like this…

Pretty exciting right?  I know, it’s not, but I want to start with the basics and build from there…

Note: I’ll refer to ansibleserver as ‘server’ and ansibleclient1 as ‘client’ throughout the remainder of the article.

If you’re following along, my assumption is that you have 2 servers that are similarly IP’d, have resolvable DNS names, and are fully updated CentOS 7 hosts.  That being said, let’s start out by building the Ansible server node.  The install process looks like this…

That’s it.  You’re done.  Do a quick check and see if Ansible is working as expected…

So it looks like everything is installed.  The next thing we want to do is configure a means to communicate with the clients.  Ansible’s default means of doing this is with SSH.  So let’s configure an SSH key on the server and send it to the client…

After the key has been installed, test it out by SSHing to the client from the server…


The login should work and you shouldn’t need to specify a password for the root user.

Note: In this example Im using the root account on both the server and the client.  By default, Ansible attempts to use the current logged in user to connect to the clients.  If you dont plan on using the root user on the server, you’ll need to tell Ansible to still use root for connectivity. I plan on covering this functionality in a later post.

The next thing we nee to do is to define the clients you want the server to work with.  This is done by defining them in Ansible ‘hosts’ file which is located in ‘/etc/ansible/hosts’.  Out of the box, this file is full of examples.  For the sake of clarity, I’ve removed the examples leaving the file looking like this…

Here you can see I’ve added a group named ‘linuxservers’ and in that group I’ve defined a client ‘ansibleclient1’.  Any host that you wish to manage must be specified in this file.  Once defined, it can be referenced either directly by name or as part of a group.  Review some of the default examples in this file to give you an idea of how you can match on different hosts.

So now that we have a client defined, how do we talk to it with Ansible?  The best test is to try and run some basic modules against the clients.  For instance, there’s a ‘ping’ module that lets you see which hosts are online…

Notice that you can run the ping module against the client server in a few different ways.  I can reference it by name, by group, or by using the ‘all’ flag which matches all clients defined in the hosts file.  As you’ll see, modules are the key component of Ansible that allow it to perform a wide variety of tasks.  For instance, there’s a module  for SELinux…

Above I call the ‘selinux’ module and pass it an argument with the ‘-a’ flag to tell it to disable SELinux.  There’s also a shell module that allows you to run shell commands on clients like I do below with the ‘ip addr’ command…


The list of all of the modules can be found on the Ansible website.  So while the modules themselves are powerful, running them in this manner isn’t much better than just executing one module at a time.  The system becomes really powerful when you couple modules with playbooks.

Ansible playbooks are written in YAML and contain one or more ‘plays’.  Let’s look at an example playbook so you can see what I’m talking about…

Here’s a fairly basic playbook that installs and starts an Apache web server.  This playbook defines one play, with one task.  Plays are defined by specifying the hosts to be part of the play followed by a series of tasks to execute against them.  Playbooks can contain multiple plays each with multiple tasks.  In this case, this play has one task which uses the module ‘yum’.  In addition, to the tasks, you can also define handlers.  These are items that you want to run ONLY if a certain task makes successful changes to the system.  So in this case, we tell the task ‘Install Apache Web Server’ to notify the handlers ‘openport80’ and ‘startwebserver’.  If the task results in the system successfully installing Apache, it will notify the handlers defined for the task.  So let’s save this playbook on the server and run it…

As you can see, the means to call a playbook is simple.  You just use the ‘ansible-playbook’ command and specify the YAML playbook definition.  As you can see from the run, Ansible successfully ran the task and in turn triggered the two defined handlers.  If we browse to the client on port 80, we should see the Apache start page…

Success!  Let’s run the playbook again and see what happens…

Note that this time the task completed without making any changes.  Since nothing changed, there’s no need to notify any of the handlers.  Pretty slick huh?

I hope this first look at Ansible has been helpful.  Stay tuned for more posts on other features and ways to automate with Ansible!


« Older entries