If you’ve been paying attention to the discussions around container networking you’ve likely heard the acronym CNI being used.  CNI stands for Container Networking Interface and it’s goal is to create a generic plugin-based networking solution for containers.  CNI is defined by a spec (read it now, its not very long) that has some interesting language in it.  Here are a couple of points I found interesting during my first read through…

  • The spec defines a container as being a Linux network namespace.  We should be comfortable with that definition as container runtimes like Docker create a new network namespace for each container.
  • Network definitions for CNI are stored as JSON files.
  • The network definitions are streamed to the plugin through STDIN.  That is – there are no configuration files sitting on the host for the network configuration.
  • Other arguments are passed to the plugin via environmental variables
  • A CNI plugin is implemented as an executable.
  • The CNI plugin is responsible wiring up the container.  That is – it needs to do all the work to get the container on the network.  In Docker, this would include connecting the container network namespace back to the host somehow.
  • The CNI plugin is responsible for IPAM which includes IP address assignment and installing any required routes.

If you’re used to dealing with Docker this doesn’t quite seem to fit the mold.  It’s apparent to me that the CNI plugin is responsible for the network end of the container, but it wasn’t initially clear to me how that was actually implemented.  So the next question might be, can I use CNI with Docker?  The answer is yes, but not as an all in one solution.  Docker has it’s own network plugin system called CNM.  CNM allows plugins to interact directly with Docker.  A CNM plugin can be registered to Docker and used directly from it.  That is, you can use Docker to run containers and directly assign their network to the CNM registered plugin.  This works well, but because Docker has CNM, they dont directly integrate with CNI (as far as I can tell).  That does not mean however, that you can’t use CNI with Docker.  Recall from the sixth bullet above that the plugin is responsible for wiring up the container.  So it seems possible that Docker could be the container runtime – but not handle the networking end of things (more on this in a future post).

At this point – I think its fair to start looking at what CNI actually does to try to get a better feel for how it fits into the picture.  Let’s look at a quick example of using one of the plugins.

Let’s start by downloading the pre-built CNI binaries…

Ok – let’s make sure we understand what we just did there.  We first created a directory called ‘cni’ to store the binaries in.  We then used the curl command to download the CNI release bundle.  When using curl to download a file we need to pass the ‘O’ parameter to tell curl to output to a file.  We also need to pass the ‘L’ parameter in this case to allow curl to follow redirects since the URL we’re downloading from is actually redirecting us elsewhere.  Once downloaded, we unpack the archive using the tar command.

After all of that we can see that we have a few new files.  For right now, let’s focus on the ‘bridge’ file which is the bridge plugin.  Bridge is one of the included plugins that ships with CNI.  It’s job, as you might have guessed, is to attach a container to a bridge interface.  So now that we have the plugins, how do we actually use them?  One of the earlier bullet points mentioned that network configuration is streamed into the plugin through STDIN.  So we know we need to use STDIN to get information about the network into the plugin but that’s not all the info the plugin needs.  The plugin also needs more information such as the action you wish to perform, the namespace you wish to work with, and other various information.  This information is passed to the plugin via environmental variables.  Confused?  No worries, let’s walk through an example.  Let’s first define a network configuration file we wish to use for our bridge…

Above we create a JSON definition for our bridge network.  There are some CNI generic definitions listed above as well as some specific to the bridge plugin itself.  Let’s walk through them one at a time.

CNI generic parameters

  • cniVersion: The version of the CNI spec in which the definition works with
  • name: The network name
  • type: The name of the plugin you wish to use.  In this case, the actual name of the plugin executable
  • args: Optional additional parameters
  • ipMasq: Configure outbound masquerade (source NAT) for this network
  • ipam:
    • type: The name of the IPAM plugin executable
    • subnet: The subnet to allocate out of (this is actually part of the IPAM plugin)
    • routes:
      • dst: The subnet you wish to reach
      • gw: The IP address of the next hop to reach the dst.  If not specified the default gateway for the subnet is assumed
  • dns:
    • nameservers: A list of nameservers you wish to use with this network
    • domain: The search domain to use for DNS requests
    • search: A list of search domains
    • options: A list of options to be passed to the receiver

Plugin (bridge) specific parameters

  • isgateway: If true, assigns an IP address to the bridge so containers connected to it may use it as a gateway.
  • isdefaultgateway: If true, sets the assigned IP address as the default route.
  • forceAddress: Tells the plugin to allocate a new IP address if the previous value has changed.
  • mtu: Define the MTU of the bridge.
  • hairpinMode: Set hairpin mode for the interfaces on the bridge

The items that are in bold above are the ones we’re using in this example.  You should play around with the others to get a feeling for how they work but most are fairly straight forward.  You’ll also note that one of the items is part of the IPAM plugin.  We arent going to cover those in this post (we will later!) but for now just know that we’re using multiple CNI plugins to make this work.

Ok – so now that we have our network definition, we want to run it.  However – at this point we’ve only defined characteristics of the bridge.  The point of CNI is to network containers so we need to tell the plugin about the container we want to work with as well.  These variables are passed to the plugin via environmental variables.  So our command might look like this…

Let’s walk through this.  I think most of you are probably familiar with using environmental variables on systems by setting them at the shell or system level.  In addition to that, you can also pass them directly to a command.  When you do this, they will be used only by the executable you are calling and only during that execution.  So in this case, the following variables will be passed to the bridge executable…

  • CNI_COMMAND=ADD – We are telling CNI that we want to add a connection
  • CNI_CONTAINER=1234567890 – We’re telling CNI that the network namespace we want to work is called ‘1234567890’ (more on this below)
  • CNI_NETNS=/var/run/netns/1234567890 – The path to the namespace in question
  • CNI_IFNAME=eth12 – The name of the interface we wish to use on the container side of the connection
  • CNI_PATH=pwd – We always need to tell CNI where the plugin executables live.  In this case, since we’re already in the ‘cni’ directory we just have the variable reference pwd (present working directory). You need the ticks around the command pwd for it to evaluate correctly. Formatting here seems to be removing them but they are in the command above correctly

Once the variables you wish to pass to the executable are defined, we then pick the plugin we want to use which in this case is bridge.  Lastly – we feed the network configuration file into the plugin using STDIN.  To do this just use the left facing bracket ‘<‘.  Before we run the command, we need to create the network namespace that the plugin is going to work with.  Tpically the container runtime would handle this but since we’re keeping things simple this first go around we’ll just create one ourselves…

Once that’s created let’s run the plugin…

Running the command returns a couple of things.  First – it returns an error since the IPAM driver can’t find the file it uses to store IP information locally.  If we ran this again for a different namespace, we wouldn’t get this error since the file is created the first time we run the plugin.  The second thing we get is a JSON return indicating the relevant IP configuration that was configured by the plugin.  In this case, the bridge itself should have received the IP address of 10.15.20.1/24 and the namespace interface would have received 10.15.20.2/24.  It also added the default route and the 1.1.1.1/32 route that we defined in the network configuration JSON.  So let’s look and see what it did…

Notice we now have a bridge interface called ‘cni_bridge0’ which has the IP interface we expected to see.  Also note at the bottom we have one side of a veth pair.  Recall that we also asked it to enable masquerading.  If we look at our hosts iptables rules we’ll see the masquerade and accept rule…

Let’s now look in the network namespace…

Our namespace is also configured as we expected.  The namespace has an interface named ‘eth12’ with an IP address of 10.15.20.2/24 and the routes we defined are also there.  So it worked!

This was a simple example but I think it highlights how CNI is implemented and works.  Next week we’ll dig further into the CNI plugins as we examine an example of how to use CNI with a container runtime.

Before I wrap up – I do want to comment briefly on one item that I initially got hung up on and that’s how the plugin is actually called.  In our example – we’re calling a specific plugin directly.  As such – I was initially confused as to why you needed to specify the location of the plugins with the ‘CNI_PATH’.  After all – we’re calling a plugin directly so obviously we already know where it is.  The reason for this is that this is not how CNI is typically used.  Typically – you have a another application or system that is reading the CNI network definitions and running them.  In those cases, the CNI_PATH will already be defined within the system.  Since the network configuration file defines what plugin to use (in our case bridge) all the system would need to know is where to find the plugins.  To find them, it references the CNI_PATH variable.  We’ll talk more about this in future posts where we discuss what other applications use CNI (cough, Kubernetes, cough) so for now just know that the example above shows how CNI works, but does not show a typical use case outside of testing.

Tags:

This is one of those ‘I must be living under a rock’ things.  I’m not sure how I’ve never heard of TMUX before but it’s really pretty awesome.  I initially came across it when searching for a way to share a terminal session with another user.  It does that quite well but it’s also a great terminal session manager allowing for pane, window, and session management.  Let’s take a look at a quick example to show you what I mean.

Here we have a server called ‘tmuxtest’.  The server already has TMUX on it by default but if it’s not there you can easily install it (sudo apt-get install tmux, etc).  So let’s say I want to start a new session.  The easiest way to do this is to just type ‘tmux’..

Now we’re in TMUX.  We are in what’s called a ‘session’.  The session can contain multiple panes and multiple windows.  For instance, if I wanted to create a second pane I could do by pressing ‘Ctrl-b + %’…

Notice that the screen on the right, the new one, has a green boarder around it.  That’s my active screen.  Now if I want to split this screen horizontally I can use ‘Ctrl-b + “‘…

You might be figuring it out, but ‘Ctrl-b’ is the magic key sequence in TMUX.  Here’s a list of commands that relate to panes…

As you can see, we can select different panes pretty easily by using the arrow keys and end them as we typically end windows with ‘exit’.  So that’s all well and good, but that didnt buy us much.  What else is there?

Well before I forget, remember that we’re in a session that I can attach and detach to.  Have you ever needed to start a download on a remote server and worried about losing your connection and having the download crash?  I have!  So why not run it in a session?  Simply start a TMUX session, start the download, and then detach from the session (Cltrl-b d)…

Now I shut my laptop and go home.  When I get home, I get on VPN, log back into the server, and see that the download is still running (the size is increasing) and the TMUX session is still there which I can reconnect to…

I can reconnect to the session by using ‘tmux attach -t <session ID or name>’ which in this case is ‘3’…

Pretty cool huh?  Here are some commands that are relevant to dealing with sessions…

And before I get too far ahead of myself, I should talk about windows as well.  A session can have multiple windows.  For instance, if I start a TMUX session I can press ‘Ctrl-b c’ (I’ll do it twice)..

Notice how along the bottom I now have 3 sessions?  They’re labelled 0, 1, and 2.  I’m currently on the ‘2’ sessions which we can tell by the asterisk being next to it.  I can change between windows by pressing ‘Ctrl-b p’ or ‘Ctrl-b n’ which stand for previous and next.  Here are some other commands related to windows…

Ok – so now for the terminal sharing.  We know how to create a session and how to attach and detach from them.  Surprisingly, that’s all we need to know to share a session between two users using the same account.  For instance, let’s have two users log into the server both using the ‘user’ account. We’ll have the first user create a session called ‘watchme’ with the command ‘tmux new -s watchme’.  Now if we have another person log in to the same server, with the same user account, they can see that session by using ‘tmux ls’…

All the second user has to do is attach to that same session using the ‘tmux attach -t watchme’ command…

Pretty slick huh?  But what about if you have two different users on the same server?  Unfortunately, it’s not as easy to share sessions between multiple users, but it is still possible by creating a socket that both users can reach.  For instance, if we log into the same server with two different users we can see that by default user2 can’t see user1’s sessions…

However, we can specify a socket file for TMUX to use and make it accessible by both users like this…

Notice how we create the session under user1 as normal with the exception of passing a parameter to the ‘-S’ flag.  The ‘-S’ flag tells TMUX to create a socket at the specified location, in this case ‘/var/tmp/tmuxsocket’. Then I detach from the session, and change the permissions on the file so that all users can access it with the ‘chmod 777 /var/tmp/tmuxsocket’ command.  Now, user2 can see the session by telling TMUX where to look for the socket.  Is this secure?  Gosh no.  Is this probably something you should only do with another user that you fully trust?  Yes.  But if you fit that criteria, it’s drop dead simple to share a session.  That or you can both use the same account as we saw in the first example.

In any case – TMUX is pretty awesome.  I plan on making it a permanent addition to my toolbox.

Update: Ivan Pepelnjak reached out to me after this was published and suggested that it would make more sense to move the functions inside the defined class.  Doing this removes the functions from the global namespace and avoids the possibility of function names overlapping.  I think that’s a great idea so I’ve updated the examples below to align with this thinking.  Thanks Ivan!

I’ve been playing around with Ansible quite a bit lately.  One of the issues I’ve started to run into is that Ansible can’t always do what you want by default.  Put more plainly, Ansible can’t always easily do what you want.    Many times I found myself writing tasks to manipulate variables and thinking to myself, “Man – if I could just run some Python code on this it would be way easier”.  As luck would have it, you can!  Ansible supports a whole slew of plugins but the type I want to talk about today are called filter plugins.  Filter plugins, in my opinion, are one of the easiest ways to manipulate your variables with native Python.  And once you know how to do it you’ll see that it opens up a whole realm of possibilities in your playbooks.  Some of the most popular filters that exist today were once custom filters that someone wrote and then contributed to Ansible.  The IP address (ipaddr) filter set is a great example of filters that can be used to manipulate IP related information.

When I first looked into writing a custom filter I found the Ansible documentation not very helpful.  It essentially points you to their GitHub repo where you can look at the current filters.  Since I learn best by example, let’s write a quick filter so you can see how easy it is to do…

Nothing to it right?  This file defines a single filter called ‘a_filter’.  When called it receives the variable being passed into it (the variable to the left of the pipe (|)), appends the string ‘ CRAZY NEW FILTER’ to it, and then returns the new variable.  Now the trick is where to put this file.  For now, let’s create a folder called ‘filter_plugins’ in the same location as your playbook.  So in my case, the file structure would look like this…

/home/user/my_playbook.yml
/home/user/filter_plugins/my_filters.py

So let’s go ahead and create a quick little test playbook too…

This is a pretty simply playbook.  All it does is use the debug module to output a variable.  However, note that instead of just outputting the word ‘test’, we’re wrapping it in double curly braces like we do for any normal Ansible variable and we’re also piping it to ‘a_filter’.   The piping piece is what’s typically used to pass the variable to any of the predefined, or built-in, filters.  In this case, we’re piping the variable to our own custom filter.

This playbook assumes that you’ve told Ansible to use a local connection when talking to the locahost.  To do this, you need to set the ‘ansible_connection’ variable for the localhost to ‘local’ in your Ansible host file…

Once this is set, and you have both the playbook and the filter files in place, we can try running the playbook…

As you can see, the filter worked as expected.  The variable ‘test’ was passed to our custom filter where it was then modified and returned.  Pretty slick huh?  This is a uber simple example but it shows just how easy it is to inject custom Python functionality into your playbooks.  You’ll notice that in this example, there was only one variable passed to our function.  In this case, it was the variable to the left of the pipe.  In the case of filters, that will always be your first variable however, you can always add more.  For instance, let’s add a new filter to our function like this…

There are a couple of interesting things to point out in our new my_filters.py file.  First off – you’ll notice that we added another Python function called ‘b_filter’.  Its worthwhile to point out that your filter names don’t need to match your function names.  Down in the filters function at the bottom you’ll notice that we map the filter name ‘another_filter’ to the Python function ‘b_filter’.  You’ll also notice that the function b_filter takes 3 arguments.  The first will be the variable to the left of the pipe and the remaining need to be passed directly to the function as we’d normally pass variables.  For example…

Here you can see that we pass the second and third variables to the filter just like we’d normally pass variables to a function.  And while these examples only show doing this with strings, you can pass many other Python data types such as lists and dicts as well.

Lastly – I want to talk about the location of the filters.  By default, Ansible will look in the directory your playbook is located for a folder called ‘filter_plugins’ and load any filters it finds in that folder.  While this works, I don’t particularly care for this as I find it confusing for when you’re moving around playbooks.  I prefer to tell Ansible to look elsewhere for the filters.  To do this, we can edit the /etc/ansible/ansible.cfg file and uncomment and update the ‘filter_plugins’ parameter with your new location.

As you can see, filter plugins make it ridiculously easy to get your variables into native Python for manipulation.  Keep in mind that there are LOTS of default filter plugins so before you go crazy search the documentation to see if what you want already exists.

« Older entries