docker

You are currently browsing the archive for the docker category.

If you’ve been paying attention to the discussions around container networking you’ve likely heard the acronym CNI being used.  CNI stands for Container Networking Interface and it’s goal is to create a generic plugin-based networking solution for containers.  CNI is defined by a spec (read it now, its not very long) that has some interesting language in it.  Here are a couple of points I found interesting during my first read through…

  • The spec defines a container as being a Linux network namespace.  We should be comfortable with that definition as container runtimes like Docker create a new network namespace for each container.
  • Network definitions for CNI are stored as JSON files.
  • The network definitions are streamed to the plugin through STDIN.  That is – there are no configuration files sitting on the host for the network configuration.
  • Other arguments are passed to the plugin via environmental variables
  • A CNI plugin is implemented as an executable.
  • The CNI plugin is responsible wiring up the container.  That is – it needs to do all the work to get the container on the network.  In Docker, this would include connecting the container network namespace back to the host somehow.
  • The CNI plugin is responsible for IPAM which includes IP address assignment and installing any required routes.

If you’re used to dealing with Docker this doesn’t quite seem to fit the mold.  It’s apparent to me that the CNI plugin is responsible for the network end of the container, but it wasn’t initially clear to me how that was actually implemented.  So the next question might be, can I use CNI with Docker?  The answer is yes, but not as an all in one solution.  Docker has it’s own network plugin system called CNM.  CNM allows plugins to interact directly with Docker.  A CNM plugin can be registered to Docker and used directly from it.  That is, you can use Docker to run containers and directly assign their network to the CNM registered plugin.  This works well, but because Docker has CNM, they dont directly integrate with CNI (as far as I can tell).  That does not mean however, that you can’t use CNI with Docker.  Recall from the sixth bullet above that the plugin is responsible for wiring up the container.  So it seems possible that Docker could be the container runtime – but not handle the networking end of things (more on this in a future post).

At this point – I think its fair to start looking at what CNI actually does to try to get a better feel for how it fits into the picture.  Let’s look at a quick example of using one of the plugins.

Let’s start by downloading the pre-built CNI binaries…

Ok – let’s make sure we understand what we just did there.  We first created a directory called ‘cni’ to store the binaries in.  We then used the curl command to download the CNI release bundle.  When using curl to download a file we need to pass the ‘O’ parameter to tell curl to output to a file.  We also need to pass the ‘L’ parameter in this case to allow curl to follow redirects since the URL we’re downloading from is actually redirecting us elsewhere.  Once downloaded, we unpack the archive using the tar command.

After all of that we can see that we have a few new files.  For right now, let’s focus on the ‘bridge’ file which is the bridge plugin.  Bridge is one of the included plugins that ships with CNI.  It’s job, as you might have guessed, is to attach a container to a bridge interface.  So now that we have the plugins, how do we actually use them?  One of the earlier bullet points mentioned that network configuration is streamed into the plugin through STDIN.  So we know we need to use STDIN to get information about the network into the plugin but that’s not all the info the plugin needs.  The plugin also needs more information such as the action you wish to perform, the namespace you wish to work with, and other various information.  This information is passed to the plugin via environmental variables.  Confused?  No worries, let’s walk through an example.  Let’s first define a network configuration file we wish to use for our bridge…

Above we create a JSON definition for our bridge network.  There are some CNI generic definitions listed above as well as some specific to the bridge plugin itself.  Let’s walk through them one at a time.

CNI generic parameters

  • cniVersion: The version of the CNI spec in which the definition works with
  • name: The network name
  • type: The name of the plugin you wish to use.  In this case, the actual name of the plugin executable
  • args: Optional additional parameters
  • ipMasq: Configure outbound masquerade (source NAT) for this network
  • ipam:
    • type: The name of the IPAM plugin executable
    • subnet: The subnet to allocate out of (this is actually part of the IPAM plugin)
    • routes:
      • dst: The subnet you wish to reach
      • gw: The IP address of the next hop to reach the dst.  If not specified the default gateway for the subnet is assumed
  • dns:
    • nameservers: A list of nameservers you wish to use with this network
    • domain: The search domain to use for DNS requests
    • search: A list of search domains
    • options: A list of options to be passed to the receiver

Plugin (bridge) specific parameters

  • isgateway: If true, assigns an IP address to the bridge so containers connected to it may use it as a gateway.
  • isdefaultgateway: If true, sets the assigned IP address as the default route.
  • forceAddress: Tells the plugin to allocate a new IP address if the previous value has changed.
  • mtu: Define the MTU of the bridge.
  • hairpinMode: Set hairpin mode for the interfaces on the bridge

The items that are in bold above are the ones we’re using in this example.  You should play around with the others to get a feeling for how they work but most are fairly straight forward.  You’ll also note that one of the items is part of the IPAM plugin.  We arent going to cover those in this post (we will later!) but for now just know that we’re using multiple CNI plugins to make this work.

Ok – so now that we have our network definition, we want to run it.  However – at this point we’ve only defined characteristics of the bridge.  The point of CNI is to network containers so we need to tell the plugin about the container we want to work with as well.  These variables are passed to the plugin via environmental variables.  So our command might look like this…

Let’s walk through this.  I think most of you are probably familiar with using environmental variables on systems by setting them at the shell or system level.  In addition to that, you can also pass them directly to a command.  When you do this, they will be used only by the executable you are calling and only during that execution.  So in this case, the following variables will be passed to the bridge executable…

  • CNI_COMMAND=ADD – We are telling CNI that we want to add a connection
  • CNI_CONTAINER=1234567890 – We’re telling CNI that the network namespace we want to work is called ‘1234567890’ (more on this below)
  • CNI_NETNS=/var/run/netns/1234567890 – The path to the namespace in question
  • CNI_IFNAME=eth12 – The name of the interface we wish to use on the container side of the connection
  • CNI_PATH=pwd – We always need to tell CNI where the plugin executables live.  In this case, since we’re already in the ‘cni’ directory we just have the variable reference pwd (present working directory). You need the ticks around the command pwd for it to evaluate correctly. Formatting here seems to be removing them but they are in the command above correctly

Once the variables you wish to pass to the executable are defined, we then pick the plugin we want to use which in this case is bridge.  Lastly – we feed the network configuration file into the plugin using STDIN.  To do this just use the left facing bracket ‘<‘.  Before we run the command, we need to create the network namespace that the plugin is going to work with.  Tpically the container runtime would handle this but since we’re keeping things simple this first go around we’ll just create one ourselves…

Once that’s created let’s run the plugin…

Running the command returns a couple of things.  First – it returns an error since the IPAM driver can’t find the file it uses to store IP information locally.  If we ran this again for a different namespace, we wouldn’t get this error since the file is created the first time we run the plugin.  The second thing we get is a JSON return indicating the relevant IP configuration that was configured by the plguin.  In this case, the bridge itself should have received the IP address of 10.15.20.1/24 and the namespace interface would have received 10.15.20.2/24.  It also added the default route and the 1.1.1.1/32 route that we defined in the network configuration JSON.  So let’s look and see what it did…

Notice we now have a bridge interface called ‘cni_bridge0’ which has the IP interface we expected to see.  Also note at the bottom we have one side of a veth pair.  Recall that we also asked it to enable masquerading.  If we look at our hosts iptables rules we’ll see the masquerade and accept rule…

Let’s now look in the network namespace…

Our namespace is also configured as we expected.  The namespace has an interface named ‘eth12’ with an IP address of 10.15.20.2/24 and the routes we defined are also there.  So it worked!

This was a simple example but I think it highlights how CNI is implemented and works.  Next week we’ll dig further into the CNI plugins as we examine an example of how to use CNI with a container runtime.

Before I wrap up – I do want to comment briefly on one item that I initially got hung up on and that’s how the plugin is actually called.  In our example – we’re calling a specific plugin directly.  As such – I was initially confused as to why you needed to specify the location of the plugins with the ‘CNI_PATH’.  After all – we’re calling a plugin directly so obviously we already know where it is.  The reason for this is that this is not how CNI is typically used.  Typically – you have a another application or system that is reading the CNI network definitions and running them.  In those cases, the CNI_PATH will already be defined within the system.  Since the network configuration file defines what plugin to use (in our case bridge) all the system would need to know is where to find the plugins.  To find them, it references the CNI_PATH variable.  We’ll talk more about this in future posts where we discuss what other applications use CNI (cough, Kubernetes, cough) so for now just know that the example above shows how CNI works, but does not show a typical use case outside of testing.

Tags:

imageAs many of you have noticed I’ve been neglecting the blog for past few months.  The main reason for this is that the majority of my free time was being spent generating content for a new book.  I’m pleased to announce that the book, Docker Networking Cookbook, has now been released! 

Here’s a brief description of the book…

“Networking functionality in Docker has changed considerably since its first release, evolving to offer a rich set of built-in networking features, as well as an extensible plugin model allowing for a wide variety of networking functionality. This book explores Docker networking capabilities from end to end. Begin by examining the building blocks used by Docker to implement fundamental containing networking before learning how to consume built-in networking constructs as well as custom networks you create on your own. Next, explore common third-party networking plugins, including detailed information on how these plugins inter-operate with the Docker engine. Consider available options for securing container networks, as well as a process for troubleshooting container connectivity.  Finally, examine advanced Docker networking functions and their relevant use cases, tying together everything you need to succeed with your own projects.”

The book is available from Packt and I believe Amazon has it as well.  If you happen to buy a copy I would greatly appreciate it if you would send me any and all feedback you have.  This is my first attempt at writing a book so any feedback and critiques you can share would be really great.

A big thank you to all of the folks at Packt that made this possible and worked with me through the editing and publishing process.  I’d also like to thank the technical reviewer Francisco Souza for his review. 

Now that the book is published I look forward to spending my free time blogging again.  Thanks for hanging in there!

Tags:

I thought it would be a good idea to revisit my last Kubernetes build in which I was using Salt to automate the deployment.  The setup worked well at the time, but much has changed with Kubernetes since I initially wrote those state files.  That being said, I wanted to update them to make sure they worked with Kubernetes 1.0 and above.  You can find my Salt config for this build over at Github…

https://github.com/jonlangemak/saltstackv2

A couple of quick notes before we walk through how to use the repo…

-While I used the last version of this repo as a starting point, I’ve stripped this down to basics (AKA – Some of the auxiliary pods aren’t here (yet)).  I’ll be adding to this constantly and I do intend to add a lot more functionality to the defined state files.
-All of the Kubernetes related communication is unsecured.  That is – it’s all over HTTP.  I already started work on adding an option to do SSL if you so choose. 

That being said, let’s jump right into how to use this.  My lab looks like this…

image 
Here we have 3 hosts.  K8stest1 will perform the role of the master while k8stest2 and k8stest3 will play the role of nodes or minions.  Each host will be running Docker and will have a routable network segment configured on it’s Docker0 bridge interface.  Your upstream layer 3 device will need to have static routes pointing each Docker0 bridge network to their respective hosts physical interface (192.168.127.100x) as shown above.  In addition to these 3 hosts, I also have a separate build server that acts as the Salt master and initiates the cluster build.  That server is called ‘kubbuild’ and isn’t pictured because it only plays a part in the initial configuration.  So let’s get right into the build…

In my case, the base lab configuration looks like this…

-All hosts are running CentOS 7.1 and are fully updated
-The 3 lab hosts (k8stest[1-3]) are configured as Salt minions and are reachable by the salt-master.  If you don’t know how to do that see the section of this post that talks about configuring the Salt master and minions.

The first thing you need to do is clone my repo onto your build server…

The next thing we want to do is download the Kubernetes binaries we need.  In earlier posts we had built them from scratch but we’re now going to download them instead.  All of the Kubernetes releases can be downloaded as a TAR file from github.  In this case, let’s work off of the 1.1.7 release.  So download this TAR file…

Next we have to unpack this file, and another TAR file inside this one, to get to the actual binaries…

Next we move those extracted binaries to the correct place in the Salt folder structure…

Alright – That’s the hardest part!  Now let’s go take a look at our Salt pillar configuration.  Take a look at the file ‘/srv/pillar/kube_data.sls’…

All you need to do is update this YAML file with your relevant configuration.  The above example is just a textual version of the network diagram shown earlier.  Keep in mind that you can add minions later by just simply adding onto this file – I’ll demo that later on.  Once you have this updated to match your configuration, let’s make sure we can reach our Salt minions and then execute the state to push the configuration out…

image
Now sit back and enjoy a cup of coffee while Salt does it’s magic.  When it’s done, you should see the results of executing the states against the hosts you defined in the ‘kube_data.sls’ file…

image
If you scroll back up through all of the results you will likely see that it errors out on this section of the master…

image
This is expected and is a result of the etcd container not coming up in time in order for the ‘pods’ state to work.  The current fix is to wait until all of the Kubernetes master containers load and then just execute the highstate again.

So let’s head over to our master server and see if things are working as expected…

image
Perfect!  Our 2 nodes have been discovered.  Since we’re going to execute the Salt highstate again, let’s update the config to include another node…

Note: I’m assuming that the server k8stest4 has been added to the Salt master as a minion.

This run should provision the pods as well as we provision a new Kubernetes node, k8stest4.  So let’s run the highstate again and see what we get…

When the run has finished, let’s head back to the master server and see how many nodes we have…

image 
Perfect!  The Salt config works as expected.  At this point, we have a functioning Kubernetes cluster on our hands.  Let’s make sure everything is working as expected by deploying the guest book demo.  On the master, run this command…

This will create the services and the replication controllers for the example and expose them on the node physical interfaces.  Take note of the port it’s using when you create the services…

image 
Now we just need to wait for the containers to deploy.  Keep an eye on them by checking the pod status…

image
Once Kubernetes finishes deploying the containers, we should see them all listed as ‘Running’…

image
Now we can try and hit the guest book front end by browsing to a minion on the specified port…

image
The example should work as expected.  That’s it for now, much more to come soon!

Tags: ,

« Older entries