Using CNI with Docker

      No Comments on Using CNI with Docker

In our last post we introduced ourselves to CNI (if you haven’t read that yet, I suggest you start there) as we worked through a simple example of connecting a network namespace to a bridge.  CNI managed both the creation of the bridge as well as connecting the namespace to the bridge using a VETH pair.  In this post we’ll explore how to do this same thing but with a container created by Docker.  As you’ll see, the process is largely the same.  Let’s jump right in.

This post assumes that you followed the steps in the first post (Understanding CNI) and have a ‘cni’ directory (~/cni) that contains the CNI binaries.  If you don’t have that – head back to the first post and follow the steps to download the pre-compiled CNI binaries.  It also assumes that you have a default Docker installation.  In my case, Im using Docker version 1.12.  

The first thing we need to do is to create a Docker container.  To do that we’ll run this command…

[email protected]:~/cni$ sudo docker run --name cnitest --net=none -d jonlangemak/web_server_1
835583cdf382520283c709b5a5ee866b9dccf4861672b95eccbc7b7688109b56
[email protected]:~/cni$

Notice that when we ran the command we told Docker to use a network of ‘none’. When Docker is told to do this, it will create the network namespace for the container, but it will not attempt to connect the containers network namespace to anything else.  If we look in the container we should see that it only has a loopback interface…

[email protected]:~/cni$ sudo docker exec cnitest ifconfig
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

[email protected]:~/cni$

So now we want to use CNI to connect the container to something. Before we do that we need some information. Namely, we need a network definition for CNI to consume as well as some information about the container itself.  For the network definition, we’ll create a new definition and specify a few more options to see how they work.  Create the configuration with this command (I assume you’re creating this file in ~/cni)…

cat > mybridge2.conf <<"EOF"
{
    "cniVersion": "0.2.0",
    "name": "mybridge",
    "type": "bridge",
    "bridge": "cni_bridge1",
    "isGateway": true,
    "ipMasq": true,
    "ipam": {
        "type": "host-local",
        "subnet": "10.15.30.0/24",
        "routes": [
            { "dst": "0.0.0.0/0" },
            { "dst": "1.1.1.1/32", "gw":"10.15.30.1"}
        ],
        "rangeStart": "10.15.30.100",
        "rangeEnd": "10.15.30.200",
        "gateway": "10.15.30.99"
    }
}
EOF

In addition to the parameters we saw in the last post, we’ve also added the following…

  • rangeStart: Defines where CNI should start allocating container IPs from within the defined subnet
  • rangeEnd: Defines the end of the range CNI can use to allocate container IPs
  • gateway: Defines the gateway that should be defined.  Previously we hadnt defined this so CNI picked the first IP for use as the bridge interface.

One thing you’ll notice that’s lacking in this configuration is anything related to DNS.  Hold that thought for now (it’s the topic of the next post).

So now that the network is defined we need some info about the container. Specifically we need the path to the container network namespace as well as the container ID. To get that info, we can grep the info from the ‘docker inspect’ command…

[email protected]:~/cni$ sudo docker inspect cnitest | grep -E 'SandboxKey|Id'
        "Id": "1018026ebc02fa0cbf2be35325f4833ec1086cf6364c7b2cf17d80255d7d4a27",
            "SandboxKey": "/var/run/docker/netns/2e4813b1a912",
[email protected]:~/cni$

In this example I used the ‘-E’ flag with grep to tell it to do expression or pattern matching as Im looking for both the container ID as well as the SandboxKey. In the world of Docker, the network namespace file location is referred to as the ‘SandboxKey’ and the ‘Id’ is the container ID assigned by Docker.  So now that we have that info, we can build the environmental variables that we’re going to use with the CNI plugin.  Those would be…

  • CNI_COMMAND=ADD
  • CNI_CONTAINERID=1018026ebc02fa0cbf2be35325f4833ec1086cf6364c7b2cf17d80255d7d4a27
  • CNI_NETNS=/var/run/docker/netns/2e4813b1a912
  • CNI_IFNAME=eth0
  • CNI_PATH=`pwd`

Put that all together in a command and you end up with this…

sudo CNI_COMMAND=ADD CNI_CONTAINERID=1018026ebc02fa0cbf2be35325f4833ec1086cf6364c7b2cf17d80255d7d4a27 CNI_NETNS=/var/run/docker/netns/2e4813b1a912 CNI_IFNAME=eth0 CNI_PATH=`pwd` ./bridge <mybridge2.conf

The only thing left to do at this point is to run the plugin…

[email protected]:~/cni$ sudo CNI_COMMAND=ADD CNI_CONTAINERID=1018026ebc02fa0cbf2be35325f4833ec1086cf6364c7b2cf17d80255d7d4a27 CNI_NETNS=/var/run/docker/netns/2e4813b1a912 CNI_IFNAME=eth0 CNI_PATH=`pwd` ./bridge <mybridge2.conf
{
    "ip4": {
        "ip": "10.15.30.100/24",
        "gateway": "10.15.30.99",
        "routes": [
            {
                "dst": "0.0.0.0/0"
            },
            {
                "dst": "1.1.1.1/32",
                "gw": "10.15.30.1"
            }
        ]
    },
    "dns": {}
}[email protected]:~/cni$

As we saw in the last post, the plugin executes and then provides us some return JSON about what it did.  So let’s look at our host and container again to see what we have…

[email protected]:~/cni$ ifconfig
cni_bridge0 Link encap:Ethernet  HWaddr 0a:58:0a:0f:14:01
          inet addr:10.15.20.1  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::a464:72ff:fe98:2652/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:536 (536.0 B)  TX bytes:648 (648.0 B)

cni_bridge1 Link encap:Ethernet  HWaddr 0a:58:0a:0f:1e:63
          inet addr:10.15.30.99  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::88f:bbff:fed9:118f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:536 (536.0 B)  TX bytes:648 (648.0 B)

docker0   Link encap:Ethernet  HWaddr 02:42:65:43:f5:a7
          inet addr:172.17.0.1  Bcast:0.0.0.0  Mask:255.255.0.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

ens32     Link encap:Ethernet  HWaddr 00:0c:29:3e:49:51
          inet addr:10.20.30.71  Bcast:10.20.30.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe3e:4951/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2568909 errors:0 dropped:67 overruns:0 frame:0
          TX packets:2057136 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:478331698 (478.3 MB)  TX bytes:1336636840 (1.3 GB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:5519471 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5519471 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:2796275357 (2.7 GB)  TX bytes:2796275357 (2.7 GB)

veth719c8174 Link encap:Ethernet  HWaddr aa:bb:6e:c7:cc:d8
          inet6 addr: fe80::a8bb:6eff:fec7:ccd8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:648 (648.0 B)  TX bytes:1206 (1.2 KB)

vethb125661a Link encap:Ethernet  HWaddr fa:54:99:46:65:08
          inet6 addr: fe80::f854:99ff:fe46:6508/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:648 (648.0 B)  TX bytes:1206 (1.2 KB)

[email protected]:~/cni$

From a host perspective, we have quite a few interfaces now. Since we picked up right where we left off with the last post we still have the cni_bridge0 interface along with it’s associated VETH pair. We now also have the cni_bridge1 bridge that we just created along with it’s associated VETH pair interface.  You can see that the cni_bridge1 interface has the IP address we defined as the ‘gateway’ as part of the network configuration.   You’ll also notice that the docker0 bridge is there since it was created by default when Docker was installed.

So now what about our container?  Let’s look…

[email protected]:~/cni$ sudo docker exec cnitest ifconfig
eth0      Link encap:Ethernet  HWaddr 0a:58:0a:0f:1e:64
          inet addr:10.15.30.100  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::f09e:73ff:fe3e:838c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:15 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1206 (1.2 KB)  TX bytes:648 (648.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

[email protected]:~/cni$ sudo docker exec cnitest ip route
default via 10.15.30.99 dev eth0
1.1.1.1 via 10.15.30.1 dev eth0
10.15.30.0/24 dev eth0  proto kernel  scope link  src 10.15.30.100
[email protected]:~/cni$

As you can see, the container has the network configuration we’d expect…

  • It has an IP address within the defined range (10.15.30.100)
  • Its interface is named ‘eth0’
  • It has a default route pointing at the gateway IP address of 10.15.30.99
  • It has an additional route for 1.1.1.1/32 pointing at 10.15.30.1

And as a final quick test we can attempt to access the service in the container from the host…

[email protected]:~/cni$ curl http://10.15.30.100
<body>
<html>
<h1><span style="color:#FF0000;font-size:72px;">Web Server #1 - Running on port 80</span></h1>
</body>
</html>
[email protected]:~/cni$

So as you can see – connecting a Docker container wasn’t much different than connecting a network namespace. In fact – the process was identical, we just had to account for where Docker stores it’s network namespace definitions. In our next post we’re going to talk about DNS related setting for a container and how those play into CNI.

Leave a Reply

Your email address will not be published. Required fields are marked *