Kubernetes DNS config on bare metal

One of the ‘newer’ functions of Kubernetes is the ability to register service names in DNS.  More specifically, to register them in a DNS server running in the Kubernetes cluster.  To do this, the clever folks at Google came up with a solution that leverages SkyDNS and another container (called kube2sky) to read the service entries and insert them as DNS entries.  Pretty slick huh?

Beyond the containers to run the DNS service, we also need to tell the pods to use this particular DNS server for DNS resolution.  This is done by adding a couple of lines of config to the kubernetes-kubelet service.  Once that’s done, we can configure the Kubernetes service and the replication controller for the SkyDNS pod.  So let’s start with the kubelet service configuration.  Let’s edit our service definition located here…

Our new config will look like this…

Notice that all we added was a line indicating that the service should use 10.100.0.10 (and IP out of our Portal Net allocation) as a DNS server and another line setting the cluster DNS namespace is ‘kubdomain.local’.  You should substitute the domain name you want to use here and make sure you remember it for when we configure the DNS server.  Once that’s done, save the file and reload the service…

Duplicate this step on all of your Kubernetes nodes.  The next step is to configure the replication controller for the actual service.  There are two files we’ll use for this, the first is for the service definition and looks like this…

The only item I edited was the PortalIP.  Note, this needs to be the same IP you used when you configured the nodes kubelet service DNS server.  Again, this IP is only relevant in the cluster so there’s no need to be picky and you can certainly use the same one I did so long as it’s in your portal net network.  Deploy this service into the cluster and then let’s move onto the replication controller configuration.  That file looks like this…

The only thing you’ll need to change here is your domain under the container definitions.  Once this is done, create the replication controller in Kubernetes.  Once it’s deployed, you should see the pod running…

image
Go ahead and connect to the host that’s running the pod.  In my case, that’s kubminion2 or 10.20.30.63.  Let’s verify that it sees the pod running…

image
Now let’s use the ‘docker logs’ command to see if the kube2sky container is working like it should…

image
Awesome!  So it pulled all of my service definitions and added DNS records for them!  Disregard the monitoring services for now, that’s for an upcoming post about Heapster.

So the true test will be to log into a Kubernetes pod container and see if this all works.    Connect into one of your containers and try resolving one of the names…

image
Looks good!  I would recommend doing this DNS setup as one of the ‘base setup’ tasks when you build a cluster.  More and more of the cluster add-ons are using this feature so its best just to have it from the get go.  Plus it’s pretty slick for using when you build your own services!

Tags: ,

  1. Tim H’s avatar

    I still want to make DNS from the node work by default, but it’s a matter of configuring the node’s resolv.conf, which suddenly gets very distro-specific. But once we do that, we can run a Docker registry as a pod, and then pull other images from that registry by name, rather than by IP.

    Reply

    1. Jon Langemak’s avatar

      So this brings up an interesting point. Up until now I’ve relied on mostly IP addresses rather than DNS names. I’m starting to see a lot of the cluster add on services adopting the DNS lookup for service resolution model (which is great!) but what’s the scope? I mean, at this point my Kubernetes nodes (minions) have lived in my existing DNS (interubernet.local) and I created a new DNS name space (kubdomain.local) for the SkyDNS setup. My thought was that this DNS could be anything I wanted since it was only being used for service lookup. But now Im wondering how containers will do external resolution, I guess I just hadnt run into that yet.

      Should service entries and the nodes be in the same DNS namespace?

      Reply

    2. qiankunli’s avatar

      there are “Failed to load services: Get http://10.100.0.1:80/api/v1beta1/services?labels=” when i run “docker logs”, help!!!

      Reply

      1. Jon Langemak’s avatar

        Can you give me some more background? Which container logs are giving you that message?

        Reply

      2. Gil Lee’s avatar

        Thanks for your wonderful tutorial. I’m trying out Kubernetes on bare metal almost exactly following your instructions. Then here, DNS configuration, I stocked. kube2sky container exited with “KUBERNETES_RO_SERVICE_HOST is not defined.” Could you give me any advice to address this ?

        Reply

        1. Jon Langemak’s avatar

          Are you running SELinux on the host? That sounds vaguely like something I ran into when I had forgot to disable it the first go around.

          Reply

          1. Gil Lee’s avatar

            Thanks for your response. I’m installing kubernetes on three bare metal machines running CentOS 7. SELinux was not enabled on them:
            # cat /etc/sysconfig/selinux

            # This file controls the state of SELinux on the system.
            # SELINUX= can take one of these three values:
            # enforcing – SELinux security policy is enforced.
            # permissive – SELinux prints warnings instead of enforcing.
            # disabled – No SELinux policy is loaded.
            SELINUX=disabled
            # SELINUXTYPE= can take one of these two values:
            # targeted – Targeted processes are protected,
            # minimum – Modification of targeted policy. Only selected processes are protected.
            # mls – Multi Level Security protection.
            SELINUXTYPE=targeted

            When I ran dns with some variable setup as follows, the error above was gone:
            – name: kube2sky
            image: kubernetes/kube2sky:latest
            env:
            – name: “KUBERNETES_RO_SERVICE_HOST”
            value: “10.100.0.1”
            – name: “KUBERNETES_RO_SERVICE_PORT”
            value: “80”

            However, in “Installing cAdvisor and Heapster on bare metal Kubernetes”, Heapster doesn’t seem to look up DNS. The following is log of Heapster container:
            Database creation failed: Post http://monitoring-influxdb/db?u=root&p=root: dial tcp: lookup monitoring-influxdb: no such host
            E0307 21:20:22.205571 6 heapster.go:52] Post http://monitoring-influxdb/db?u=root&p=root: dial tcp: lookup monitoring-influxdb: no such host

            The following shows some logs:
            minion#docker logs
            2015/03/09 02:45:46 watchLoop channel closed
            2015/03/09 02:45:46 Setting dns record: kube-dns.default.kubacis.local. -> 10.100.0.10:53
            2015/03/09 02:45:46 Setting dns record: kubernetes.default.kubacis.local. -> 10.100.0.2:443
            2015/03/09 02:45:46 Setting dns record: kubernetes-ro.default.kubacis.local. -> 10.100.0.1:80

            minion#docker logs
            [skydns] Mar 8 05:45:47.004 INFO | falling back to default configuration, could not read from etcd:100: Key not found (/skydns/config) [5]
            [skydns] Mar 8 05:45:47.004 INFO | ready for queries on kubacis.local. for tcp://0.0.0.0:53 [rcache 0]
            [skydns] Mar 8 05:45:47.004 INFO | ready for queries on kubacis.local. for udp://0.0.0.0:53 [rcache 0]

            minion#docker logs
            [etcd] Mar 8 05:45:42.311 WARNING | Using the directory kube-dns-cegta.etcd as the etcd curation directory because a directory was not specified.
            [etcd] Mar 8 05:45:42.311 INFO | kube-dns-cegta is starting a new cluster
            [etcd] Mar 8 05:45:42.313 INFO | etcd server [name kube-dns-cegta, listen on :4001, advertised url http://127.0.0.1:4001%5D
            [etcd] Mar 8 05:45:42.313 INFO | peer server [name kube-dns-cegta, listen on :7001, advertised url http://127.0.0.1:7001%5D
            [etcd] Mar 8 05:45:42.313 INFO | kube-dns-cegta starting in peer mode
            [etcd] Mar 8 05:45:42.313 INFO | kube-dns-cegta: state changed from ‘initialized’ to ‘follower’.
            [etcd] Mar 8 05:45:42.314 INFO | kube-dns-cegta: state changed from ‘follower’ to ‘leader’.
            [etcd] Mar 8 05:45:42.314 INFO | kube-dns-cegta: leader changed from ” to ‘kube-dns-cegta’.
            [etcd] Mar 8 07:08:36.870 INFO | kube-dns-cegta: snapshot of 10001 events at index 10001 completed
            [etcd] Mar 8 08:31:34.353 INFO | kube-dns-cegta: snapshot of 10006 events at index 20007 completed
            [etcd] Mar 8 09:54:31.872 INFO | kube-dns-cegta: snapshot of 10003 events at index 30010 completed
            [etcd] Mar 8 11:17:29.379 INFO | kube-dns-cegta: snapshot of 10006 events at index 40016 completed
            [etcd] Mar 8 12:40:26.847 INFO | kube-dns-cegta: snapshot of 10003 events at index 50019 completed
            [etcd] Mar 8 14:03:24.436 INFO | kube-dns-cegta: snapshot of 10006 events at index 60025 completed
            [etcd] Mar 8 15:26:21.996 INFO | kube-dns-cegta: snapshot of 10006 events at index 70031 completed
            [etcd] Mar 8 16:49:19.536 INFO | kube-dns-cegta: snapshot of 10003 events at index 80034 completed
            [etcd] Mar 8 18:12:17.065 INFO | kube-dns-cegta: snapshot of 10006 events at index 90040 completed
            [etcd] Mar 8 19:35:14.585 INFO | kube-dns-cegta: snapshot of 10003 events at index 100043 completed
            [etcd] Mar 8 20:58:12.040 INFO | kube-dns-cegta: snapshot of 10006 events at index 110049 completed
            [etcd] Mar 8 22:21:09.544 INFO | kube-dns-cegta: snapshot of 10006 events at index 120055 completed
            [etcd] Mar 8 23:44:07.043 INFO | kube-dns-cegta: snapshot of 10003 events at index 130058 completed
            [etcd] Mar 9 01:07:04.639 INFO | kube-dns-cegta: snapshot of 10006 events at index 140064 completed
            [etcd] Mar 9 02:30:02.131 INFO | kube-dns-cegta: snapshot of 10003 events at index 150067 completed

            [root@webpod /]# dig kubernetes-ro.default.kubacis.local
            ; <> DiG 9.8.2rc1-RedHat-9.8.2-0.30.rc1.el6 <> kubernetes-ro.default.kubacis .local
            ;; global options: +cmd
            ;; connection timed out; no servers could be reached

            Reply

            1. Gil Lee’s avatar

              I got the container logs from skydns, kube2sky, and etcd in order.

              Reply

            2. Jon Langemak’s avatar

              So the issue is with Heapster? I don’t see the Heapster service as a registered entry in DNS on the kube-sky container. Is the service created in the cluster? The influx-db and Heapster services need to be created before the DNS record gets registered for them.

              Reply

              1. Gil Lee’s avatar

                Yes, I created the services in the cluster and I launched Heapster services before the DNS service runs:
                # docker logs {kube2sky container id}
                2015/03/09 23:02:39 Using http://10.100.0.1:80 for kubernetes master
                2015/03/09 23:02:39 Using kubernetes API v1beta1
                2015/03/09 23:02:39 Setting dns record: kube-dns.default.kubacis.local. -> 10.100.0.10:53
                2015/03/09 23:02:39 Setting dns record: kubernetes.default.kubacis.local. -> 10.100.0.2:443
                2015/03/09 23:02:39 Setting dns record: kubernetes-ro.default.kubacis.local. -> 10.100.0.1:80
                2015/03/09 23:03:50 Setting dns record: monitoring-grafana.default.kubacis.local. -> 10.100.83.95:80
                2015/03/09 23:03:55 Setting dns record: monitoring-heapster.default.kubacis.local. -> 10.100.112.254:80
                2015/03/09 23:04:00 Setting dns record: monitoring-influxdb.default.kubacis.local. -> 10.100.80.76:80

                The main issue was heapster. But digging the issue, I realized that I cannot access the local etcd container for DNS service. The etcd is running in a minion as a container:
                # ps aux | grep etcd
                root 2047 0.2 0.1 423064 11724 ? Ssl Mar07 8:50 /opt/kubernetes/kube-proxy –etcd_servers=http://192.168.2.1:4001 –logtostderr=true
                root 21725 0.3 0.2 432248 17472 ? Ssl Mar08 8:10 /opt/kubernetes/kubelet –address=192.168.2.2 –port=10250 –hostname_override=192.168.2.2 –etcd_servers=http://192.168.2.1:4001 –logtostderr=true –config=/etc/kubernetes/manifests/ –cluster_dns=10.100.0.10 –cluster_domain=kubacis.local
                root 24531 0.0 0.0 14892 5836 ? Ssl 22:41 0:00 /etcd /etcd -bind-addr=127.0.0.1 -peer-bind-addr=127.0.0.1

                Curling to local etcd is not working:
                # curl http://127.0.0.1:4001/version
                curl: (7) Failed connect to 127.0.0.1:4001; Connection refused

                So I guess that due to this issue, heapster isn’t able to look up monitoring-influxdb. I have no idea what I should do to resolve this in my environment.

              2. Jon Langemak’s avatar

                Can you attach to a running container and make sure that the DNS entries are resolving correctly?

              3. Gil Lee’s avatar

                I attached to your webpod container and tried to look up an entries. There is a connection problem.
                [root@webpod /]# dig monitoring-influxdb.default.kubacis.local

                ; <> DiG 9.8.2rc1-RedHat-9.8.2-0.30.rc1.el6 <> monitoring-influxdb.default.kubacis.local
                ;; global options: +cmd
                ;; connection timed out; no servers could be reached

                This is a route result in the container:
                [root@webpod /]# route -n
                Kernel IP routing table
                Destination Gateway Genmask Flags Metric Ref Use Iface
                0.0.0.0 10.244.204.1 0.0.0.0 UG 0 0 0 eth0
                10.244.204.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

                This is docker config in the host where the webpod container is running:
                minion# more /etc/sysconfig/docker
                OPTIONS=–selinux-enabled -d -H fd:// –bip=10.244.204.1/24 –iptables=fal
                se –ip-masq=false

                master# kubectl get pods
                POD IP CONTAINER(S) IMAGE(S) HOST LABELS STATUS CREATED
                kube-dns-4s9mi 10.244.202.12 etcd quay.io/coreos/etcd:latest 192.168.2.2/192.168.2.2 k8s-app=kube-dns,kubernetes.io/cluster-service=true Running 32 minutes
                kube2sky kubernetes/kube2sky:latest
                skydns kubernetes/skydns:latest
                monitoring-heapster-controller-gb50a 10.244.204.3 heapster kubernetes/heapster:v0.8 192.168.2.4/192.168.2.4 name=heapster,uses=monitoring-influxdb Running 4 hours
                monitoring-influx-grafana-controller-ckn42 10.244.202.2 influxdb kubernetes/heapster_influxdb:v0.3 192.168.2.2/192.168.2.2 name=influxGrafana Running 4 hours
                grafana kubernetes/heapster_grafana:v0.4
                webpod 10.244.204.6 webpod80 jonlangemak/docker:web_container_80 192.168.2.4/192.168.2.4 name=web Running 2 hours
                webpod8080 jonlangemak/docker:web_container_8080

                master# kubectl get services
                NAME LABELS SELECTOR IP PORT
                kube-dns k8s-app=kube-dns,kubernetes.io/cluster-service=true k8s-app=kube-dns 10.100.0.10 53
                kubernetes component=apiserver,provider=kubernetes 10.100.0.2 443
                kubernetes-ro component=apiserver,provider=kubernetes 10.100.0.1 80
                monitoring-grafana name=influxGrafana 10.100.83.95 80
                monitoring-heapster name=heapster 10.100.112.254 80
                monitoring-influxdb name=influxGrafana 10.100.80.76 80

                Reply

                1. Jon Langemak’s avatar

                  So you are unable to resolve service DNS entries from a container? What’s your kubelet configuration look like? Did you specify the DNS namespace and DNS service IP address in there?

                  Reply

                  1. Gil Lee’s avatar

                    Yes. I’m unable to do. I attached my kubelet configuration:
                    minion# more /usr/lib/systemd/system/kubernetes-kubelet.service
                    [Unit]
                    Description=Kubernetes Kubelet
                    After=etcd.service
                    After=docker.service
                    Wants=etcd.service
                    Wants=docker.service

                    [Service]
                    ExecStart=/opt/kubernetes/kubelet \
                    –address=192.168.2.2 \
                    –port=10250 \
                    –hostname_override=192.168.2.2 \
                    –etcd_servers=http://192.168.2.1:4001 \
                    –logtostderr=true \
                    –config=/etc/kubernetes/manifests/ \
                    –cluster_dns=10.100.0.10 \
                    –cluster_domain=kubacis.local \
                    Restart=on-failure
                    RestartSec=5

                    [Install]
                    WantedBy=multi-user.target

                    Reply

                    1. Jon Langemak’s avatar

                      Ok – That config looks right. Does the kube-dns service show up as being 10.100.0.10 on the output of kubectl get services? Have you rebooted the nodes since you put the kubelet config in?

                      Do you see any logs in the skyDNS container when you attempt to resolve the DNS names it shows as registered in kube2sky?

                      Reply

                      1. Gil Lee’s avatar

                        @Does the kube-dns service show up as being 10.100.0.10 on the output of kubectl get services? – Yes.
                        master# kubectl get services
                        NAME LABELS SELECTOR IP PORT
                        kube-dns k8s-app=kube-dns,kubernetes.io/cluster-service=true k8s-app=kube-dns 10.100.0.10 53

                        @ Have you rebooted the nodes since you put the kubelet config in? Yes I have. I just rebooted all nodes again and re-launched all services.

                        @Do you see any logs in the skyDNS container when you attempt to resolve the DNS names it shows as registered in kube2sky? I don’t see any additional logs using docker logs -f {skyDNS container id} during my DNS resolve.

                      2. Jon Langemak’s avatar

                        So it seems like DNS resolution in general isnt working. Can you resolve any of the skyDNS entries? Check your resolv.conf in the container and make sure the DNS server is set to the service IP. I’d also try resolving directly at the skyDNS server rather than the service to rule out any issues on the service end of things.

                      3. Gil Lee’s avatar

                        Interesting!
                        I have two minions. I realized that when webpod container is running where kube-dns runs, DNS lookup is working:

                        Case 1- kube-dns and webpod are running in the same minion:
                        [root@webpod /]# dig kube-dns.default.kubacis.local

                        ; <> DiG 9.8.2rc1-RedHat-9.8.2-0.30.rc1.el6 <> kube-dns.default.kubacis.local
                        ;; global options: +cmd
                        ;; Got answer:
                        ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48592
                        ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

                        ;; QUESTION SECTION:
                        ;kube-dns.default.kubacis.local. IN A

                        ;; ANSWER SECTION:
                        kube-dns.default.kubacis.local. 30 IN A 10.100.0.10

                        ;; Query time: 2 msec
                        ;; SERVER: 10.100.0.10#53(10.100.0.10)
                        ;; WHEN: Wed Mar 11 15:52:03 2015
                        ;; MSG SIZE rcvd: 64

                        Case 2 – Distinct minions:
                        [root@webpod2 /]# dig kube-dns.default.kubacis.local

                        ; <> DiG 9.8.2rc1-RedHat-9.8.2-0.30.rc1.el6 <> kube-dns.default.kubacis.local
                        ;; global options: +cmd
                        ;; connection timed out; no servers could be reached

                        Containers in both case 1 and case 2 have the same resolve.conf.
                        # more /etc/resolve.conf
                        search default.kubacis.local kubacis.local
                        nameserver 10.100.0.10

                        Reply

                        1. Jon Langemak’s avatar

                          I’d start looking at the network and iptables end of things. Can you ensure that both minions have the same iptable redirect rules?

                          Reply

                        2. Rick Pettit’s avatar

                          Great write up. One very minor typo though I think in the section regarding the replication controller. The bit that reads, “Once it’s deployed, you should see the pods running…” I think should either have “pods” replaced with “pod” (singular), or with “containers”.

                          Reply

                          1. Jon Langemak’s avatar

                            Good catch on that. The previous wording was confusing.

                            Thanks!

                            Reply

Reply

Your email address will not be published. Required fields are marked *