Kubernetes DNS config on bare metal

      22 Comments on Kubernetes DNS config on bare metal

One of the ‘newer’ functions of Kubernetes is the ability to register service names in DNS.  More specifically, to register them in a DNS server running in the Kubernetes cluster.  To do this, the clever folks at Google came up with a solution that leverages SkyDNS and another container (called kube2sky) to read the service entries and insert them as DNS entries.  Pretty slick huh?

Beyond the containers to run the DNS service, we also need to tell the pods to use this particular DNS server for DNS resolution.  This is done by adding a couple of lines of config to the kubernetes-kubelet service.  Once that’s done, we can configure the Kubernetes service and the replication controller for the SkyDNS pod.  So let’s start with the kubelet service configuration.  Let’s edit our service definition located here…

/usr/lib/systemd/system/kubernetes-kubelet.service

Our new config will look like this…

[Unit]
Description=Kubernetes Kubelet
After=etcd.service
After=docker.service
Wants=etcd.service
Wants=docker.service

[Service]
ExecStart=/opt/kubernetes/kubelet \
--address=10.20.30.62 \
--port=10250 \
--hostname_override=10.20.30.62 \
--etcd_servers=http://10.20.30.61:4001 \
--logtostderr=true \
--cluster_dns=10.100.0.10 \
--cluster_domain=kubdomain.local
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Notice that all we added was a line indicating that the service should use 10.100.0.10 (and IP out of our Portal Net allocation) as a DNS server and another line setting the cluster DNS namespace is ‘kubdomain.local’.  You should substitute the domain name you want to use here and make sure you remember it for when we configure the DNS server.  Once that’s done, save the file and reload the service…

systemctl daemon-reload
systemctl restart kubernetes-kubelet

Duplicate this step on all of your Kubernetes nodes.  The next step is to configure the replication controller for the actual service.  There are two files we’ll use for this, the first is for the service definition and looks like this…

kind: Service
apiVersion: v1beta1
id: kube-dns
namespace: default
protocol: UDP
port: 53
portalIP: 10.100.0.10
containerPort: 53
labels:
  k8s-app: kube-dns
  kubernetes.io/cluster-service: "true"
selector:
  k8s-app: kube-dns

The only item I edited was the PortalIP.  Note, this needs to be the same IP you used when you configured the nodes kubelet service DNS server.  Again, this IP is only relevant in the cluster so there’s no need to be picky and you can certainly use the same one I did so long as it’s in your portal net network.  Deploy this service into the cluster and then let’s move onto the replication controller configuration.  That file looks like this…

kind: ReplicationController
apiVersion: v1beta1
id: kube-dns
namespace: default
labels:
  k8s-app: kube-dns
  kubernetes.io/cluster-service: "true"
desiredState:
  replicas: 1
  replicaSelector:
    k8s-app: kube-dns
  podTemplate:
    labels:
      k8s-app: kube-dns
      kubernetes.io/cluster-service: "true"
    desiredState:
      manifest:
        version: v1beta2
        id: kube-dns
        dnsPolicy: "Default"  # Don't use cluster DNS.
        containers:
          - name: etcd
            image: quay.io/coreos/etcd:latest
            command: [
                    "/etcd",
                    "-bind-addr=127.0.0.1",
                    "-peer-bind-addr=127.0.0.1",
            ]
          - name: kube2sky
            image: kubernetes/kube2sky:1.0
            command: [
                    # entrypoint = "/kube2sky",
                    "-domain=kubdomain.local",
            ]
          - name: skydns
            image: kubernetes/skydns:2014-12-23-001
            command: [
                    # entrypoint = "/skydns",
                    "-machines=http://localhost:4001",
                    "-addr=0.0.0.0:53",
                    "-domain=kubdomain.local",
            ]
            ports:
              - name: dns
                containerPort: 53
                protocol: UDP

The only thing you’ll need to change here is your domain under the container definitions.  Once this is done, create the replication controller in Kubernetes.  Once it’s deployed, you should see the pod running…

image
Go ahead and connect to the host that’s running the pod.  In my case, that’s kubminion2 or 10.20.30.63.  Let’s verify that it sees the pod running…

image
Now let’s use the ‘docker logs’ command to see if the kube2sky container is working like it should…

image
Awesome!  So it pulled all of my service definitions and added DNS records for them!  Disregard the monitoring services for now, that’s for an upcoming post about Heapster.

So the true test will be to log into a Kubernetes pod container and see if this all works.    Connect into one of your containers and try resolving one of the names…

image
Looks good!  I would recommend doing this DNS setup as one of the ‘base setup’ tasks when you build a cluster.  More and more of the cluster add-ons are using this feature so its best just to have it from the get go.  Plus it’s pretty slick for using when you build your own services!

22 thoughts on “Kubernetes DNS config on bare metal

  1. Tim H

    I still want to make DNS from the node work by default, but it’s a matter of configuring the node’s resolv.conf, which suddenly gets very distro-specific. But once we do that, we can run a Docker registry as a pod, and then pull other images from that registry by name, rather than by IP.

    Reply
    1. Jon Langemak Post author

      So this brings up an interesting point. Up until now I’ve relied on mostly IP addresses rather than DNS names. I’m starting to see a lot of the cluster add on services adopting the DNS lookup for service resolution model (which is great!) but what’s the scope? I mean, at this point my Kubernetes nodes (minions) have lived in my existing DNS (interubernet.local) and I created a new DNS name space (kubdomain.local) for the SkyDNS setup. My thought was that this DNS could be anything I wanted since it was only being used for service lookup. But now Im wondering how containers will do external resolution, I guess I just hadnt run into that yet.

      Should service entries and the nodes be in the same DNS namespace?

      Reply
  2. Gil Lee

    Thanks for your wonderful tutorial. I’m trying out Kubernetes on bare metal almost exactly following your instructions. Then here, DNS configuration, I stocked. kube2sky container exited with “KUBERNETES_RO_SERVICE_HOST is not defined.” Could you give me any advice to address this ?

    Reply
    1. Jon Langemak Post author

      Are you running SELinux on the host? That sounds vaguely like something I ran into when I had forgot to disable it the first go around.

      Reply
      1. Gil Lee

        Thanks for your response. I’m installing kubernetes on three bare metal machines running CentOS 7. SELinux was not enabled on them:
        # cat /etc/sysconfig/selinux

        # This file controls the state of SELinux on the system.
        # SELINUX= can take one of these three values:
        # enforcing – SELinux security policy is enforced.
        # permissive – SELinux prints warnings instead of enforcing.
        # disabled – No SELinux policy is loaded.
        SELINUX=disabled
        # SELINUXTYPE= can take one of these two values:
        # targeted – Targeted processes are protected,
        # minimum – Modification of targeted policy. Only selected processes are protected.
        # mls – Multi Level Security protection.
        SELINUXTYPE=targeted

        When I ran dns with some variable setup as follows, the error above was gone:
        – name: kube2sky
        image: kubernetes/kube2sky:latest
        env:
        – name: “KUBERNETES_RO_SERVICE_HOST”
        value: “10.100.0.1”
        – name: “KUBERNETES_RO_SERVICE_PORT”
        value: “80”

        However, in “Installing cAdvisor and Heapster on bare metal Kubernetes”, Heapster doesn’t seem to look up DNS. The following is log of Heapster container:
        Database creation failed: Post http://monitoring-influxdb/db?u=root&p=root: dial tcp: lookup monitoring-influxdb: no such host
        E0307 21:20:22.205571 6 heapster.go:52] Post http://monitoring-influxdb/db?u=root&p=root: dial tcp: lookup monitoring-influxdb: no such host

        The following shows some logs:
        minion#docker logs
        2015/03/09 02:45:46 watchLoop channel closed
        2015/03/09 02:45:46 Setting dns record: kube-dns.default.kubacis.local. -> 10.100.0.10:53
        2015/03/09 02:45:46 Setting dns record: kubernetes.default.kubacis.local. -> 10.100.0.2:443
        2015/03/09 02:45:46 Setting dns record: kubernetes-ro.default.kubacis.local. -> 10.100.0.1:80

        minion#docker logs
        [skydns] Mar 8 05:45:47.004 INFO | falling back to default configuration, could not read from etcd:100: Key not found (/skydns/config) [5]
        [skydns] Mar 8 05:45:47.004 INFO | ready for queries on kubacis.local. for tcp://0.0.0.0:53 [rcache 0]
        [skydns] Mar 8 05:45:47.004 INFO | ready for queries on kubacis.local. for udp://0.0.0.0:53 [rcache 0]

        minion#docker logs
        [etcd] Mar 8 05:45:42.311 WARNING | Using the directory kube-dns-cegta.etcd as the etcd curation directory because a directory was not specified.
        [etcd] Mar 8 05:45:42.311 INFO | kube-dns-cegta is starting a new cluster
        [etcd] Mar 8 05:45:42.313 INFO | etcd server [name kube-dns-cegta, listen on :4001, advertised url http://127.0.0.1:4001%5D
        [etcd] Mar 8 05:45:42.313 INFO | peer server [name kube-dns-cegta, listen on :7001, advertised url http://127.0.0.1:7001%5D
        [etcd] Mar 8 05:45:42.313 INFO | kube-dns-cegta starting in peer mode
        [etcd] Mar 8 05:45:42.313 INFO | kube-dns-cegta: state changed from ‘initialized’ to ‘follower’.
        [etcd] Mar 8 05:45:42.314 INFO | kube-dns-cegta: state changed from ‘follower’ to ‘leader’.
        [etcd] Mar 8 05:45:42.314 INFO | kube-dns-cegta: leader changed from ” to ‘kube-dns-cegta’.
        [etcd] Mar 8 07:08:36.870 INFO | kube-dns-cegta: snapshot of 10001 events at index 10001 completed
        [etcd] Mar 8 08:31:34.353 INFO | kube-dns-cegta: snapshot of 10006 events at index 20007 completed
        [etcd] Mar 8 09:54:31.872 INFO | kube-dns-cegta: snapshot of 10003 events at index 30010 completed
        [etcd] Mar 8 11:17:29.379 INFO | kube-dns-cegta: snapshot of 10006 events at index 40016 completed
        [etcd] Mar 8 12:40:26.847 INFO | kube-dns-cegta: snapshot of 10003 events at index 50019 completed
        [etcd] Mar 8 14:03:24.436 INFO | kube-dns-cegta: snapshot of 10006 events at index 60025 completed
        [etcd] Mar 8 15:26:21.996 INFO | kube-dns-cegta: snapshot of 10006 events at index 70031 completed
        [etcd] Mar 8 16:49:19.536 INFO | kube-dns-cegta: snapshot of 10003 events at index 80034 completed
        [etcd] Mar 8 18:12:17.065 INFO | kube-dns-cegta: snapshot of 10006 events at index 90040 completed
        [etcd] Mar 8 19:35:14.585 INFO | kube-dns-cegta: snapshot of 10003 events at index 100043 completed
        [etcd] Mar 8 20:58:12.040 INFO | kube-dns-cegta: snapshot of 10006 events at index 110049 completed
        [etcd] Mar 8 22:21:09.544 INFO | kube-dns-cegta: snapshot of 10006 events at index 120055 completed
        [etcd] Mar 8 23:44:07.043 INFO | kube-dns-cegta: snapshot of 10003 events at index 130058 completed
        [etcd] Mar 9 01:07:04.639 INFO | kube-dns-cegta: snapshot of 10006 events at index 140064 completed
        [etcd] Mar 9 02:30:02.131 INFO | kube-dns-cegta: snapshot of 10003 events at index 150067 completed

        [root@webpod /]# dig kubernetes-ro.default.kubacis.local
        ; <> DiG 9.8.2rc1-RedHat-9.8.2-0.30.rc1.el6 <> kubernetes-ro.default.kubacis .local
        ;; global options: +cmd
        ;; connection timed out; no servers could be reached

        Reply
        1. Jon Langemak Post author

          So the issue is with Heapster? I don’t see the Heapster service as a registered entry in DNS on the kube-sky container. Is the service created in the cluster? The influx-db and Heapster services need to be created before the DNS record gets registered for them.

          Reply
          1. Gil Lee

            Yes, I created the services in the cluster and I launched Heapster services before the DNS service runs:
            # docker logs {kube2sky container id}
            2015/03/09 23:02:39 Using http://10.100.0.1:80 for kubernetes master
            2015/03/09 23:02:39 Using kubernetes API v1beta1
            2015/03/09 23:02:39 Setting dns record: kube-dns.default.kubacis.local. -> 10.100.0.10:53
            2015/03/09 23:02:39 Setting dns record: kubernetes.default.kubacis.local. -> 10.100.0.2:443
            2015/03/09 23:02:39 Setting dns record: kubernetes-ro.default.kubacis.local. -> 10.100.0.1:80
            2015/03/09 23:03:50 Setting dns record: monitoring-grafana.default.kubacis.local. -> 10.100.83.95:80
            2015/03/09 23:03:55 Setting dns record: monitoring-heapster.default.kubacis.local. -> 10.100.112.254:80
            2015/03/09 23:04:00 Setting dns record: monitoring-influxdb.default.kubacis.local. -> 10.100.80.76:80

            The main issue was heapster. But digging the issue, I realized that I cannot access the local etcd container for DNS service. The etcd is running in a minion as a container:
            # ps aux | grep etcd
            root 2047 0.2 0.1 423064 11724 ? Ssl Mar07 8:50 /opt/kubernetes/kube-proxy –etcd_servers=http://192.168.2.1:4001 –logtostderr=true
            root 21725 0.3 0.2 432248 17472 ? Ssl Mar08 8:10 /opt/kubernetes/kubelet –address=192.168.2.2 –port=10250 –hostname_override=192.168.2.2 –etcd_servers=http://192.168.2.1:4001 –logtostderr=true –config=/etc/kubernetes/manifests/ –cluster_dns=10.100.0.10 –cluster_domain=kubacis.local
            root 24531 0.0 0.0 14892 5836 ? Ssl 22:41 0:00 /etcd /etcd -bind-addr=127.0.0.1 -peer-bind-addr=127.0.0.1

            Curling to local etcd is not working:
            # curl http://127.0.0.1:4001/version
            curl: (7) Failed connect to 127.0.0.1:4001; Connection refused

            So I guess that due to this issue, heapster isn’t able to look up monitoring-influxdb. I have no idea what I should do to resolve this in my environment.

          2. Jon Langemak Post author

            Can you attach to a running container and make sure that the DNS entries are resolving correctly?

  3. Gil Lee

    I attached to your webpod container and tried to look up an entries. There is a connection problem.
    [root@webpod /]# dig monitoring-influxdb.default.kubacis.local

    ; <> DiG 9.8.2rc1-RedHat-9.8.2-0.30.rc1.el6 <> monitoring-influxdb.default.kubacis.local
    ;; global options: +cmd
    ;; connection timed out; no servers could be reached

    This is a route result in the container:
    [root@webpod /]# route -n
    Kernel IP routing table
    Destination Gateway Genmask Flags Metric Ref Use Iface
    0.0.0.0 10.244.204.1 0.0.0.0 UG 0 0 0 eth0
    10.244.204.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

    This is docker config in the host where the webpod container is running:
    minion# more /etc/sysconfig/docker
    OPTIONS=–selinux-enabled -d -H fd:// –bip=10.244.204.1/24 –iptables=fal
    se –ip-masq=false

    master# kubectl get pods
    POD IP CONTAINER(S) IMAGE(S) HOST LABELS STATUS CREATED
    kube-dns-4s9mi 10.244.202.12 etcd quay.io/coreos/etcd:latest 192.168.2.2/192.168.2.2 k8s-app=kube-dns,kubernetes.io/cluster-service=true Running 32 minutes
    kube2sky kubernetes/kube2sky:latest
    skydns kubernetes/skydns:latest
    monitoring-heapster-controller-gb50a 10.244.204.3 heapster kubernetes/heapster:v0.8 192.168.2.4/192.168.2.4 name=heapster,uses=monitoring-influxdb Running 4 hours
    monitoring-influx-grafana-controller-ckn42 10.244.202.2 influxdb kubernetes/heapster_influxdb:v0.3 192.168.2.2/192.168.2.2 name=influxGrafana Running 4 hours
    grafana kubernetes/heapster_grafana:v0.4
    webpod 10.244.204.6 webpod80 jonlangemak/docker:web_container_80 192.168.2.4/192.168.2.4 name=web Running 2 hours
    webpod8080 jonlangemak/docker:web_container_8080

    master# kubectl get services
    NAME LABELS SELECTOR IP PORT
    kube-dns k8s-app=kube-dns,kubernetes.io/cluster-service=true k8s-app=kube-dns 10.100.0.10 53
    kubernetes component=apiserver,provider=kubernetes 10.100.0.2 443
    kubernetes-ro component=apiserver,provider=kubernetes 10.100.0.1 80
    monitoring-grafana name=influxGrafana 10.100.83.95 80
    monitoring-heapster name=heapster 10.100.112.254 80
    monitoring-influxdb name=influxGrafana 10.100.80.76 80

    Reply
    1. Jon Langemak Post author

      So you are unable to resolve service DNS entries from a container? What’s your kubelet configuration look like? Did you specify the DNS namespace and DNS service IP address in there?

      Reply
      1. Gil Lee

        Yes. I’m unable to do. I attached my kubelet configuration:
        minion# more /usr/lib/systemd/system/kubernetes-kubelet.service
        [Unit]
        Description=Kubernetes Kubelet
        After=etcd.service
        After=docker.service
        Wants=etcd.service
        Wants=docker.service

        [Service]
        ExecStart=/opt/kubernetes/kubelet \
        –address=192.168.2.2 \
        –port=10250 \
        –hostname_override=192.168.2.2 \
        –etcd_servers=http://192.168.2.1:4001 \
        –logtostderr=true \
        –config=/etc/kubernetes/manifests/ \
        –cluster_dns=10.100.0.10 \
        –cluster_domain=kubacis.local \
        Restart=on-failure
        RestartSec=5

        [Install]
        WantedBy=multi-user.target

        Reply
        1. Jon Langemak Post author

          Ok – That config looks right. Does the kube-dns service show up as being 10.100.0.10 on the output of kubectl get services? Have you rebooted the nodes since you put the kubelet config in?

          Do you see any logs in the skyDNS container when you attempt to resolve the DNS names it shows as registered in kube2sky?

          Reply
          1. Gil Lee

            @Does the kube-dns service show up as being 10.100.0.10 on the output of kubectl get services? – Yes.
            master# kubectl get services
            NAME LABELS SELECTOR IP PORT
            kube-dns k8s-app=kube-dns,kubernetes.io/cluster-service=true k8s-app=kube-dns 10.100.0.10 53

            @ Have you rebooted the nodes since you put the kubelet config in? Yes I have. I just rebooted all nodes again and re-launched all services.

            @Do you see any logs in the skyDNS container when you attempt to resolve the DNS names it shows as registered in kube2sky? I don’t see any additional logs using docker logs -f {skyDNS container id} during my DNS resolve.

          2. Jon Langemak Post author

            So it seems like DNS resolution in general isnt working. Can you resolve any of the skyDNS entries? Check your resolv.conf in the container and make sure the DNS server is set to the service IP. I’d also try resolving directly at the skyDNS server rather than the service to rule out any issues on the service end of things.

  4. Gil Lee

    Interesting!
    I have two minions. I realized that when webpod container is running where kube-dns runs, DNS lookup is working:

    Case 1- kube-dns and webpod are running in the same minion:
    [root@webpod /]# dig kube-dns.default.kubacis.local

    ; <> DiG 9.8.2rc1-RedHat-9.8.2-0.30.rc1.el6 <> kube-dns.default.kubacis.local
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48592
    ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

    ;; QUESTION SECTION:
    ;kube-dns.default.kubacis.local. IN A

    ;; ANSWER SECTION:
    kube-dns.default.kubacis.local. 30 IN A 10.100.0.10

    ;; Query time: 2 msec
    ;; SERVER: 10.100.0.10#53(10.100.0.10)
    ;; WHEN: Wed Mar 11 15:52:03 2015
    ;; MSG SIZE rcvd: 64

    Case 2 – Distinct minions:
    [root@webpod2 /]# dig kube-dns.default.kubacis.local

    ; <> DiG 9.8.2rc1-RedHat-9.8.2-0.30.rc1.el6 <> kube-dns.default.kubacis.local
    ;; global options: +cmd
    ;; connection timed out; no servers could be reached

    Containers in both case 1 and case 2 have the same resolve.conf.
    # more /etc/resolve.conf
    search default.kubacis.local kubacis.local
    nameserver 10.100.0.10

    Reply
    1. Jon Langemak Post author

      I’d start looking at the network and iptables end of things. Can you ensure that both minions have the same iptable redirect rules?

      Reply
  5. Rick Pettit

    Great write up. One very minor typo though I think in the section regarding the replication controller. The bit that reads, “Once it’s deployed, you should see the pods running…” I think should either have “pods” replaced with “pod” (singular), or with “containers”.

    Reply
  6. Pingback: Overview Container Orchestration Tools | blog.elsdoerfer.name

Leave a Reply to Jon Langemak Cancel reply

Your email address will not be published. Required fields are marked *