Installing cAdvisor and Heapster on bare metal Kubernetes

NOTE: Kubernetes has evolved! The cAdvisor module is now integrated directly into the Kubelet binary.  These directions are no longer entirely accurate because of this.  The cAdvisor manifest does not need to be deployed.  If you’re looking for current info on deploying Kubernetes, please reference the related config files in my GitHub salt repo.  I’ll be doing my best to keep those as up to date as possible.

If you’ve spent some time with Kubernetes, or docker in general, you probably start to wonder about performance.  Moreover, you’re probably wondering how to gauge performance of he overall host as well as the containers running on it.  This is where cAdvisor comes in.  CAdvisor is a open source tool for monitoring docker and it’s running containers.  The best part about cAdvisor is that it has native docker support and is super easy to integrate into an existing Kubernetes cluster.  Additionally, cAdvisor runs in a container (starting to see why docker is awesome?) so the configuration changes required on the host are super minimal.  AKA – You just need to tell the host to run the container.

In addition to installing cAdvisor on our bare metal Kubernetes cluster, we’re also going to install another awesome open source Google tool call Heapster.  Heapster gathers all of the data from each Kubernetes node via cAdvisor and puts it all together for you in one spot.

So let’s get started with installing cAdvisor…

The cAdvisor container needs to run on each host you want cAdvisor to monitor.  While we could do this through the Kubernetes master, it’s easier just to tell each host to run the container when it boots.  Luckily, the Kubernetes-Kubelet service can be configured to read and process local ‘manifests’.  Manifests are VERY similar to a pod configuration in Kubernetes.  In fact, they’re identical.  You essentially just remove the piece of the pod config telling Kubernetes that it’s a pod since by default, a manifest is a pod.  So the first thing we need to do is edit our Kubelet service configuration to tell it to look for local manifests.  Let’s edit our service definition located here…

/usr/lib/systemd/system/kubernetes-kubelet.service

Our new config will look like this…

[Unit]
Description=Kubernetes Kubelet
After=etcd.service
After=docker.service
Wants=etcd.service
Wants=docker.service

[Service]
ExecStart=/opt/kubernetes/kubelet \
--address=10.20.30.62 \
--port=10250 \
--hostname_override=10.20.30.62 \
--etcd_servers=http://10.20.30.61:4001 \
--logtostderr=true \
--config=/etc/kubernetes/manifests/
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Notice that all we added was the line indicating that the service should look for it’s configuration in the directory ‘/etc/kubernetes/manifests’.  This directory shouldn’t exist since we’ve never used it before so let’s save this file, and create the directories that we need…

mkdir /etc/kubernetes
mkdir /etc/kubernetes/manifests/

The next step is to create the manifest.  So let’s move into our new folder and create a file called ‘cadvisor.manifest’.  In this file, let’s put the following configuration…

version: v1beta2
id: cadvisor-agent
containers:
  - name: cadvisor
    image: google/cadvisor:0.8.0
    ports:
      - name: http
        containerPort: 8080
        hostPort: 4194
    volumeMounts:
      - name: varrun
        mountPath: /var/run
        readOnly: false
      - name: varlibdocker
        mountPath: /var/lib/docker
        readOnly: true
      - name: sysfs
        mountPath: /sys
        readOnly: true
volumes:
  - name: varrun
    source:
      hostDir:
        path: /var/run
  - name: varlibdocker
    source:
      hostDir:
        path: /var/lib/docker
  - name: sysfs
    source:
      hostDir:
        path: /sys

So this format should look pretty familiar to us from pod definitions we’ve seen in the past.  We can see that the container is mounting a number of volumes from which it will read the data it needs for cAdvisor to pull statistics.  Now that we have everything in place, let’s tell systemd to reload the service.  We’ll do so using these two commands…

systemctl daemon-reload
systemctl restart kubernetes-kubelet

Once we do that, the kubelet service will read the manifest file and start the process of downloading the starting the cAdvisor container.  It’s a small container image so within a matter of seconds we should see it running on kubminion1…

Once it’s running, all we have to do is browse to the host on port 4914 and we should see the cAdvisor page…

Scroll down further and we’ll see some interesting data about CPU and memory usage…

We can also dive into any running container and check to see how it’s running by scrolling up to the top and picking the ‘Containers’ link and then selecting the container we want to look at.  Pretty cool stuff!

Now let’s do this same configuration on our other 3 hosts.  When you’re done, verify that you can connect to the cAdvisor status page on all 4 hosts.  Now let’s configure Heapster…

Note: Heapster relies on the service DNS resolution.  If you don’t have that running yet, see this.

This is the awesome part.  I mean, the really awesome part.  Heapster uses two pods and three services deployed into the cluster for it to work.  The first pod is the heapster pod which does all of the data collection.  The second pod runs a database server (InfluxDB) along with a web front end to visualize the stats (Grafana).  That being said, now that cAdvisor is deployed, the hard work for us is over.  Now its just a matter of deploying some pods and services into the cluster and letting the magic happen.  Let’s look at the pod definitions we’ll be using…

Note(s): I had some issues in the latest release of Grafana.  There were two issues…

First – It turns out that the API port wasn’t configurable in the official release.  I suspect this is another difference between running this on bare metal and running this in GCE.  Regardless, I opened a pull request and had the code I used to fix this merged into the project (yay me!).  That being said, we can use the revised templates from GitHub, but the docker image isn’t updated yet with the fix I made.  I’ve got a docker image out there that does have the fix so I’ll be using that rather than the official released image.  I reached out to the guys that manage this to see when the official docker image will be updated. When that happens, we can go back to using the original containers from the project.

Second – We need to use a newer version of the Heapster container than what is specified in the pod template.  This is due to a bug that I’m only seeing on the bare metal side of things and isn’t reproducible on the GCE side.  They released a newer image that fixes this issue for me but the issue still needs to be debugged further.  I’m assuming that sooner or later the template will be updated to specify the new version but for now we need to update it from Heapster version .7 to version .8

All of the templates I show below have the two changes I listed above already made.  If you download the pod templates off of GitHub  directly you’ll need to make these changes yourself.  

The new deployment files (and associated containers) out on the Heapster GitHub page work as is!  The only change I had to make was to set my KUBERNETES_API_PORT port to 8443 in the InfluxDB/Grafana controller file.  The service and controller files located below are not updated so please see the new YAML files on the GitHub page located here https://github.com/GoogleCloudPlatform/heapster/tree/master/deploy  

The Grafana Kubernetes service (grafana-service.yaml)…

apiVersion: "v1beta1"
kind: "Service"
id: "monitoring-grafana"
port: 80
containerPort: 80
selector:
  name: "influxGrafana"

The Heapster Kubernetes service (heapster-service.yaml)…

apiVersion: "v1beta1"
kind: "Service"
id: "monitoring-heapster"
port: 80
containerPort: 8082
selector:
  name: "heapster"

The InfluxDB Kubernetes service (influxdb-service.yaml)…

apiVersion: "v1beta1"
kind: "Service"
id: "monitoring-influxdb"
port: 80
containerPort: 8086
selector:
  name: "influxGrafana"

The InfluxDB and Grafana Replication controller configuration (influxdb-grafana-controller.yaml)…

apiVersion: "v1beta1"
kind: "ReplicationController"
id: "monitoring-influx-grafana-controller"
desiredState:
  replicas: 1
  replicaSelector:
    name: "influxGrafana"
  podTemplate:
    labels:
      name: "influxGrafana"
    desiredState:
      manifest:
        version: "v1beta1"
        id: "monitoring-influx-grafana-controller"
        containers:
          - name: "influxdb"
            image: "kubernetes/heapster_influxdb:v0.3"
            ports:
              - containerPort: 8083
                hostPort: 8083
              - containerPort: 8086
                hostPort: 8086
          - name: "grafana"
            image: "jonlangemak/docker:heasptergrafana"
            ports:
              - containerPort: 80
            env:
              - name: "HTTP_USER"
                value: "admin"
              - name: "HTTP_PASS"
                value: "**None**"
              - name: "INFLUXDB_PROTO"
                value: "https"
              - name: "INFLUXDB_HOST"
                value: "localhost"
              - name: "INFLUXDB_PORT"
                value: ""
              - name: "KUBERNETES_API_PORT"
                value: "8443"
labels:
  name: "influxGrafana"

The Heapster Replication controller configuration (heapster-controller.yaml)…

apiVersion: "v1beta1"
id: "monitoring-heapster-controller"
kind: "ReplicationController"
desiredState:
  replicas: 1
  replicaSelector:
    name: "heapster"
  podTemplate:
    desiredState:
      manifest:
        version: "v1beta1"
        id: "monitoring-heapster-controller"
        containers:
          - name: "heapster"
            image: "kubernetes/heapster:v0.8"
            env:
              - name: "INFLUXDB_HOST"
                value: "monitoring-influxdb"
              - name: "DEBUG"
                value: "true"
            volumeMounts:
              - name: ssl-certs
                mountPath: /etc/ssl/certs
                readOnly: true
        volumes:
          - name: ssl-certs
            source:
              hostDir:
                path: /etc/ssl/certs
    labels:
      name: "heapster"
      uses: "monitoring-influxdb"
labels:
  name: "heapster"

Once we have the configuration files on our Kubernetes master we can deploy them into the cluster like this…

kubectl create -f grafana-service.yaml
kubectl create -f heapster-service.yaml
kubectl create -f influxdb-service.yaml
kubectl create -f heapster-controller.yaml 
kubectl create -f influxdb-grafana-controller.yaml

Once the pods deploy, we should be able to browse to the Grafana front end at this URL…

https://kubmasta:8443/api/v1beta1/proxy/services/monitoring-grafana/

Note: In my case, my Kubernetes master is called ‘kubmasta’ and the API service is listening on port 8443.  Since you’re using HTTPS we’re also going to get cert warnings on your initial connection to the API server.

Notice that in this case we aren’t connecting directly to the pod providing the service.  Rather, we’re connecting through the API server to the backend service.  This seems to work well but I’m still not 100% sure on the mechanics behind how this works but I’m working to get more info on it.  Either way, the Grafana frontend and default Kubernetes dashboard is pretty awesome…

As I mentioned above, you’re going to get cert errors unless you have the certificates sorted out.  An alternative would be to tell Grafana to use HTTP rather than HTTPS.  This cold be done by changing the Grafana replication controller config to something like this…

apiVersion: "v1beta1"
kind: "ReplicationController"
id: "monitoring-influx-grafana-controller"
desiredState:
  replicas: 1
  replicaSelector:
    name: "influxGrafana"
  podTemplate:
    labels:
      name: "influxGrafana"
    desiredState:
      manifest:
        version: "v1beta1"
        id: "monitoring-influx-grafana-controller"
        containers:
          - name: "influxdb"
            image: "kubernetes/heapster_influxdb:v0.3"
            ports:
              - containerPort: 8083
                hostPort: 8083
              - containerPort: 8086
                hostPort: 8086
          - name: "grafana"
            image: "jonlangemak/docker:heasptergrafana"
            ports:
              - containerPort: 80
            env:
              - name: "HTTP_USER"
                value: "admin"
              - name: "HTTP_PASS"
                value: "**None**"
              - name: "INFLUXDB_PROTO"
                value: "http"
              - name: "INFLUXDB_HOST"
                value: "localhost"
              - name: "INFLUXDB_PORT"
                value: ""
              - name: "KUBERNETES_API_PORT"
                value: "8080"
labels:
  name: "influxGrafana"

With this configuration, we could access the Grafana frontend on HTTP by accessing this URL…

http://kubmasta:8080/api/v1beta1/proxy/services/monitoring-grafana/

Either way, pretty cool stuff.  I hope you’re starting to see the power of this model.  Deploying an entire platform like Heapster was done with 5 command on the Kubernetes master.