Dynamic Kubernetes installation/configuration with SaltStack

I’ve been playing more with SaltStack recently and I realized that my first attempt at using Salt to provision my cluster was a little shortsighted.  The problem was, it only worked for my exact lab configuration.  After playing with Salt some more, I realized that the Salt configuration could be MUCH more dynamic than what I had initially deployed.  That being said, I developed a set of Salt states that I believe can be consumed by anyone wanting to deploy a Kubernetes lab on bare metal.  To do this, I used a few more of the features that SaltStack has to offer.  Namely, pillars and the built-in Jinja templating language.

My goal was to let anyone with some Salt experience be able to quickly deploy a fully working Kubernetes cluster.  That being said, the Salt configuration can be tuned to your specific environment.  Have 3 servers you want to try Kubernetes on?  Have 10?  All you need to do is have some servers that meet the following prerequisites and tune the Salt config to your environment.

Environment Prerequisites
-You need at least 2 servers, one for the master and one for the minion (might work with 1 but I haven’t tried it)
-All servers used in the config must be resolvable in local DNS
-All servers used as minions need to have a subnet routed to them for the docker0 bridge IP space (I use a /27 but you decide based on your own requirements)
-You have Salt installed and configured on the servers you wish to deploy against.  The Salt master needs to be configured to host files and pillars.  You can see the pillar configuration I did in this post and the base Salt config I did in this post.
-You have Git and Docker installed on the server you intend to build the Kubernetes binaries on (yum -y install docker git)

That’s all you need.  To prove that this works, I built another little lab to try and deploy this against.  It looks like this…

In this example, I have 3 servers.  K8stest1 will be used as the Kubernetes master and the remaining two will act as Kubernetes minions.  From a Salt perspective, k8stest1 will be the master and all the servers (including k8stest1) will be minions.  Notice that all the servers live on the 192.168.127.0/24 segment and I’ve routed space from 172.10.10.0 /24 to each Kubernetes minion in /27 allocations.  Let’s verify that Salt is working as I expect by ‘pinging’ each minion from the master…

Cool – So Salt can talk to all the servers we’re using in this lab.  The next step is to download and build the Kubernetes code we’ll be using in this lab.  To do this, we’ll clone the Kubernetes GitHub repo and then build the binaries…

Note: Kubernetes is changing VERY quickly.  I’ve been doing all of my testing based off of the .13 branch and that’s what all of these scripts are built to use.  Once things settle down I’ll update these posts with the most recent stable code but at the time of this writing the stable version I found was .13.  Make sure you clone the right branch!

git clone -b release-0.13 https://github.com/GoogleCloudPlatform/kubernetes.git

Once the code is downloaded, build the binaries using these commands…

cd kubernetes/build
./run.sh hack/build-cross.sh

Again – this will take some time.  Go fill up the coffee cup and wait for the code to finish building.  (Hit Y to accept downloading the golang image before you walk away though!)

Once the binaries are built, we can clone down my Salt build scripts from my repository…

git clone https://github.com/jonlangemak/saltstackv2.git /srv/

This will put the code into the ‘/srv/’ directory and create it if it doesn’t exist.  Next up, we need to move the freshly built binaries into the salt state directory so that we can pull them from the client…

mkdir /srv/salt/kubebinaries
cd /root/kubernetes/_output/dockerized/bin/linux/amd64
cp * /srv/salt/kubebinaries

The last step before we build the cluster is to tell the Salt configuration what our environment looks like.  This is done by editing the ‘kube_data’ pillar and possibly changing the Salt top.sls file to make sure we match the correct hosts.  Let’s look at the ‘kube_data’ pillar first.  It’s located at ‘/srv/pillar/kube_data.sls’…

kube_master:
  ipaddress: 10.20.30.61
  portal_net: 10.100.0.0/16

kube_minions:
  kubminion1:
    ipaddress: 10.20.30.62
    docker0_bip: 10.10.10.1/24
  kubminion2:
    ipaddress: 10.20.30.63
    docker0_bip: 10.10.20.1/24
  kubminion3:
    ipaddress: 192.168.10.64
    docker0_bip: 10.10.30.1/24
  kubminion4:
    ipaddress: 192.168.10.65
    docker0_bip: 10.10.40.1/24

kube_pods:
  skydns:
    portalip: 10.100.0.10
    dnsname: kubdomain.local

So this is a sample pillar file that I used to build the config in my initial lab.  You’ll notice that I have 4 minions and 1 master defined.  To change this to look like our new environment, I would update the file to look like this…

kube_master:
  ipaddress: 192.168.127.100
  portal_net: 10.100.0.0/16

kube_minions:
  k8stest2:
    ipaddress: 192.168.127.101
    docker0_bip: 172.10.10.1/27
  k8stest3:
    ipaddress: 192.168.127.102
    docker0_bip: 172.10.10.33/27

kube_pods:
  skydns:
    portalip: 10.100.0.10
    dnsname: kubdomain.local

So not too hard right?  Let’s walk through what I changed.  I had said that I wanted to use k8stest1(192.168.127.100) as the master so I changed the relevant IP address under the ‘kube_master’ definition to match k8stest1.  I have 2 minions in this lab so I removed the other 2 definitions and updated the first 2 to match my new environment.  Notice that you define the minion name and then define it’s IP address and docker0 bridge as attributes.  If you had more minions you could define them in here as well.

Note: The minion names have to be exact and do NOT include the FQDN, just the host part of the name.

The last thing we have to look at is the ‘top.sls’ file for salt.  Let’s look at it to see what the default is…

# This file assumes that you use the words 'master' and 'minion' in your
# naming convention for your nodes.  If thats not the case, you'll need
# to update these so the proper state files match the proper servers 
base:
  '*':
    - baseinstall
  '*minion*':
    - minioninstall
  '*master*':
    - masterinstall
    - pods

Notice  that if you happen to have named your servers according to role, then you might not have to change anything.  If your host names happen to include ‘master’ and ‘minion’ in their name based on the role they’ll play in the Kubernetes cluster, then you’re already done.  If not, we’ll need to change this to match our lab.  Let’s update it to match our lab…

# This file assumes that you use the words 'master' and 'minion' in your
# naming convention for your nodes.  If thats not the case, you'll need
# to update these so the proper state files match the proper servers 
base:
  '*':
    - baseinstall
  'k8stest[2-3]*':
    - minioninstall
  'k8stest1*':
    - masterinstall
    - pods

Above you can see what used glob matching and specified the hostnames we were using for the master and minion roles.  Pretty straight forward, just make sure that you match the right server to the right role based on the defaults.

Now all we have to do is let Salt take care of building the Kubernetes cluster for us…

salt ‘*’ state.highstate

Once Salt has finished executing, we should be able to check and see what the Kubernetes cluster is doing…

Hard to see there (click to make it bigger) but Kubernetes is now busily working to deploy all of the pods.  The default build will provision SkyDNS, Heapster, and the fluentD/ElasticSearch Logging combo.  After all of the pods are deployed, we can access those cluster add-ons at the following URLs…

Heapster

http://<Kubernetes Master>:8080/api/v1beta1/proxy/services/monitoring-grafana/

FluentD/ElasticSearch

http://<Kubernetes Master>:8080/api/v1beta1/proxy/services/kibana-logging/

So there you have it.  Pretty easy to do right?  You should have a fully operational Kubernetes cluster at this point.  Would love to have some other people try this out and give me feedback on it.  Let me know what you think!