Salt – The basics

In my last post, I showed you how I automated my Kubernetes lab build out by using Salt.  This took the build time and cut it by more than 70% (Im guessing here but you get the point).  In addition, I’ve been making all of my changes for the cluster in Salt rather than applying them directly to the host.  Not only does this give me better documentation, it allows me to apply changes across multiple nodes very quickly.  You might be wondering why I chose Salt since I’ve blogged about Chef in the past.  The answer isn’t cut and dry, but Salt just made sense to me.  On top of that, there is VERY good documentation out there about all of the state and state functions so it’s pretty easily consumable.    As I walk through the process I used to create the lab build scripts, I hope you’ll start to catch onto some of the reasons that made me decide to learn Salt.

Let’s start by taking a look at me GitHub repo…

While there’s a lot here, the pieces we really want to talk about are the files that end in ‘sls’.  These are what Salt calls ‘State’ files.  To quote their own page, Salt defines state files as…

“The core of the Salt State system is the SLS, or SaLt State file. The SLS is a representation of the state in which a system should be in, and is set up to contain this data in a simple format. This is often called configuration management.”

So that being said, we can assume that each of these files represents a ‘state’ of configuration that we can apply to end hosts.  In my case, the master (kubbuild) applies these states to the minions (kubmasta and kubminion[1-4]) to give me the desired end result of a working Kubernetes lab.  State files can be applied to hosts by issuing the following command on the master…

salt '*' state.sls <state file to be applied>

This will apply the state you specify to all of the minions.  If I want to apply a state file to a single minion, I can do so by removing the wild card and specifying an exact minion name or a glob for matching.  Here are some examples…

!Applies the state baseinstall to kubminion1
salt kubminion1.interubernet.local state.sls baseinstall

!Applies the state baseinstall to any node that has 'minion' in the name
salt *minion* state.sls baseinstall

!Applies the state baseinstall and minion1 to any node with a name matching 'minion1'
salt *minion1* state.sls baseinstall,minion1

So while this makes, good sense, it’s not exactly what I did in the last post to apply state to me minions.  I ran the command…

salt '*' state.highstate

This causes the nodes to examine the file ‘top.sls’ to determine what states should be applied to what minions.  Let’s look at my top.sls file to see what we’re talking about…

base:
  '*':
    - baseinstall
  '*masta*':
    - masterinstall
  'host:kubminion1':
    - match: grain
    - minion1
  'host:kubminion2':
    - match: grain
    - minion2
  'host:kubminion3':
    - match: grain
    - minion3
  'host:kubminion4':
    - match: grain
    - minion4

The goal of the top file is to apply state ,or a series of states, to a series of minions based on matching criteria.  We can see that the 2nd and 3rd line tell Salt that all (*) hosts should have the state ‘baseinstall’ applied to them.

Note: You’ll notice that you dont need to add the file extension (.sls) when referencing state files in Salt.  Keep this in mind when writing your state files.

The 4th and 5th line do a glob match for hosts with ‘masta’ in the name and apply the ‘masterinstall’ state to them.  The rest of the config uses a different type of matching which is referred to as ‘grain matching’.  Salt uses the concept of ‘grains’ to define specific attributes of a specific minion.  By default, there are a preset number of predefined grains that you can query against.  To see what Grains are available, we can use the following command to query them from a particular minion…

salt *masta* grains.items

This output will show us a list of grains on the kubmasta server.  The output looks like this…

This is just a section of the output, there’s ~50 predefined grains that you can query against.  So in our case, we applied state in the top file based on the grain called ‘host’.  We need to tell the top file that we want to match on a grain and then tell it what states to run against those matches.  Pretty slick huh?  You could see how powerful this would be if you needed to apply a patch against all ‘CentOS 7’ hosts or something along those lines.

So now that we know how the state files are applied, let’s look at the actual state files.  Let’s take one of the minion configs to use as an example and walk through a couple of the sections…

#Pull down docker config file 
/etc/sysconfig/docker:
  file:
    - managed
    - source: salt://minion1/docker
    - user: root
    - group: root
    - mode: 755

In this example, we’re pulling down the docker configuration file from the master to the host.  Recall in our last post we talked about the master ‘file_root’ that we defined to be ‘/srv/salt/’.  This tells the minion to download the file ‘/srv/salt/minion1/docker’ and place it on the minion as ‘/etc/sysconfig/docker’.  We also set some attributes of the file such as owner, group, and permissions.

Before we move on, let’s talk about the layout of this state file as that was something that was initially confusing to me…

#Pull down docker config file          <-- Comment
/etc/sysconfig/docker:                 <-- ID Declaration (I call it a label)
  file:                                <-- State Declaration
    - managed                          <-- Function Declaration
    - source: salt://minion1/docker    <-- Function Argument
    - user: root                       <-- Function Argument
    - group: root                      <-- Function Argument
    - mode: 755                        <-- Function Argument

The first thing we need to pay attention to is the ID Declaration.  In this case, I’m defining the file name I want to use as the target.  This is a little confusing because the ID in this case not only defines the identification for this state, but it’s also the target destination of the state, AKA where I want to put the file.  In other cases we’ll see later, the ID is nothing more than a unique identifier and plays not part in the actual configuration.

Note: The ID can NOT be duplicated either in a single state file, or a set of state files executed on the same minion.

Once we define an ID, we need to tell it what state we want to use.  In this case, we’re manipulating a file so we want to use the ‘file’ state.

Note: SaltStack has some great doco out there and this doc in particular is awesome since it lists all of the states inherently built into Salt –> http://docs.saltstack.com/en/latest/ref/states/all/

It’s important to note here that lines 3 and 4 above could be made into one line that read ‘file.managed:’.  I didn’t know that at first and that syntax makes a lot more sense to me so I think I’ll use that going forward.

So now that we know what state we’re using, we can use one of that states functions.  In this case, we use the managed function which allows you to download files from the master and place them on the minions.  The rest of the configuration consists of function arguments on what file we want to place and the associated ownership.

Now that we know the basic outline, let’s checkout some of the other states that the minion1.sls file uses and see what else Salt can do…

#Install docker, enable it, and run it
docker:
pkg:
- installed
service:
- running
- enable: true

Here we have a state that does two things.  Here the ID is ‘docker’ and we’re applying both the pkg state to install docker as well as the service state to enable and start the process.

#Make the kubernetes binary directory
/opt/kubernetes:
  file.directory:
    - user: root
    - group: root
    - dir_mode: 755
    - file_mode: 755

Here’s an example of how to make a directory on the minion.  I hope by this point, the state configuration layout is making more sense to you.  Let’s look at a couple other states from the masterinstall.sls to see a couple more things Salt can do…

Create a symlink

#Make symlink for etcdctl
/usr/local/bin/etcdctl:
  file.symlink:
    - target: /opt/etcd-v2.0.5-linux-amd64/etcdctl

Running a command on the minion

grafana-svc:
  cmd.run:
    - name: kubectl create -f /root/heapster/grafana-svc.yaml

Downloading and un-tarring a file on the minion

#Download etcd
etcd-install:
  archive.extracted:
    - name: /opt/
    - source: https://github.com/coreos/etcd/releases/download/v2.0.5/etcd-v2.0.5-linux-amd64.tar.gz
    - source_hash: md5=4d8ccff28c383980b52397a7664b3342
    - archive_format: tar
    - user: root
    - group: root
    - if_missing: /opt/etcd-v2.0.5-linux-amd64/

So as you can see, Salt is pretty flexible and this isn’t even scratching the surface of the power of salt.  The next item up for investigation are what Salt calls Pillars which look pretty awesome.

I know this was a brief intro, but I wanted to get some info out there after my last post so you could start digging into Salt some more.  I’m hoping to have some time later this week to dig more into Salt pillars and as I learn more I’ll do my best to blog about them.