Automation

You are currently browsing the archive for the Automation category.

Some of you will recall that I had previously written a set of SaltStack states to provision a bare metal Kubernetes cluster.  The idea was that you could use it to quickly deploy (and redeploy) a Kubernetes lab so you could get more familiar with the project and do some lab work on a real cluster.  Kubernetes is a fast moving project and I think you’ll find that those states likely no longer work with all of the changes that have been introduced into Kubernetes.  As I looked to refresh the posts I found that I was now much more comfortable with Ansible than I was with SaltStack so this time around I decided to write the automation using Ansible (I did also update the SaltStack version but I’ll be focusing on Ansible going forward).

However – before I could automate the configuration I had to remind myself how to do the install manually. To do this, I leaned heavily on Kelsey Hightower’s ‘Kubernetes the hard way‘ instructions.  These are a great resource and if you haven’t installed a cluster manually before I suggest you do that before attempting to automate an install.  You’ll find that the Ansible role I wrote VERY closely mimics Kelsey’s directions so if you’re looking to understand what the role does I’d suggest reading through Kelsey’s guide.  A big thank you to him for publishing it!

So let’s get started…

This is what my small lab looks like.  A couple of brief notes on the base setup…

  • All hosts are running a fresh install of Ubuntu 16.04.  The only options selected for package installation were for the OpenSSH server so we could access the servers via SSH
  • The servers all have static IPs as shown in the diagram and a default gateway as listed on the L3 gateway
  • All servers reference a local DNS server 10.20.30.13 (not pictured) and are resolvable in the local ‘interubernet.local’ domain (ex: ubuntu-1.interubernet.local).  They are also reachable via short name since the configuration also specifies a search domain (interubernet.local).
  • All of these servers can reach the internet through the L3 gateway and use the aforementioned DNS server to resolve public names.  This is important since the nodes will download binaries from the internet during cluster configuration.
  • In my example – I’m using 5 hosts.  You don’t need 5 but I think you’d want at least 3 so you could have 1 master and 2 minions but the role is configurable so you can have as many as you want
  • I’ll be using the first host (ubuntu-1) as both the Kubernetes master as well as the Ansible controller.  The remaining hosts will be Ansible clients and Kubernetes minions
  • The servers all have a user called ‘user’ which I’ll be using throughout the configuration

With that out of the way let’s get started.  The first thing we want to do is to get Ansible up and running.  To do that, we’ll start on the Ansible controller (ubuntu-1) by getting SSH prepped for Ansible.  The first thing we’ll do is generate an SSH key for our user (remember this is a new box, you might already have a key)…

To do this, we use the ‘ssh-keygen’ command to create a key for the user. In my case, I just hit enter to accept the defaults and to not set a password on the key (remember – this is a lab). Next we need to copy the public key to all of the servers that the Ansible controller needs to talk to. To perform the copy we’ll use the ‘ssh-copy-id’ command to move the key to all of the hosts…

Above I copied the key over for the user ‘user’ on the server ubuntu-5. You’ll need to do this for all 5 servers (including the master).  Now that’s in place make sure that the keys are working by trying to SSH directly into the servers…

While you’re in each server make sure that Python is installed on each host. Besides the above SSH setup – having Python installed is the only other requirement for the hosts to be able to communicate and work with the Ansible controller…

In my case Python wasn’t installed (these were really stripped down OS installs so that makes sense) but there’s a good chance your servers will already have Python. Once all of the clients are tested we can move on to install Ansible on the controller node. To do this we’ll use the following commands…

I wont bother showing the output since these are all pretty standard operations. Note that in addition to installing Ansible we also are installing Python PIP. Some of the Jinja templating I do with the playbook requires the Python netaddr library.  After you install Ansible and PIP take care of installing the netaddr package to get that out of the way…

Now we need to tell Ansible what hosts we’re working with. This is done by defining hosts in the ‘/etc/ansible/hosts’ file. The Kubernetes role I wrote expects two host groups. Once called ‘masters’ and once call ‘minions’. When you edit the host file for the first time there will likely be a series of comments with examples. To clean things up I like to delete all of the example comments and just add the two required groups. In the end my Ansible hosts file looks like this…

You’ll note that the ‘masters’ group is plural but at this time the role only supports defining a single master.

Now that we told Ansible what hosts it should talk to we can verify that Ansible can talk to them. To do that, run this command…

You should see a ‘pong’ result from each host indicating that it worked. Pretty easy right? Now we need to install the role. To do this we’ll create a new role directory called ‘kubernetes’ and then clone my repository into it like this…

Make sure you put the ‘.’ at the end of the git command otherwise git will create a new folder in the kubernetes directory to put all of the files in.  Once you’ve download the repository you need to update the variables that Ansible will use for the Kubernetes installation. To do that, you’ll need to edit roles variable file which should now be located at ‘/etc/ansible/roles/kubernetes/vars/main.yaml’. Let’s take a look at that file…

I’ve done my best to make ‘useful’ comments in here but there’s a lot more that needs to be explained (and will be in a future post) but for now you need to definitely pay attention to the following items…

  • The host_roles list needs to be updated to reflect your actual hosts.  You can add more or have less but the type of the host you define in this list needs to match what group its a member of in your Ansible hosts file.  That is, a minion type in the var file needs to be in the minion group in the Ansible host file.
  • Under cluster_info you need to make sure you pick two network that don’t overlap with your existing network.
    • For service_network_cidr pick an unused /24.  This wont ever get routed on your network but it should be unique.
    • For cluster_node_cidr pick a large network that you aren’t using (a /16 is ideal).  Kubernetes will allocate a /24 for each host out of this network to use for pods.  You WILL need to route this traffic on your L3 gateway but we’ll walk through how to do that once we get the cluster up and online.

Once the vars file is updated the last thing we need to do is tell Ansible what to do with the role. To do this, we’ll build a simple playbook that looks like this…

The playbook says “Run the role kubernetes on hosts in the masters and the minions group”.  Save the playbook as a YAML file somewhere on the system (in my case I just saved it in ~ as k8s_install.yaml). Now all we need to do is run it! To do that run this command…

Note the addition of the ‘–ask-become-pass’ parameter. When you run this command, Ansible will ask you for the sudo password to use on the hosts. Many of the tasks in the role require sudo access so this is required. An alternative to having to pass this parameter is to edit the sudoers file on each host and allow the user ‘user’ to perform passwordless sudo. However – using the parameter is just easier to get you up and running quickly.

Once you run the command Ansible will start doing it’s thing. The output is rather verbose and there will be lots of it…

If you encounter any failures using the role please contact me (or better yet open an issue in the repo on GitHub).  Once Ansible finishes running we should be able to check the status of the cluster almost immediately…

The component status should return a status of ‘Healthy’ for each component and the nodes should all move to a ‘Ready’ state.  The nodes will take a minute or two in order to transition from ‘NotReady’ to ‘Ready’ state so be patient. Once it’s up we need to work on setting up our network routing. As I mentioned above we need to route the network Kubernetes used for the pod networks to the appropriate hosts. To find which hosts got which network we can use this command which is pointed out in the ‘Kubernetes the hard way’ documentation…

Slick – ok. So now it’s up to us to make sure that each of those /24’s gets routed to each of those hosts. On my gateway, I want the routing to look like this…

Make sure you add the routes on your L3 gateway before you move on.  Once routing is in place we can deploy our first pods.  The first pod we’re going to deploy is kube-dns which is used by the cluster for name resolution.  The Ansible role already took care of placing the pod definition files for kube-dns on the controller for you, you just need to tell the cluster to run them…

As you can see there is both a service and a pod definition you need to install by passing the YAML file to the kubectl command with the ‘-f’ parameter. Once you’ve done that we can check to see the status of both the service and the pod…

If all went well you should see that each pod is running all three containers (denoted by the 3/3) and that the service is present.  At this point you’ve got yourself your very own Kubernetes cluster.  In the next post we’ll walk through deploying an example pod and step through how Kubernetes does networking.  Stay tuned!

Update: Ivan Pepelnjak reached out to me after this was published and suggested that it would make more sense to move the functions inside the defined class.  Doing this removes the functions from the global namespace and avoids the possibility of function names overlapping.  I think that’s a great idea so I’ve updated the examples below to align with this thinking.  Thanks Ivan!

I’ve been playing around with Ansible quite a bit lately.  One of the issues I’ve started to run into is that Ansible can’t always do what you want by default.  Put more plainly, Ansible can’t always easily do what you want.    Many times I found myself writing tasks to manipulate variables and thinking to myself, “Man – if I could just run some Python code on this it would be way easier”.  As luck would have it, you can!  Ansible supports a whole slew of plugins but the type I want to talk about today are called filter plugins.  Filter plugins, in my opinion, are one of the easiest ways to manipulate your variables with native Python.  And once you know how to do it you’ll see that it opens up a whole realm of possibilities in your playbooks.  Some of the most popular filters that exist today were once custom filters that someone wrote and then contributed to Ansible.  The IP address (ipaddr) filter set is a great example of filters that can be used to manipulate IP related information.

When I first looked into writing a custom filter I found the Ansible documentation not very helpful.  It essentially points you to their GitHub repo where you can look at the current filters.  Since I learn best by example, let’s write a quick filter so you can see how easy it is to do…

Nothing to it right?  This file defines a single filter called ‘a_filter’.  When called it receives the variable being passed into it (the variable to the left of the pipe (|)), appends the string ‘ CRAZY NEW FILTER’ to it, and then returns the new variable.  Now the trick is where to put this file.  For now, let’s create a folder called ‘filter_plugins’ in the same location as your playbook.  So in my case, the file structure would look like this…

/home/user/my_playbook.yml
/home/user/filter_plugins/my_filters.py

So let’s go ahead and create a quick little test playbook too…

This is a pretty simply playbook.  All it does is use the debug module to output a variable.  However, note that instead of just outputting the word ‘test’, we’re wrapping it in double curly braces like we do for any normal Ansible variable and we’re also piping it to ‘a_filter’.   The piping piece is what’s typically used to pass the variable to any of the predefined, or built-in, filters.  In this case, we’re piping the variable to our own custom filter.

This playbook assumes that you’ve told Ansible to use a local connection when talking to the locahost.  To do this, you need to set the ‘ansible_connection’ variable for the localhost to ‘local’ in your Ansible host file…

Once this is set, and you have both the playbook and the filter files in place, we can try running the playbook…

As you can see, the filter worked as expected.  The variable ‘test’ was passed to our custom filter where it was then modified and returned.  Pretty slick huh?  This is a uber simple example but it shows just how easy it is to inject custom Python functionality into your playbooks.  You’ll notice that in this example, there was only one variable passed to our function.  In this case, it was the variable to the left of the pipe.  In the case of filters, that will always be your first variable however, you can always add more.  For instance, let’s add a new filter to our function like this…

There are a couple of interesting things to point out in our new my_filters.py file.  First off – you’ll notice that we added another Python function called ‘b_filter’.  Its worthwhile to point out that your filter names don’t need to match your function names.  Down in the filters function at the bottom you’ll notice that we map the filter name ‘another_filter’ to the Python function ‘b_filter’.  You’ll also notice that the function b_filter takes 3 arguments.  The first will be the variable to the left of the pipe and the remaining need to be passed directly to the function as we’d normally pass variables.  For example…

Here you can see that we pass the second and third variables to the filter just like we’d normally pass variables to a function.  And while these examples only show doing this with strings, you can pass many other Python data types such as lists and dicts as well.

Lastly – I want to talk about the location of the filters.  By default, Ansible will look in the directory your playbook is located for a folder called ‘filter_plugins’ and load any filters it finds in that folder.  While this works, I don’t particularly care for this as I find it confusing for when you’re moving around playbooks.  I prefer to tell Ansible to look elsewhere for the filters.  To do this, we can edit the /etc/ansible/ansible.cfg file and uncomment and update the ‘filter_plugins’ parameter with your new location.

As you can see, filter plugins make it ridiculously easy to get your variables into native Python for manipulation.  Keep in mind that there are LOTS of default filter plugins so before you go crazy search the documentation to see if what you want already exists.

Ansible Roles and Variables

While automation is great, we have to be careful not to recreate past problems. What I mean is that playbooks should be written in a generic fashion that can be applied to more than one host. If we’re writing playbooks that only work on one single host, we aren’t much further ahead than we were before.

Two of the key components of making playbooks reusable are Ansible variables and roles.  Let’s try and define each of them individually and while showing some examples along the way.

Roles
Roles allow you to call a set of variables, tasks, and handlers by simply specifying a defined role.  Roles require the use of a defined file structure in order to work.  Per the Ansible documentation, that structure looks like this…

image 
Roles are really just a way to split up your playbook into smaller reusable parts.  For instance, let’s consider we added another host to our lab…

image 
Now look at this playbook…

Its a modification of the playbook we used in our first post.  It includes the original play, but also a second play that targets only the second Ansible client.  The second play copies an amazing page off the of the well known website www.purple.com and makes it the index page for our 2nd client.  Since both clients need the web server configuration, we can simplify this playbook by making the web server configuration its own role.  To do this, we create a file structure as show above for this role…

Note: Theres an easier way to make this folder structure using Ansible galaxy but that’s out of scope for this post.

While we’re creating all the possible folders required in a role, we’ll only be focusing on the tasks and handlers folders at this point.  The first thing we want to do is create a ‘main.yml’ file in the tasks folder that looks like this…

Notice that this is just the task that was defined in our initial playbook.  Next create another ‘main.yml’ file under the handler directory that looks like this…

Same thing here.  All we did was take the handler information from the original playbook definition and move it into the handler directory.  The last piece is to modify the playbook to call the role rather than the task.  Modify your playbook to look like this…

Petty simple huh?  Now if we want to run the tasks associated with building a web server we can just call that role.  If we put ansibleclient1 back to it’s original default state, running the playbook should look like this…

image
Very similar output to what we saw the last time we ran a playbook.  A quick test of browsing to each server should show the results we’re looking for…

image 
As you can see, roles can come in pretty handy for generic tasks.  Keep in mind that we’ve only scratched the surface of the role construct here to get you up and running.  We’ll be talking more about other aspects of roles in later posts. 

Variables
One of the ways to make playbooks more generic is to use Ansible variables. Variables can be defined in multiple locations.  Let’s walk through each one of them and a quick example…

In the inventory file
Variables can be assigned right along with the host definition in your inventory file.  For instance, look at this example host file…

Note: In many places you’ll see variables assigned in the above manner referred to as ‘host_vars’ and ‘group_vars’.

In the above example, we set variables at both a group and an individual host level.  Here we define that the variable ‘webserver_package’ is httpd for each host.  Then, at the group level, we set the variable ‘http_server_port’ to 80.  We can then change our task definition (main.yml) to look like this…

And our handler definition to look like this…

In both cases, we’re making use of the variables defined as part of the role definition.  While we can certainly define variables as part of the inventory file itself, best practice calls for these variables to be stored in separate files.  Much like many other components of Ansible, this relies on a default folder structure. Assuming you are using the default inventory file (/etc/ansible/hosts) you need to create two additional folders…

For host variables – /etc/ansible/host_vars
For group variables – /etc/ansible/group_vars

Inside of these folders, you place YAML files named after the host or group you wish to define variables for.  For instance, my folder structure might look like this…

image
And the files definitions themselves would look like this…

In the playbook itself
These same variables can also be defined directly within the plays.  If we remove the variables from the host inventory file, we can add them directly into one of the plays in the playbook definition.  Keep in mind, this is in the playbook definition, not in the role definition, we’ll cover that shortly.  So the playbook would look like this…

Notice that we now have a ‘vars:’ section defined at the top of the playbook. 

In the role definition
Variables can also be defined as part of a role.  There are two places this can be done with variables defined in either the ‘vars’ or the ‘default’ directory of the role.  Much in the same way we defined other role functions, we can also define a ‘main.yml’ file in both of these directories that define variable to use as part of the role.  For instance let’s assume that we have the following defined as ‘/etc/ansible/roles/webserver/defaults/main.yml’

and we have the following defined as ‘/etc/ansible/roles/webserver/vars/main.yml’

Above we define the same variables as previous examples, but in the case of ‘webserver_package’ we’ve defined them in both files.  When the playbook is run, we’ll find that the value from ‘vars’ directory is the one that’s used.  So why is that?  The ‘default’ directory stores values that are role default.  That is, they are sane default settings which can be overridden in certain circumstances.  Defaults can be overridden using values from the ‘vars’ directory, group vars, or host vars.  At the time of execution the variables are merged and precedence rules are applied to figure out the winning values for all of the variables.  The default precedence model looks like this…

-Variables defined in role ‘defaults’
-Variables defined as group_vars
-Variables defined as host_vars
-Variables defined in plays
-Variables defined in role ‘vars’
-Variables defined at runtime with –e switch

All of those should be familiar to you now except for the runtime option which we’ll cover last.  Priority increases as you move down the list.  That is the, if you want to override absolutely any other defined variable, you should do it at  runtime with the ‘–e’ flag. 

At runtime
The last, and most powerful, place to define variables is when you execute a playbook.  To prove this, lets go back to our role example and change the variables we defined.  We’ll remove the ‘main.yml’ file from the ‘default’ folder, and change the ‘main.yml’ in the ‘vars’ folder to look like this…

If we run the playbook now, it will fail since these values are the only ones defined (and because this a poorly written example playbook that doesn’t use all the variables in all the right places).  If we want it to work as intended, we can rerun the playbook by specifying variables at runtime.  To do that, we run this command…

Note: Its important to put the single quotes around the variables when passing more than one variable in this manner.

When we do this, the variables we provide at runtime will have precedence over the ones defined in the ‘vars’ directory and we’ll get a successful playbook execution resulting in the web servers working as expected…

image

The more I play with Ansible the more I like it. The role construct coupled with scoped variables is pretty powerful.  I hope you found this look at roles and variables useful, comments always welcome!

Tags:

« Older entries