Python

You are currently browsing the archive for the Python category.

Over the last several years I’ve made a couple of efforts to become better at Python. As it stands now – I’d consider myself comfortable with Python but more in terms of scripting rather than actual Python development. What does Python Development mean? To me – at this point – it’s means writing more than a single Python file that does one thing. So why have my previous efforts failed? I think there have been a few reasons I never progressed much past scripting…

  • I didn’t have a good understanding of Python tooling and the best way to build a project. This meant that I spent a large chunk of time messing around with Python versions, packages, and not actually writing any code.  I was missing the fundamentals in terms of how to work in Python.
  • I didn’t have a real goal in mind.  Trying to learn something through ‘Hello World’ type examples doesn’t work for me.  I need to see real world examples of something I’d actually use in order to understand how it works.  I think this is likely why some of the higher level concepts in Python didn’t fully take hold on the first few attempts.
  • I got stuck in the ‘make it work’ mindset which led me to the copying code snippets kind of mentality.  Again – not a great learning tool for me.
  • As a follow up to the previous point – I also got stuck on the other end of the spectrum.  The ‘Import this and just use it without knowing how it works’ idea didn’t sit with me causing me to try and comprehend all of the code in other packages and modules (Im a detail person.  I need the details).  In some cases this is beneficial but I’m now also of the opinion that some packages you use can just be used without reading all of the code.  Trying to understand code written by Python experts instead of just using it was a great way for me to spend hours spinning.  This boils down to the ‘walk before run’ concept in my opinion.
  • I completely ignored testing.  These is no excuse for this.

So this time I’m going to try something different.  I’m going to start from scratch on a new project and see if I can use what I perceive to be (hoping people call me out on things that aren’t) proper Python development.  The first few posts will cover what I perceive to be Python fundamentals. Things you need to know about setting up Python even before you start writing any serious code.

Enough talk – let’s start…

Virtual Environments

One of the first things that really clicked with me about Python development was the use of Python virtual environments.  If you’ve ever struggled with having multiple versions of Python on your system, and figuring out which one you were using, or which version had which package, you need to look at virtual environments. Virtual environments create a sort of isolated sandbox for you to run your code in.  They can contain a specific version of Python and specific Python packages installed through a tool like PIP.  There’s lots of options and possible configurations but let’s just look at some of the basics of using virtual environments.  To create a virtual environment, we need the virtualenv Python package installed.  If you don’t have it already, you can easily install it through PIP with pip install virtualenv.  Once installed, you can create a virtual environment rather simply by using the following syntax…

In this case – we created a couple of base folders to store our project in and then created a virtual environment called my_venv. If we look inside of our project directory we’ll now see that we have a folder called my_venv

If we dig in a little deeper we see a bin folder that contains all of the Python binaries we need. Also notice that there’s a file called activate in the bin directory.  If we source this file, our terminal will activate this virtual environment.  A common way to do this is from the base project directory as shown below…

Notice how the command line is now prefaced with (my_venv) to indicate that we are running in a given virtual environment. Now if we run a command like pip list we should see a limited number of packages…

Note that we don’t see the package virtualenv which we know exists on our base system (since we’re using it right now). We don’t see any of the base system packages because we’re running in a sandbox. In order to use certain packages in this sandbox (virtual environment) we’d need to install them in the virtual environment. Another less common option is to allow the virtual environment to inherit the packages from the base system. This would be done by passing the flag --system-site-packages when you create the virtual environment. We can demonstrate that here by creating a second virtual environment…

Note: To deactivate a virtual environment simply type deactivate into the terminal.

Notice that we have a ton of packages that the virtual environment perceives to be installed since it’s inheriting them from the base system. So what about the actual Pyhton version we’re using in the virtual environment? If not specified when created, the virtual environment will use the Python version used by the base system. That is – the one specific in your PATH or PYTHONPATH environmental variable. In my case, that’s Python 2.7 which we can see just by starting the Python interpreter from the CLI outside of the virtual environment…

And if we once again activate our virtual environment we’ll see that the versions align perfectly…

To use a different version, we need to tell the virtualenv command which version to use. To do that we can pass the --python flag. The trick to this is passing the path to exact Python executable you wish to use. So let’s dig around a little bit. Im on a Mac so most of my Python versions were installed through HomeBrew. So first let’s see which executable I’m using when I type python outside of my virtual environment…

Whoa. There’s a lot of versions there. The interesting thing is that these are mostly all symlinks pointing to other places. For instance…

Ok so we can see that these are really pointing to HomeBrew install locations. So if we wanted to – we could tell the virtualenv command to point directly to these exact destinations. Or, since the symlinks are there and work, we can just reference the version we want to use so long as it’s resolvable through the PATH variable. For instance…

So as you can see – it just needs some way to find the Python executable you wish to use. In both cases, we ended up with the same version of Python (3.6) in the virtual environment. Nice! So now let’s talk a little bit about package management.

Package Management with PIP

Installing packages for Python is drop dead easy. You can use pip and simply pip install whatever and it works awesome. The problem is – how do you keep track of all of this? Better yet – how do you keep track of what version of what package you are using? If you’ve poked around Python projects for any length of time, you’ve likely seen a file called requirements.txt. This file keeps track of the Python packages and their versions that you need to make the code you downloaded run. Pretty easy right? But tracking these manually can be a sincere pain and something often forgotten. Luckily, PIP (not sure if this should be capitalized or not all the time (I’ll just keep flipping between caps and no caps)) has a feature that can create this for you called freeze

In the above output we installed a package in our virtual environment called pyyaml. We then used the traditional pip list command to show that it was installed. However – we also got info about the default Pyhton packages which we dont really care about since, well, they’re default. When we use the pip freeze command we get only installed packages and in a slightly different format (notice the used of ==). To store it in a file we can simply > it into a file as shown below…

Ok – so now we have two packages in our requirements.txt file. This is great and all, but what do we do with this? Using PIP we can install packages based on this file like so…

Notice here that we first deactivated the virtual environment we were in, then we activated another, showed that only the base packages were there, and then we used the requirements.txt file to install all of the packages we needed through PIP. Pretty slick right?

So now we know how to handle Python versions and packages inside of virtual environments. This is powerful by itself, but there’s even more ways we can optimize this. We’ll build on this in our next post. Stay tuned!

Update: Ivan Pepelnjak reached out to me after this was published and suggested that it would make more sense to move the functions inside the defined class.  Doing this removes the functions from the global namespace and avoids the possibility of function names overlapping.  I think that’s a great idea so I’ve updated the examples below to align with this thinking.  Thanks Ivan!

I’ve been playing around with Ansible quite a bit lately.  One of the issues I’ve started to run into is that Ansible can’t always do what you want by default.  Put more plainly, Ansible can’t always easily do what you want.    Many times I found myself writing tasks to manipulate variables and thinking to myself, “Man – if I could just run some Python code on this it would be way easier”.  As luck would have it, you can!  Ansible supports a whole slew of plugins but the type I want to talk about today are called filter plugins.  Filter plugins, in my opinion, are one of the easiest ways to manipulate your variables with native Python.  And once you know how to do it you’ll see that it opens up a whole realm of possibilities in your playbooks.  Some of the most popular filters that exist today were once custom filters that someone wrote and then contributed to Ansible.  The IP address (ipaddr) filter set is a great example of filters that can be used to manipulate IP related information.

When I first looked into writing a custom filter I found the Ansible documentation not very helpful.  It essentially points you to their GitHub repo where you can look at the current filters.  Since I learn best by example, let’s write a quick filter so you can see how easy it is to do…

Nothing to it right?  This file defines a single filter called ‘a_filter’.  When called it receives the variable being passed into it (the variable to the left of the pipe (|)), appends the string ‘ CRAZY NEW FILTER’ to it, and then returns the new variable.  Now the trick is where to put this file.  For now, let’s create a folder called ‘filter_plugins’ in the same location as your playbook.  So in my case, the file structure would look like this…

/home/user/my_playbook.yml
/home/user/filter_plugins/my_filters.py

So let’s go ahead and create a quick little test playbook too…

This is a pretty simply playbook.  All it does is use the debug module to output a variable.  However, note that instead of just outputting the word ‘test’, we’re wrapping it in double curly braces like we do for any normal Ansible variable and we’re also piping it to ‘a_filter’.   The piping piece is what’s typically used to pass the variable to any of the predefined, or built-in, filters.  In this case, we’re piping the variable to our own custom filter.

This playbook assumes that you’ve told Ansible to use a local connection when talking to the locahost.  To do this, you need to set the ‘ansible_connection’ variable for the localhost to ‘local’ in your Ansible host file…

Once this is set, and you have both the playbook and the filter files in place, we can try running the playbook…

As you can see, the filter worked as expected.  The variable ‘test’ was passed to our custom filter where it was then modified and returned.  Pretty slick huh?  This is a uber simple example but it shows just how easy it is to inject custom Python functionality into your playbooks.  You’ll notice that in this example, there was only one variable passed to our function.  In this case, it was the variable to the left of the pipe.  In the case of filters, that will always be your first variable however, you can always add more.  For instance, let’s add a new filter to our function like this…

There are a couple of interesting things to point out in our new my_filters.py file.  First off – you’ll notice that we added another Python function called ‘b_filter’.  Its worthwhile to point out that your filter names don’t need to match your function names.  Down in the filters function at the bottom you’ll notice that we map the filter name ‘another_filter’ to the Python function ‘b_filter’.  You’ll also notice that the function b_filter takes 3 arguments.  The first will be the variable to the left of the pipe and the remaining need to be passed directly to the function as we’d normally pass variables.  For example…

Here you can see that we pass the second and third variables to the filter just like we’d normally pass variables to a function.  And while these examples only show doing this with strings, you can pass many other Python data types such as lists and dicts as well.

Lastly – I want to talk about the location of the filters.  By default, Ansible will look in the directory your playbook is located for a folder called ‘filter_plugins’ and load any filters it finds in that folder.  While this works, I don’t particularly care for this as I find it confusing for when you’re moving around playbooks.  I prefer to tell Ansible to look elsewhere for the filters.  To do this, we can edit the /etc/ansible/ansible.cfg file and uncomment and update the ‘filter_plugins’ parameter with your new location.

As you can see, filter plugins make it ridiculously easy to get your variables into native Python for manipulation.  Keep in mind that there are LOTS of default filter plugins so before you go crazy search the documentation to see if what you want already exists.

One of the fist things you’ll most likely encounter with Python are the datatypes lists and dicts.  While they initially seem quite simple, things can get awfully complex, awfully fast when you start intermingling the two datatypes.  So we’ll start with the basics, then dive into some more complex examples. 

Lists
Lists are defined as ‘a collection of different pieces of information as a sequence under a single variable name’.  So that’s a fancy way of saying it’s just a list.  In Python, lists are defined by using the ‘[]’ brackets.  So for example…

Items in lists can be accessed by index.  For example…

We can also iterate through the list with a simple loop…

Lists can be added to by using the list attribute ‘append’.  For instance…

Additionally, lists can contain items of multiple different types…

Dicts
Dicts are more of a ‘key/value’ kind of arrangement.  Like lists, they are mutable, and can be initialized with data or empty.  The major difference in definition is that dicts use ‘{}’ whereas lists used ‘[]’.  For example…

You might have noticed that we’re just printing the values.  If you need to print the key, you can do so as well but that doesn’t need to be returned from the dict since you’re using it to find the value.  So you can either just print it, or you can iterate through the items in the dict using a for loop and return both the key and the value…

Getting more complicated
Note that above the dict is holding different kinds of value.  ‘Langemak’ was type string and ‘30’ was type integer.  This means that dicts can hold a variety of different datatypes including lists!  Let’s take a quick example so you can see what I mean…

Conversely, lists can hold a variety of datatypes such as dicts…

So you can see that I can store multiple types of data in a dictionary.  When we print the data out our for loop needs to check and see what type of data the value matching the key holds.  For the values that are lists we execute an additional loop to run through the entire list.  So this is pretty easy to understand, but take a look at this example that I came across when I experimenting with a network switch API…

image
Interestingly enough, Python sees the ‘[‘ and interprets that this is a list.  Lists are delineated by commas and defined within brackets.  So if we look at this, we can see that what we really have is a one list, with one item in it.  What’s more interesting is that the lists single item is a dict.  We can see this by using the following code…

We can see that Python sees the lists one object as a dict.  What’s more interesting is that what we really have is a bunch of nested dicts…

image So in this example, the first dict has a key of ‘vrfs’ and a value of ‘test’ which happens to be another dict.  The dict ‘test’ has 4 key/value pairs with keys ‘asn’, ‘peers’, ‘routerId’, and ‘vrf’.  Then from there each peer value is also a dict which contain more key/value pairs describing the given peer.  We can get an idea of how you access each of the dicts by looking at this example which returns how many keys are in each nested dict…

So as you can see, you lists and dicts in Python can be pretty flexible.  Next up, more Python!

« Older entries