Getting Started with Python Development

      No Comments on Getting Started with Python Development

Over the last several years I’ve made a couple of efforts to become better at Python. As it stands now – I’d consider myself comfortable with Python but more in terms of scripting rather than actual Python development. What does Python Development mean? To me – at this point – it’s means writing more than a single Python file that does one thing. So why have my previous efforts failed? I think there have been a few reasons I never progressed much past scripting…

  • I didn’t have a good understanding of Python tooling and the best way to build a project. This meant that I spent a large chunk of time messing around with Python versions, packages, and not actually writing any code.  I was missing the fundamentals in terms of how to work in Python.
  • I didn’t have a real goal in mind.  Trying to learn something through ‘Hello World’ type examples doesn’t work for me.  I need to see real world examples of something I’d actually use in order to understand how it works.  I think this is likely why some of the higher level concepts in Python didn’t fully take hold on the first few attempts.
  • I got stuck in the ‘make it work’ mindset which led me to the copying code snippets kind of mentality.  Again – not a great learning tool for me.
  • As a follow up to the previous point – I also got stuck on the other end of the spectrum.  The ‘Import this and just use it without knowing how it works’ idea didn’t sit with me causing me to try and comprehend all of the code in other packages and modules (Im a detail person.  I need the details).  In some cases this is beneficial but I’m now also of the opinion that some packages you use can just be used without reading all of the code.  Trying to understand code written by Python experts instead of just using it was a great way for me to spend hours spinning.  This boils down to the ‘walk before run’ concept in my opinion.
  • I completely ignored testing.  These is no excuse for this.

So this time I’m going to try something different.  I’m going to start from scratch on a new project and see if I can use what I perceive to be (hoping people call me out on things that aren’t) proper Python development.  The first few posts will cover what I perceive to be Python fundamentals. Things you need to know about setting up Python even before you start writing any serious code.

Enough talk – let’s start…

Virtual Environments

One of the first things that really clicked with me about Python development was the use of Python virtual environments.  If you’ve ever struggled with having multiple versions of Python on your system, and figuring out which one you were using, or which version had which package, you need to look at virtual environments. Virtual environments create a sort of isolated sandbox for you to run your code in.  They can contain a specific version of Python and specific Python packages installed through a tool like PIP.  There’s lots of options and possible configurations but let’s just look at some of the basics of using virtual environments.  To create a virtual environment, we need the virtualenv Python package installed.  If you don’t have it already, you can easily install it through PIP with pip install virtualenv.  Once installed, you can create a virtual environment rather simply by using the following syntax…

Jons-MacBook-Pro:~ jon$ mkdir python_projects
Jons-MacBook-Pro:~ jon$ mkdir python_projects/my_first_project
Jons-MacBook-Pro:~ jon$ cd python_projects/my_first_project
Jons-MacBook-Pro:my_first_project jon$ virtualenv my_venv
New python executable in /Users/jon/python_projects/my_first_project/my_venv/bin/python2.7
Also creating executable in /Users/jon/python_projects/my_first_project/my_venv/bin/python
Installing setuptools, pip, wheel...done.
Jons-MacBook-Pro:my_first_project jon$

In this case – we created a couple of base folders to store our project in and then created a virtual environment called my_venv. If we look inside of our project directory we’ll now see that we have a folder called my_venv

Jons-MacBook-Pro:my_first_project jon$ ls
my_venv
Jons-MacBook-Pro:my_first_project jon$ cd my_venv/
Jons-MacBook-Pro:my_venv jon$ ls
bin			include			lib			pip-selfcheck.json
Jons-MacBook-Pro:my_venv jon$ cd bin/
Jons-MacBook-Pro:bin jon$ ls
activate		easy_install		pip2.7			python2.7
activate.csh		easy_install-2.7	python			wheel
activate.fish		pip			python-config
activate_this.py	pip2			python2
Jons-MacBook-Pro:bin jon$

If we dig in a little deeper we see a bin folder that contains all of the Python binaries we need. Also notice that there’s a file called activate in the bin directory.  If we source this file, our terminal will activate this virtual environment.  A common way to do this is from the base project directory as shown below…

Jons-MacBook-Pro:bin jon$ cd ../..
Jons-MacBook-Pro:my_first_project jon$ source my_venv/bin/activate
(my_venv) Jons-MacBook-Pro:my_first_project jon$

Notice how the command line is now prefaced with (my_venv) to indicate that we are running in a given virtual environment. Now if we run a command like pip list we should see a limited number of packages…

(my_venv) Jons-MacBook-Pro:my_first_project jon$ pip list
pip (9.0.1)
setuptools (36.5.0)
wheel (0.30.0)
(my_venv) Jons-MacBook-Pro:my_first_project jon$

Note that we don’t see the package virtualenv which we know exists on our base system (since we’re using it right now). We don’t see any of the base system packages because we’re running in a sandbox. In order to use certain packages in this sandbox (virtual environment) we’d need to install them in the virtual environment. Another less common option is to allow the virtual environment to inherit the packages from the base system. This would be done by passing the flag --system-site-packages when you create the virtual environment. We can demonstrate that here by creating a second virtual environment…

Note: To deactivate a virtual environment simply type deactivate into the terminal.

Jons-MacBook-Pro:my_first_project jon$ virtualenv my_venv2 --system-site-packages
New python executable in /Users/jon/python_projects/my_first_project/my_venv2/bin/python2.7
Also creating executable in /Users/jon/python_projects/my_first_project/my_venv2/bin/python
Installing setuptools, pip, wheel...done.
Jons-MacBook-Pro:my_first_project jon$ ls
my_venv		my_venv2
Jons-MacBook-Pro:my_first_project jon$ source my_venv2/bin/activate
(my_venv2) Jons-MacBook-Pro:my_first_project jon$ pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
aniso8601 (1.3.0)
ansible (2.3.1.0)
appdirs (1.4.3)
asn1crypto (0.22.0)
Babel (2.3.4)
bcrypt (3.1.3)
<output removed for brevity>

Notice that we have a ton of packages that the virtual environment perceives to be installed since it’s inheriting them from the base system. So what about the actual Pyhton version we’re using in the virtual environment? If not specified when created, the virtual environment will use the Python version used by the base system. That is – the one specific in your PATH or PYTHONPATH environmental variable. In my case, that’s Python 2.7 which we can see just by starting the Python interpreter from the CLI outside of the virtual environment…

Jons-MacBook-Pro:my_first_project jon$ python
Python 2.7.13 (default, Apr  4 2017, 08:47:57) 
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()
Jons-MacBook-Pro:my_first_project jon$

And if we once again activate our virtual environment we’ll see that the versions align perfectly…

Jons-MacBook-Pro:my_first_project jon$ source my_venv/bin/activate
(my_venv) Jons-MacBook-Pro:my_first_project jon$ python
Python 2.7.13 (default, Apr  4 2017, 08:47:57) 
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()
(my_venv) Jons-MacBook-Pro:my_first_project jon$

To use a different version, we need to tell the virtualenv command which version to use. To do that we can pass the --python flag. The trick to this is passing the path to exact Python executable you wish to use. So let’s dig around a little bit. Im on a Mac so most of my Python versions were installed through HomeBrew. So first let’s see which executable I’m using when I type python outside of my virtual environment…

Jons-MacBook-Pro:my_first_project jon$ which python
/usr/local/bin/python
Jons-MacBook-Pro:my_first_project jon$ ls /usr/local/bin/ | grep python
python
python-config
python2
python2-config
python2.7
python2.7-config
python3
python3-config
python3.6
python3.6-config
python3.6m
python3.6m-config
pythonw
pythonw2
pythonw2.7
Jons-MacBook-Pro:my_first_project jon$

Whoa. There’s a lot of versions there. The interesting thing is that these are mostly all symlinks pointing to other places. For instance…

Jons-MacBook-Pro:my_first_project jon$ ls -al /usr/local/bin/ | grep python
<some out removed for brevity>
lrwxr-xr-x    1 jon   admin     34 Jul  6 00:20 python -> ../Cellar/python/2.7.13/bin/python
lrwxr-xr-x    1 jon   admin     41 Jul  6 00:20 python-config -> ../Cellar/python/2.7.13/bin/python-config
lrwxr-xr-x    1 jon   admin     35 Jul  6 00:20 python2 -> ../Cellar/python/2.7.13/bin/python2
lrwxr-xr-x    1 jon   admin     42 Jul  6 00:20 python2-config -> ../Cellar/python/2.7.13/bin/python2-config
lrwxr-xr-x    1 jon   admin     37 Jul  6 00:20 python2.7 -> ../Cellar/python/2.7.13/bin/python2.7
lrwxr-xr-x    1 jon   admin     44 Jul  6 00:20 python2.7-config -> ../Cellar/python/2.7.13/bin/python2.7-config
lrwxr-xr-x    1 jon   admin     35 Sep 25 21:16 python3 -> ../Cellar/python3/3.6.2/bin/python3
lrwxr-xr-x    1 jon   admin     42 Sep 25 21:16 python3-config -> ../Cellar/python3/3.6.2/bin/python3-config
lrwxr-xr-x    1 jon   admin     37 Sep 25 21:16 python3.6 -> ../Cellar/python3/3.6.2/bin/python3.6
lrwxr-xr-x    1 jon   admin     44 Sep 25 21:16 python3.6-config -> ../Cellar/python3/3.6.2/bin/python3.6-config
lrwxr-xr-x    1 jon   admin     38 Sep 25 21:16 python3.6m -> ../Cellar/python3/3.6.2/bin/python3.6m
lrwxr-xr-x    1 jon   admin     45 Sep 25 21:16 python3.6m-config -> ../Cellar/python3/3.6.2/bin/python3.6m-config
lrwxr-xr-x    1 jon   admin     35 Jul  6 00:20 pythonw -> ../Cellar/python/2.7.13/bin/pythonw
lrwxr-xr-x    1 jon   admin     36 Jul  6 00:20 pythonw2 -> ../Cellar/python/2.7.13/bin/pythonw2
lrwxr-xr-x    1 jon   admin     38 Jul  6 00:20 pythonw2.7 -> ../Cellar/python/2.7.13/bin/pythonw2.7
<some out removed for brevity>
Jons-MacBook-Pro:my_first_project jon$

Ok so we can see that these are really pointing to HomeBrew install locations. So if we wanted to – we could tell the virtualenv command to point directly to these exact destinations. Or, since the symlinks are there and work, we can just reference the version we want to use so long as it’s resolvable through the PATH variable. For instance…

Jons-MacBook-Pro:my_first_project jon$ virtualenv my_venv3 --python=python3.6
Running virtualenv with interpreter /usr/local/bin/python3.6
Using base prefix '/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/jon/python_projects/my_first_project/my_venv3/bin/python3.6
Also creating executable in /Users/jon/python_projects/my_first_project/my_venv3/bin/python
Installing setuptools, pip, wheel...done.
Jons-MacBook-Pro:my_first_project jon$ virtualenv my_venv4 --python=/usr/local/Cellar/python3/3.6.2/bin/python3.6
Running virtualenv with interpreter /usr/local/Cellar/python3/3.6.2/bin/python3.6
Using base prefix '/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/jon/python_projects/my_first_project/my_venv4/bin/python3.6
Also creating executable in /Users/jon/python_projects/my_first_project/my_venv4/bin/python
Installing setuptools, pip, wheel...done.
Jons-MacBook-Pro:my_first_project jon$ 
Jons-MacBook-Pro:my_first_project jon$ source my_venv3/bin/activate
(my_venv3) Jons-MacBook-Pro:my_first_project jon$ python --version
Python 3.6.2
(my_venv3) Jons-MacBook-Pro:my_first_project jon$ deactivate
Jons-MacBook-Pro:my_first_project jon$ source my_venv4/bin/activate
(my_venv4) Jons-MacBook-Pro:my_first_project jon$ python --version
Python 3.6.2
(my_venv4) Jons-MacBook-Pro:my_first_project jon$

So as you can see – it just needs some way to find the Python executable you wish to use. In both cases, we ended up with the same version of Python (3.6) in the virtual environment. Nice! So now let’s talk a little bit about package management.

Package Management with PIP

Installing packages for Python is drop dead easy. You can use pip and simply pip install whatever and it works awesome. The problem is – how do you keep track of all of this? Better yet – how do you keep track of what version of what package you are using? If you’ve poked around Python projects for any length of time, you’ve likely seen a file called requirements.txt. This file keeps track of the Python packages and their versions that you need to make the code you downloaded run. Pretty easy right? But tracking these manually can be a sincere pain and something often forgotten. Luckily, PIP (not sure if this should be capitalized or not all the time (I’ll just keep flipping between caps and no caps)) has a feature that can create this for you called freeze

(my_venv) Jons-MacBook-Pro:my_first_project jon$ pip install pyyaml
Collecting pyyaml
Installing collected packages: pyyaml
Successfully installed pyyaml-3.12
(my_venv) Jons-MacBook-Pro:my_first_project jon$ pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
pip (9.0.1)
PyYAML (3.12)
setuptools (36.5.0)
wheel (0.30.0)
(my_venv) Jons-MacBook-Pro:my_first_project jon$ pip freeze
PyYAML==3.12
(my_venv) Jons-MacBook-Pro:my_first_project jon$

In the above output we installed a package in our virtual environment called pyyaml. We then used the traditional pip list command to show that it was installed. However – we also got info about the default Pyhton packages which we dont really care about since, well, they’re default. When we use the pip freeze command we get only installed packages and in a slightly different format (notice the used of ==). To store it in a file we can simply > it into a file as shown below…

(my_venv) Jons-MacBook-Pro:my_first_project jon$ pip freeze > requirements.txt
(my_venv) Jons-MacBook-Pro:my_first_project jon$ more requirements.txt 
PyYAML==3.12
(my_venv) Jons-MacBook-Pro:my_first_project jon$ pip install netaddr
Collecting netaddr
  Using cached netaddr-0.7.19-py2.py3-none-any.whl
Installing collected packages: netaddr
Successfully installed netaddr-0.7.19
(my_venv) Jons-MacBook-Pro:my_first_project jon$ pip freeze > requirements.txt
(my_venv) Jons-MacBook-Pro:my_first_project jon$ more requirements.txt 
netaddr==0.7.19
PyYAML==3.12
(my_venv) Jons-MacBook-Pro:my_first_project jon$

Ok – so now we have two packages in our requirements.txt file. This is great and all, but what do we do with this? Using PIP we can install packages based on this file like so…

(my_venv) Jons-MacBook-Pro:my_first_project jon$ deactivate
Jons-MacBook-Pro:my_first_project jon$ source my_venv3/bin/activate
(my_venv3) Jons-MacBook-Pro:my_first_project jon$ pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
pip (9.0.1)
setuptools (36.5.0)
wheel (0.30.0)
(my_venv3) Jons-MacBook-Pro:my_first_project jon$ pip install -r requirements.txt 
Collecting netaddr==0.7.19 (from -r requirements.txt (line 1))
  Using cached netaddr-0.7.19-py2.py3-none-any.whl
Collecting PyYAML==3.12 (from -r requirements.txt (line 2))
Installing collected packages: netaddr, PyYAML
Successfully installed PyYAML-3.12 netaddr-0.7.19
(my_venv3) Jons-MacBook-Pro:my_first_project jon$ 
(my_venv3) Jons-MacBook-Pro:my_first_project jon$ pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
netaddr (0.7.19)
pip (9.0.1)
PyYAML (3.12)
setuptools (36.5.0)
wheel (0.30.0)
(my_venv3) Jons-MacBook-Pro:my_first_project jon$

Notice here that we first deactivated the virtual environment we were in, then we activated another, showed that only the base packages were there, and then we used the requirements.txt file to install all of the packages we needed through PIP. Pretty slick right?

So now we know how to handle Python versions and packages inside of virtual environments. This is powerful by itself, but there’s even more ways we can optimize this. We’ll build on this in our next post. Stay tuned!

Leave a Reply

Your email address will not be published.