Python Pieces: Using PyEnv

      No Comments on Python Pieces: Using PyEnv

If you’re like me – one of the most frustrating things about Python is version management. You get a new Mac, the system default is 2.x something, you need 3.x something, and you’re wondering what the best (right) way to get the version you want installed. You install Python 3 but the default Python version stays the same until you do some symlink hack thing that you know is just creating a mess. So for awhile you just call python3 explicitly but then you realize that all of the packages you installed using pip are no longer available and you need to install them again using pip3.

Sound familiar? Maybe I’m the only one that struggles with this – but I tend to muddle my way through just making things work while in the back of my head I know that Im creating a complete disaster of the local Python installation. I shall muddle no longer thanks to PyEnv. I was recently introduced to the tool and it’s a total game changer. It allows you to seemlessly manage your local Python install, easily install different versions, easily switch versions, and even has the capability of automgically switching versions for you based on the project you’re currently working on. So – let’s dive right into how we make this all work.

First things first – we need to install PyEnv. To do that – hit up the PyEnv-Installer page. You can also clone the PyEnv repo and do things that way if you’re uncomfortable with piping randomness into bash. For the sake of simplicity though, we’ll use the installer version which has you run this command…

curl https://pyenv.run | bash

The install should be pretty straight forward but you’ll likely hit this warning message at the end of the install…

WARNING: seems you still have not added 'pyenv' to the load path.

# Load pyenv automatically by adding
# the following to ~/.bashrc:

export PATH="/Users/user/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
```

Note: I’ve run into issues with my terminal responding super slowly if I include the last line in my .zshrc. For now, I’ve omitted eval "$(pyenv virtualenv-init -)" from the .zshrc. We’ll talk more about that later in my next post where we talk about PyEnv and virtual environments. It’s not needed for this post.

As the message describes – we need to do add PyEnv to the load path. We’ll see why we need to do this shortly but for now let’s just focus on getting that done. Check and see what shell you’re using…

users-MacBook-Pro:~ user$ echo $SHELL
/bin/zsh
users-MacBook-Pro:~ user$ 

In my case Im using zsh so I need to add that stuff to .zshrc. If you’re using bash you’ll need to to add it to .bashrc. Once done, restart your terminal (or source the requisite file). Now we should have access to the pyenv CLI tool. Take a look at what versions PyEnv knows about…

user@users-MacBook-Pro ~ % pyenv versions
* system (set by /Users/user/.pyenv/version)
user@users-MacBook-Pro ~ % 

So right now – it only knows about the system default version. Let’s see what that is…

user@users-MacBook-Pro ~ % which python
/usr/bin/python
user@users-MacBook-Pro ~ % python --version
Python 2.7.16
user@users-MacBook-Pro ~ % 

Nice – Python 2.7.16. Almost certainly not the version you want to use. So let’s take a look at what versions PyEnv knows about that are available for install. Specifically, let’s look for versions in 3.8…

user@users-MacBook-Pro ~ % pyenv install --list | grep "3.8"   
  3.8.0
  3.8-dev
  3.8.1
  3.8.2
  3.8.3
  3.8.4
  3.8.5
  miniconda-3.8.3
  miniconda3-3.8.3
user@users-MacBook-Pro ~ % 

If you omit the grep you’ll see an outstanding list of Python versions you can install. For right now – let’s focus on installing 3.8.5. To do that we simply do this…

user@users-MacBook-Pro ~ % pyenv install 3.8.5
python-build: use openssl from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.5.tar.xz...
-> https://www.python.org/ftp/python/3.8.5/Python-3.8.5.tar.xz
Installing Python-3.8.5...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.8.5 to /Users/user/.pyenv/versions/3.8.5

user@users-MacBook-Pro ~ % 

Note that this will take some time as it’s building Python from scratch. Once installed, we should see it in our versions…

user@users-MacBook-Pro ~ % pyenv versions
* system (set by /Users/user/.pyenv/version)
  3.8.5
user@users-MacBook-Pro ~ % 

Note that the star is still next to the system row meaning that we’re still currently using the system default. If we wish to switch our new global version to 3.8.5 we simply say…

user@users-MacBook-Pro ~ % pyenv global 3.8.5
user@users-MacBook-Pro ~ % 
user@users-MacBook-Pro ~ % which python
/Users/user/.pyenv/shims/python
user@users-MacBook-Pro ~ % python --version
Python 3.8.5
user@users-MacBook-Pro ~ % 

Nice! So now our global version of Python will be 3.8.5. Now that we switched Python versions for global use, let’s talk about local use. This is where PyEnv really starts to shine. Let’s install a couple more versions of Python so we can play around…

user@users-MacBook-Pro ~ % pyenv install 2.7.18
python-build: use openssl from homebrew
python-build: use readline from homebrew
Downloading Python-2.7.18.tar.xz...
-> https://www.python.org/ftp/python/2.7.18/Python-2.7.18.tar.xz
Installing Python-2.7.18...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-2.7.18 to /Users/user/.pyenv/versions/2.7.18

user@users-MacBook-Pro ~ % pyenv install 3.5.1 
python-build: use openssl from homebrew
python-build: use readline from homebrew
Downloading Python-3.5.1.tar.xz...
-> https://www.python.org/ftp/python/3.5.1/Python-3.5.1.tar.xz
Installing Python-3.5.1...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.5.1 to /Users/user/.pyenv/versions/3.5.1

user@users-MacBook-Pro ~ % 

Ok – so we should now see those as options…

user@users-MacBook-Pro ~ % pyenv versions
  system
  2.7.18
  3.5.1
* 3.8.5 (set by /Users/user/.pyenv/version)
user@users-MacBook-Pro ~ % 

Awesome! Now let’s create a set of test projects and associated folders…

user@users-MacBook-Pro ~ % cd test_projects   
user@users-MacBook-Pro test_projects % mkdir project_1
user@users-MacBook-Pro test_projects % mkdir project_2
user@users-MacBook-Pro test_projects % mkdir project_3

Alright – so now let’s start in project_1 and setup PyEnv…

user@users-MacBook-Pro test_projects % cd project_1
user@users-MacBook-Pro project_1 % pyenv local 2.7.18
user@users-MacBook-Pro project_1 % python --version
Python 2.7.18
user@users-MacBook-Pro project_1 % 

Notice how we use the command pyenv local rather than pyenv global. Using the local syntax sets the Python distro you wish to use in this local directory. So now let’s go to project_2…

user@users-MacBook-Pro test_projects % cd project_2
user@users-MacBook-Pro project_2 % pyenv local 3.5.1
user@users-MacBook-Pro project_2 % python --version
Python 3.5.1
user@users-MacBook-Pro project_2 % 

Nothing too surprising here. So now let’s go to project_3 and just see what the default is…

user@users-MacBook-Pro project_2 % cd ..
user@users-MacBook-Pro test_projects % cd project_3
user@users-MacBook-Pro project_3 % python --version
Python 3.8.5
user@users-MacBook-Pro project_3 % 

So we’re back to 3.8.5 which makes sense given that 3.8.5 is our global version and we haven’t set a local version here. Ready for the cool part? Go back to project_1 or project_2 and check again…

user@users-MacBook-Pro project_3 % cd ..
user@users-MacBook-Pro test_projects % cd project_1
user@users-MacBook-Pro project_1 % python --version
Python 2.7.18
user@users-MacBook-Pro project_1 % cd ..
user@users-MacBook-Pro test_projects % cd project_2    
user@users-MacBook-Pro project_2 % python --version
Python 3.5.1
user@users-MacBook-Pro project_2 % 

It automagically switches to the correct version based on what directory we’re in! How cool is that?! How does it do that? Well when we set the local version it creates a local file .python-version that holds the version number…

user@users-MacBook-Pro project_2 % ls -al
total 8
drwxr-xr-x  3 user  staff   96 Sep 11 20:10 .
drwxr-xr-x  6 user  staff  192 Sep 11 20:07 ..
-rw-r--r--  1 user  staff    6 Sep 11 20:10 .python-version
user@users-MacBook-Pro project_2 % more .python-version 
3.5.1
user@users-MacBook-Pro project_2 % 

Now you might be wondering how this all works though. The fundamental thing that makes PyEnv work are “shims”. They’re basically scripts that redirect commands to PyEnv. We enabled this by making those initial changes to our .zshrc file above. If we look at our $PATH variable now we’ll see it has changed…

user@users-MacBook-Pro ~ % echo $PATH
/Users/user/.pyenv/shims:/Users/user/.pyenv/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
user@users-MacBook-Pro ~ % 

Notice how the PyEnv directories are now at the front of the path? And if we look in that directory we’ll see “shims” for all of the relevant Python commands…

user@users-MacBook-Pro shims % ls    
2to3			easy_install-3.8	pip2.7			pydoc3.8		python2.7-gdb.py	python3.5m-config
2to3-3.5		idle			pip3			python			python3			python3.8
2to3-3.8		idle3			pip3.5			python-config		python3-config		python3.8-config
chardetect		idle3.5			pip3.8			python2			python3.5		python3.8-gdb.py
easy_install		idle3.8			pydoc			python2-config		python3.5-config	pyvenv
easy_install-2.7	pip			pydoc3			python2.7		python3.5-gdb.py	pyvenv-3.5
easy_install-3.5	pip2			pydoc3.5		python2.7-config	python3.5m		smtpd.py
user@users-MacBook-Pro shims % 

In fact – we can even see this if we just look at what Python version we’re using right now…

user@users-MacBook-Pro shims % which python
/Users/user/.pyenv/shims/python
user@users-MacBook-Pro shims % 

So just by calling python a search in PATH catches the PyEnv shim first and returns that rather than the system default Python. And if we look at this file we’ll see it’s a script that just puts PyEnv in the way of our command…

user@users-MacBook-Pro shims % more /Users/user/.pyenv/shims/python
#!/usr/bin/env bash
set -e
[ -n "$PYENV_DEBUG" ] && set -x

program="${0##*/}"
if [[ "$program" = "python"* ]]; then
  for arg; do
    case "$arg" in
    -c* | -- ) break ;;
    */* )
      if [ -f "$arg" ]; then
        export PYENV_FILE_ARG="$arg"
        break
      fi
      ;;
    esac
  done
fi

export PYENV_ROOT="/Users/user/.pyenv"
exec "/Users/user/.pyenv/libexec/pyenv" exec "$program" "$@"
user@users-MacBook-Pro shims % 

Pretty cool huh? If you want to see the actual Python you’re using you can call the pyenv which command to see the actual path…

user@users-MacBook-Pro project_1 % pyenv which python
/Users/user/.pyenv/versions/2.7.18/bin/python
user@users-MacBook-Pro project_1 % pyenv which pip   
/Users/user/.pyenv/versions/2.7.18/bin/pip
user@users-MacBook-Pro project_1 % 

Same goes for PIP and any other Python tools that have shim files. Speaking of PIP, how does that work with PyEnv? Let’s take a look…

user@users-MacBook-Pro project_1 % pyenv which python
/Users/user/.pyenv/versions/2.7.18/bin/python
user@users-MacBook-Pro project_1 % pip list
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package    Version  
---------- ---------
certifi    2020.6.20
chardet    3.0.4    
idna       2.10     
pip        19.2.3   
setuptools 41.2.0   
urllib3    1.25.10  
WARNING: You are using pip version 19.2.3, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
user@users-MacBook-Pro project_1 % 

Lets say we want to use pyyaml. Let’s install that quick…

user@users-MacBook-Pro project_1 % pip install pyyaml
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting pyyaml
  Using cached https://files.pythonhosted.org/packages/64/c2/b80047c7ac2478f9501676c988a5411ed5572f35d1beff9cae07d321512c/PyYAML-5.3.1.tar.gz
Installing collected packages: pyyaml
  Running setup.py install for pyyaml ... done
Successfully installed pyyaml-5.3.1
WARNING: You are using pip version 19.2.3, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
user@users-MacBook-Pro project_1 % pip list
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package    Version  
---------- ---------
certifi    2020.6.20
chardet    3.0.4    
idna       2.10     
pip        19.2.3   
PyYAML     5.3.1    
setuptools 41.2.0   
urllib3    1.25.10  
WARNING: You are using pip version 19.2.3, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
user@users-MacBook-Pro project_1 % 

Cool – now let’s go and take a look at one of our other Python installs….

user@users-MacBook-Pro project_1 % cd ..
user@users-MacBook-Pro test_projects % cd project_2
user@users-MacBook-Pro project_2 % pyenv which python
/Users/user/.pyenv/versions/3.5.1/bin/python
user@users-MacBook-Pro project_2 % pip list
pip (7.1.2)
setuptools (18.2)
You are using pip version 7.1.2, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
user@users-MacBook-Pro project_2 % 

No pyyaml. So it seems to be that the PIP installs are per “environment” or version. Let’s test this out by setting a local Python version for project_3 that matches project_1…

user@users-MacBook-Pro project_3 % cd ..
user@users-MacBook-Pro test_projects % cd project_2
user@users-MacBook-Pro project_2 % cd ..
user@users-MacBook-Pro test_projects % cd project_3
user@users-MacBook-Pro project_3 % pyenv which python
/Users/user/.pyenv/versions/3.8.5/bin/python
user@users-MacBook-Pro project_3 % pip list
Package    Version
---------- ---------
certifi    2020.6.20
chardet    3.0.4
idna       2.10
pip        20.1.1
requests   2.24.0
setuptools 47.1.0
urllib3    1.25.10
WARNING: You are using pip version 20.1.1; however, version 20.2.3 is available.
You should consider upgrading via the '/Users/user/.pyenv/versions/3.8.5/bin/python3.8 -m pip install --upgrade pip' command.
user@users-MacBook-Pro project_3 % pyenv local 2.7.18
user@users-MacBook-Pro project_3 % pip list
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package    Version  
---------- ---------
certifi    2020.6.20
chardet    3.0.4    
idna       2.10     
pip        19.2.3   
PyYAML     5.3.1    
setuptools 41.2.0   
urllib3    1.25.10  
WARNING: You are using pip version 19.2.3, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
user@users-MacBook-Pro project_3 % 

We can see that in the global version (3.8.5) we didn’t have the pyyaml library but the instant we switched to version 2.7.18 we inherited all of the packages that came with that version. Pretty cool right? So while we have package management by version we still don’t have package management by project which is what we really want. The solution for that? Virtual environments of course! Luckily PyEnv works well with those as well and we’ll start exploring those in our next post. Stay tuned!

Leave a Reply

Your email address will not be published. Required fields are marked *