Python Pieces: Using PyEnv

      No Comments on Python Pieces: Using PyEnv

If you’re like me – one of the most frustrating things about Python is version management. You get a new Mac, the system default is 2.x something, you need 3.x something, and you’re wondering what the best (right) way to get the version you want installed. You install Python 3 but the default Python version stays the same until you do some symlink hack thing that you know is just creating a mess. So for awhile you just call python3 explicitly but then you realize that all of the packages you installed using pip are no longer available and you need to install them again using pip3.

Sound familiar? Maybe I’m the only one that struggles with this – but I tend to muddle my way through just making things work while in the back of my head I know that Im creating a complete disaster of the local Python installation. I shall muddle no longer thanks to PyEnv. I was recently introduced to the tool and it’s a total game changer. It allows you to seemlessly manage your local Python install, easily install different versions, easily switch versions, and even has the capability of automgically switching versions for you based on the project you’re currently working on. So – let’s dive right into how we make this all work.

First things first – we need to install PyEnv. To do that – hit up the PyEnv-Installer page. You can also clone the PyEnv repo and do things that way if you’re uncomfortable with piping randomness into bash. For the sake of simplicity though, we’ll use the installer version which has you run this command…

curl https://pyenv.run | bash

The install should be pretty straight forward but you’ll likely hit this warning message at the end of the install…

WARNING: seems you still have not added 'pyenv' to the load path.

# Load pyenv automatically by adding
# the following to ~/.bashrc:

export PATH="/Users/user/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
```

Note: I’ve run into issues with my terminal responding super slowly if I include the last line in my .zshrc. For now, I’ve omitted eval "$(pyenv virtualenv-init -)" from the .zshrc. We’ll talk more about that later in my next post where we talk about PyEnv and virtual environments. It’s not needed for this post.

As the message describes – we need to do add PyEnv to the load path. We’ll see why we need to do this shortly but for now let’s just focus on getting that done. Check and see what shell you’re using…

users-MacBook-Pro:~ user$ echo $SHELL
/bin/zsh
users-MacBook-Pro:~ user$ 

In my case Im using zsh so I need to add that stuff to .zshrc. If you’re using bash you’ll need to to add it to .bashrc. Once done, restart your terminal (or source the requisite file). Now we should have access to the pyenv CLI tool. Take a look at what versions PyEnv knows about…

[email protected] ~ % pyenv versions
* system (set by /Users/user/.pyenv/version)
[email protected] ~ % 

So right now – it only knows about the system default version. Let’s see what that is…

[email protected] ~ % which python
/usr/bin/python
[email protected] ~ % python --version
Python 2.7.16
[email protected] ~ % 

Nice – Python 2.7.16. Almost certainly not the version you want to use. So let’s take a look at what versions PyEnv knows about that are available for install. Specifically, let’s look for versions in 3.8…

[email protected] ~ % pyenv install --list | grep "3.8"   
  3.8.0
  3.8-dev
  3.8.1
  3.8.2
  3.8.3
  3.8.4
  3.8.5
  miniconda-3.8.3
  miniconda3-3.8.3
[email protected] ~ % 

If you omit the grep you’ll see an outstanding list of Python versions you can install. For right now – let’s focus on installing 3.8.5. To do that we simply do this…

[email protected] ~ % pyenv install 3.8.5
python-build: use openssl from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.5.tar.xz...
-> https://www.python.org/ftp/python/3.8.5/Python-3.8.5.tar.xz
Installing Python-3.8.5...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.8.5 to /Users/user/.pyenv/versions/3.8.5

[email protected] ~ % 

Note that this will take some time as it’s building Python from scratch. Once installed, we should see it in our versions…

[email protected] ~ % pyenv versions
* system (set by /Users/user/.pyenv/version)
  3.8.5
[email protected] ~ % 

Note that the star is still next to the system row meaning that we’re still currently using the system default. If we wish to switch our new global version to 3.8.5 we simply say…

[email protected] ~ % pyenv global 3.8.5
[email protected] ~ % 
[email protected] ~ % which python
/Users/user/.pyenv/shims/python
[email protected] ~ % python --version
Python 3.8.5
[email protected] ~ % 

Nice! So now our global version of Python will be 3.8.5. Now that we switched Python versions for global use, let’s talk about local use. This is where PyEnv really starts to shine. Let’s install a couple more versions of Python so we can play around…

[email protected] ~ % pyenv install 2.7.18
python-build: use openssl from homebrew
python-build: use readline from homebrew
Downloading Python-2.7.18.tar.xz...
-> https://www.python.org/ftp/python/2.7.18/Python-2.7.18.tar.xz
Installing Python-2.7.18...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-2.7.18 to /Users/user/.pyenv/versions/2.7.18

[email protected] ~ % pyenv install 3.5.1 
python-build: use openssl from homebrew
python-build: use readline from homebrew
Downloading Python-3.5.1.tar.xz...
-> https://www.python.org/ftp/python/3.5.1/Python-3.5.1.tar.xz
Installing Python-3.5.1...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.5.1 to /Users/user/.pyenv/versions/3.5.1

[email protected] ~ % 

Ok – so we should now see those as options…

[email protected] ~ % pyenv versions
  system
  2.7.18
  3.5.1
* 3.8.5 (set by /Users/user/.pyenv/version)
[email protected] ~ % 

Awesome! Now let’s create a set of test projects and associated folders…

[email protected] ~ % cd test_projects   
[email protected] test_projects % mkdir project_1
[email protected] test_projects % mkdir project_2
[email protected] test_projects % mkdir project_3

Alright – so now let’s start in project_1 and setup PyEnv…

[email protected] test_projects % cd project_1
[email protected] project_1 % pyenv local 2.7.18
[email protected] project_1 % python --version
Python 2.7.18
[email protected] project_1 % 

Notice how we use the command pyenv local rather than pyenv global. Using the local syntax sets the Python distro you wish to use in this local directory. So now let’s go to project_2…

[email protected] test_projects % cd project_2
[email protected] project_2 % pyenv local 3.5.1
[email protected] project_2 % python --version
Python 3.5.1
[email protected] project_2 % 

Nothing too surprising here. So now let’s go to project_3 and just see what the default is…

[email protected] project_2 % cd ..
[email protected] test_projects % cd project_3
[email protected] project_3 % python --version
Python 3.8.5
[email protected] project_3 % 

So we’re back to 3.8.5 which makes sense given that 3.8.5 is our global version and we haven’t set a local version here. Ready for the cool part? Go back to project_1 or project_2 and check again…

[email protected] project_3 % cd ..
[email protected] test_projects % cd project_1
[email protected] project_1 % python --version
Python 2.7.18
[email protected] project_1 % cd ..
[email protected] test_projects % cd project_2    
[email protected] project_2 % python --version
Python 3.5.1
[email protected] project_2 % 

It automagically switches to the correct version based on what directory we’re in! How cool is that?! How does it do that? Well when we set the local version it creates a local file .python-version that holds the version number…

[email protected] project_2 % ls -al
total 8
drwxr-xr-x  3 user  staff   96 Sep 11 20:10 .
drwxr-xr-x  6 user  staff  192 Sep 11 20:07 ..
-rw-r--r--  1 user  staff    6 Sep 11 20:10 .python-version
[email protected] project_2 % more .python-version 
3.5.1
[email protected] project_2 % 

Now you might be wondering how this all works though. The fundamental thing that makes PyEnv work are “shims”. They’re basically scripts that redirect commands to PyEnv. We enabled this by making those initial changes to our .zshrc file above. If we look at our $PATH variable now we’ll see it has changed…

[email protected] ~ % echo $PATH
/Users/user/.pyenv/shims:/Users/user/.pyenv/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
[email protected] ~ % 

Notice how the PyEnv directories are now at the front of the path? And if we look in that directory we’ll see “shims” for all of the relevant Python commands…

[email protected] shims % ls    
2to3			easy_install-3.8	pip2.7			pydoc3.8		python2.7-gdb.py	python3.5m-config
2to3-3.5		idle			pip3			python			python3			python3.8
2to3-3.8		idle3			pip3.5			python-config		python3-config		python3.8-config
chardetect		idle3.5			pip3.8			python2			python3.5		python3.8-gdb.py
easy_install		idle3.8			pydoc			python2-config		python3.5-config	pyvenv
easy_install-2.7	pip			pydoc3			python2.7		python3.5-gdb.py	pyvenv-3.5
easy_install-3.5	pip2			pydoc3.5		python2.7-config	python3.5m		smtpd.py
[email protected] shims % 

In fact – we can even see this if we just look at what Python version we’re using right now…

[email protected] shims % which python
/Users/user/.pyenv/shims/python
[email protected] shims % 

So just by calling python a search in PATH catches the PyEnv shim first and returns that rather than the system default Python. And if we look at this file we’ll see it’s a script that just puts PyEnv in the way of our command…

[email protected] shims % more /Users/user/.pyenv/shims/python
#!/usr/bin/env bash
set -e
[ -n "$PYENV_DEBUG" ] && set -x

program="${0##*/}"
if [[ "$program" = "python"* ]]; then
  for arg; do
    case "$arg" in
    -c* | -- ) break ;;
    */* )
      if [ -f "$arg" ]; then
        export PYENV_FILE_ARG="$arg"
        break
      fi
      ;;
    esac
  done
fi

export PYENV_ROOT="/Users/user/.pyenv"
exec "/Users/user/.pyenv/libexec/pyenv" exec "$program" "[email protected]"
[email protected] shims % 

Pretty cool huh? If you want to see the actual Python you’re using you can call the pyenv which command to see the actual path…

[email protected] project_1 % pyenv which python
/Users/user/.pyenv/versions/2.7.18/bin/python
[email protected] project_1 % pyenv which pip   
/Users/user/.pyenv/versions/2.7.18/bin/pip
[email protected] project_1 % 

Same goes for PIP and any other Python tools that have shim files. Speaking of PIP, how does that work with PyEnv? Let’s take a look…

[email protected] project_1 % pyenv which python
/Users/user/.pyenv/versions/2.7.18/bin/python
[email protected] project_1 % pip list
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package    Version  
---------- ---------
certifi    2020.6.20
chardet    3.0.4    
idna       2.10     
pip        19.2.3   
setuptools 41.2.0   
urllib3    1.25.10  
WARNING: You are using pip version 19.2.3, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
[email protected] project_1 % 

Lets say we want to use pyyaml. Let’s install that quick…

[email protected] project_1 % pip install pyyaml
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting pyyaml
  Using cached https://files.pythonhosted.org/packages/64/c2/b80047c7ac2478f9501676c988a5411ed5572f35d1beff9cae07d321512c/PyYAML-5.3.1.tar.gz
Installing collected packages: pyyaml
  Running setup.py install for pyyaml ... done
Successfully installed pyyaml-5.3.1
WARNING: You are using pip version 19.2.3, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
[email protected] project_1 % pip list
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package    Version  
---------- ---------
certifi    2020.6.20
chardet    3.0.4    
idna       2.10     
pip        19.2.3   
PyYAML     5.3.1    
setuptools 41.2.0   
urllib3    1.25.10  
WARNING: You are using pip version 19.2.3, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
[email protected] project_1 % 

Cool – now let’s go and take a look at one of our other Python installs….

[email protected] project_1 % cd ..
[email protected] test_projects % cd project_2
[email protected] project_2 % pyenv which python
/Users/user/.pyenv/versions/3.5.1/bin/python
[email protected] project_2 % pip list
pip (7.1.2)
setuptools (18.2)
You are using pip version 7.1.2, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
[email protected] project_2 % 

No pyyaml. So it seems to be that the PIP installs are per “environment” or version. Let’s test this out by setting a local Python version for project_3 that matches project_1…

[email protected] project_3 % cd ..
[email protected] test_projects % cd project_2
[email protected] project_2 % cd ..
[email protected] test_projects % cd project_3
[email protected] project_3 % pyenv which python
/Users/user/.pyenv/versions/3.8.5/bin/python
[email protected] project_3 % pip list
Package    Version
---------- ---------
certifi    2020.6.20
chardet    3.0.4
idna       2.10
pip        20.1.1
requests   2.24.0
setuptools 47.1.0
urllib3    1.25.10
WARNING: You are using pip version 20.1.1; however, version 20.2.3 is available.
You should consider upgrading via the '/Users/user/.pyenv/versions/3.8.5/bin/python3.8 -m pip install --upgrade pip' command.
[email protected] project_3 % pyenv local 2.7.18
[email protected] project_3 % pip list
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package    Version  
---------- ---------
certifi    2020.6.20
chardet    3.0.4    
idna       2.10     
pip        19.2.3   
PyYAML     5.3.1    
setuptools 41.2.0   
urllib3    1.25.10  
WARNING: You are using pip version 19.2.3, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
[email protected] project_3 % 

We can see that in the global version (3.8.5) we didn’t have the pyyaml library but the instant we switched to version 2.7.18 we inherited all of the packages that came with that version. Pretty cool right? So while we have package management by version we still don’t have package management by project which is what we really want. The solution for that? Virtual environments of course! Luckily PyEnv works well with those as well and we’ll start exploring those in our next post. Stay tuned!

Leave a Reply

Your email address will not be published. Required fields are marked *