If you’re like me – one of the most frustrating things about Python is version management. You get a new Mac, the system default is 2.x something, you need 3.x something, and you’re wondering what the best (right) way to get the version you want installed. You install Python 3 but the default Python version stays the same until you do some symlink hack thing that you know is just creating a mess. So for awhile you just call python3
explicitly but then you realize that all of the packages you installed using pip
are no longer available and you need to install them again using pip3
.
Sound familiar? Maybe I’m the only one that struggles with this – but I tend to muddle my way through just making things work while in the back of my head I know that Im creating a complete disaster of the local Python installation. I shall muddle no longer thanks to PyEnv. I was recently introduced to the tool and it’s a total game changer. It allows you to seemlessly manage your local Python install, easily install different versions, easily switch versions, and even has the capability of automgically switching versions for you based on the project you’re currently working on. So – let’s dive right into how we make this all work.
First things first – we need to install PyEnv. To do that – hit up the PyEnv-Installer page. You can also clone the PyEnv repo and do things that way if you’re uncomfortable with piping randomness into bash
. For the sake of simplicity though, we’ll use the installer version which has you run this command…
curl https://pyenv.run | bash
The install should be pretty straight forward but you’ll likely hit this warning message at the end of the install…
WARNING: seems you still have not added 'pyenv' to the load path.
# Load pyenv automatically by adding
# the following to ~/.bashrc:
export PATH="/Users/user/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
```
Note: I’ve run into issues with my terminal responding super slowly if I include the last line in my .zshrc
. For now, I’ve omitted eval "$(pyenv virtualenv-init -)"
from the .zshrc
. We’ll talk more about that later in my next post where we talk about PyEnv and virtual environments. It’s not needed for this post.
As the message describes – we need to do add PyEnv to the load path. We’ll see why we need to do this shortly but for now let’s just focus on getting that done. Check and see what shell you’re using…
users-MacBook-Pro:~ user$ echo $SHELL
/bin/zsh
users-MacBook-Pro:~ user$
In my case Im using zsh
so I need to add that stuff to .zshrc
. If you’re using bash
you’ll need to to add it to .bashrc
. Once done, restart your terminal (or source the requisite file). Now we should have access to the pyenv
CLI tool. Take a look at what versions PyEnv knows about…
user@users-MacBook-Pro ~ % pyenv versions
* system (set by /Users/user/.pyenv/version)
user@users-MacBook-Pro ~ %
So right now – it only knows about the system default version. Let’s see what that is…
user@users-MacBook-Pro ~ % which python
/usr/bin/python
user@users-MacBook-Pro ~ % python --version
Python 2.7.16
user@users-MacBook-Pro ~ %
Nice – Python 2.7.16. Almost certainly not the version you want to use. So let’s take a look at what versions PyEnv knows about that are available for install. Specifically, let’s look for versions in 3.8…
user@users-MacBook-Pro ~ % pyenv install --list | grep "3.8"
3.8.0
3.8-dev
3.8.1
3.8.2
3.8.3
3.8.4
3.8.5
miniconda-3.8.3
miniconda3-3.8.3
user@users-MacBook-Pro ~ %
If you omit the grep
you’ll see an outstanding list of Python versions you can install. For right now – let’s focus on installing 3.8.5
. To do that we simply do this…
user@users-MacBook-Pro ~ % pyenv install 3.8.5
python-build: use openssl from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.5.tar.xz...
-> https://www.python.org/ftp/python/3.8.5/Python-3.8.5.tar.xz
Installing Python-3.8.5...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.8.5 to /Users/user/.pyenv/versions/3.8.5
user@users-MacBook-Pro ~ %
Note that this will take some time as it’s building Python from scratch. Once installed, we should see it in our versions…
user@users-MacBook-Pro ~ % pyenv versions
* system (set by /Users/user/.pyenv/version)
3.8.5
user@users-MacBook-Pro ~ %
Note that the star is still next to the system
row meaning that we’re still currently using the system default. If we wish to switch our new global version to 3.8.5
we simply say…
user@users-MacBook-Pro ~ % pyenv global 3.8.5
user@users-MacBook-Pro ~ %
user@users-MacBook-Pro ~ % which python
/Users/user/.pyenv/shims/python
user@users-MacBook-Pro ~ % python --version
Python 3.8.5
user@users-MacBook-Pro ~ %
Nice! So now our global version of Python will be 3.8.5
. Now that we switched Python versions for global use, let’s talk about local use. This is where PyEnv really starts to shine. Let’s install a couple more versions of Python so we can play around…
user@users-MacBook-Pro ~ % pyenv install 2.7.18
python-build: use openssl from homebrew
python-build: use readline from homebrew
Downloading Python-2.7.18.tar.xz...
-> https://www.python.org/ftp/python/2.7.18/Python-2.7.18.tar.xz
Installing Python-2.7.18...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-2.7.18 to /Users/user/.pyenv/versions/2.7.18
user@users-MacBook-Pro ~ % pyenv install 3.5.1
python-build: use openssl from homebrew
python-build: use readline from homebrew
Downloading Python-3.5.1.tar.xz...
-> https://www.python.org/ftp/python/3.5.1/Python-3.5.1.tar.xz
Installing Python-3.5.1...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.5.1 to /Users/user/.pyenv/versions/3.5.1
user@users-MacBook-Pro ~ %
Ok – so we should now see those as options…
user@users-MacBook-Pro ~ % pyenv versions
system
2.7.18
3.5.1
* 3.8.5 (set by /Users/user/.pyenv/version)
user@users-MacBook-Pro ~ %
Awesome! Now let’s create a set of test projects and associated folders…
user@users-MacBook-Pro ~ % cd test_projects
user@users-MacBook-Pro test_projects % mkdir project_1
user@users-MacBook-Pro test_projects % mkdir project_2
user@users-MacBook-Pro test_projects % mkdir project_3
Alright – so now let’s start in project_1 and setup PyEnv…
user@users-MacBook-Pro test_projects % cd project_1
user@users-MacBook-Pro project_1 % pyenv local 2.7.18
user@users-MacBook-Pro project_1 % python --version
Python 2.7.18
user@users-MacBook-Pro project_1 %
Notice how we use the command pyenv local
rather than pyenv global
. Using the local syntax sets the Python distro you wish to use in this local directory. So now let’s go to project_2…
user@users-MacBook-Pro test_projects % cd project_2
user@users-MacBook-Pro project_2 % pyenv local 3.5.1
user@users-MacBook-Pro project_2 % python --version
Python 3.5.1
user@users-MacBook-Pro project_2 %
Nothing too surprising here. So now let’s go to project_3 and just see what the default is…
user@users-MacBook-Pro project_2 % cd ..
user@users-MacBook-Pro test_projects % cd project_3
user@users-MacBook-Pro project_3 % python --version
Python 3.8.5
user@users-MacBook-Pro project_3 %
So we’re back to 3.8.5 which makes sense given that 3.8.5
is our global version and we haven’t set a local version here. Ready for the cool part? Go back to project_1 or project_2 and check again…
user@users-MacBook-Pro project_3 % cd ..
user@users-MacBook-Pro test_projects % cd project_1
user@users-MacBook-Pro project_1 % python --version
Python 2.7.18
user@users-MacBook-Pro project_1 % cd ..
user@users-MacBook-Pro test_projects % cd project_2
user@users-MacBook-Pro project_2 % python --version
Python 3.5.1
user@users-MacBook-Pro project_2 %
It automagically switches to the correct version based on what directory we’re in! How cool is that?! How does it do that? Well when we set the local version it creates a local file .python-version
that holds the version number…
user@users-MacBook-Pro project_2 % ls -al
total 8
drwxr-xr-x 3 user staff 96 Sep 11 20:10 .
drwxr-xr-x 6 user staff 192 Sep 11 20:07 ..
-rw-r--r-- 1 user staff 6 Sep 11 20:10 .python-version
user@users-MacBook-Pro project_2 % more .python-version
3.5.1
user@users-MacBook-Pro project_2 %
Now you might be wondering how this all works though. The fundamental thing that makes PyEnv work are “shims”. They’re basically scripts that redirect commands to PyEnv. We enabled this by making those initial changes to our .zshrc
file above. If we look at our $PATH
variable now we’ll see it has changed…
user@users-MacBook-Pro ~ % echo $PATH
/Users/user/.pyenv/shims:/Users/user/.pyenv/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
user@users-MacBook-Pro ~ %
Notice how the PyEnv directories are now at the front of the path? And if we look in that directory we’ll see “shims” for all of the relevant Python commands…
user@users-MacBook-Pro shims % ls
2to3 easy_install-3.8 pip2.7 pydoc3.8 python2.7-gdb.py python3.5m-config
2to3-3.5 idle pip3 python python3 python3.8
2to3-3.8 idle3 pip3.5 python-config python3-config python3.8-config
chardetect idle3.5 pip3.8 python2 python3.5 python3.8-gdb.py
easy_install idle3.8 pydoc python2-config python3.5-config pyvenv
easy_install-2.7 pip pydoc3 python2.7 python3.5-gdb.py pyvenv-3.5
easy_install-3.5 pip2 pydoc3.5 python2.7-config python3.5m smtpd.py
user@users-MacBook-Pro shims %
In fact – we can even see this if we just look at what Python version we’re using right now…
user@users-MacBook-Pro shims % which python
/Users/user/.pyenv/shims/python
user@users-MacBook-Pro shims %
So just by calling python
a search in PATH
catches the PyEnv shim first and returns that rather than the system default Python. And if we look at this file we’ll see it’s a script that just puts PyEnv in the way of our command…
user@users-MacBook-Pro shims % more /Users/user/.pyenv/shims/python
#!/usr/bin/env bash
set -e
[ -n "$PYENV_DEBUG" ] && set -x
program="${0##*/}"
if [[ "$program" = "python"* ]]; then
for arg; do
case "$arg" in
-c* | -- ) break ;;
*/* )
if [ -f "$arg" ]; then
export PYENV_FILE_ARG="$arg"
break
fi
;;
esac
done
fi
export PYENV_ROOT="/Users/user/.pyenv"
exec "/Users/user/.pyenv/libexec/pyenv" exec "$program" "$@"
user@users-MacBook-Pro shims %
Pretty cool huh? If you want to see the actual Python you’re using you can call the pyenv which
command to see the actual path…
user@users-MacBook-Pro project_1 % pyenv which python
/Users/user/.pyenv/versions/2.7.18/bin/python
user@users-MacBook-Pro project_1 % pyenv which pip
/Users/user/.pyenv/versions/2.7.18/bin/pip
user@users-MacBook-Pro project_1 %
Same goes for PIP and any other Python tools that have shim files. Speaking of PIP, how does that work with PyEnv? Let’s take a look…
user@users-MacBook-Pro project_1 % pyenv which python
/Users/user/.pyenv/versions/2.7.18/bin/python
user@users-MacBook-Pro project_1 % pip list
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package Version
---------- ---------
certifi 2020.6.20
chardet 3.0.4
idna 2.10
pip 19.2.3
setuptools 41.2.0
urllib3 1.25.10
WARNING: You are using pip version 19.2.3, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
user@users-MacBook-Pro project_1 %
Lets say we want to use pyyaml
. Let’s install that quick…
user@users-MacBook-Pro project_1 % pip install pyyaml
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting pyyaml
Using cached https://files.pythonhosted.org/packages/64/c2/b80047c7ac2478f9501676c988a5411ed5572f35d1beff9cae07d321512c/PyYAML-5.3.1.tar.gz
Installing collected packages: pyyaml
Running setup.py install for pyyaml ... done
Successfully installed pyyaml-5.3.1
WARNING: You are using pip version 19.2.3, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
user@users-MacBook-Pro project_1 % pip list
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package Version
---------- ---------
certifi 2020.6.20
chardet 3.0.4
idna 2.10
pip 19.2.3
PyYAML 5.3.1
setuptools 41.2.0
urllib3 1.25.10
WARNING: You are using pip version 19.2.3, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
user@users-MacBook-Pro project_1 %
Cool – now let’s go and take a look at one of our other Python installs….
user@users-MacBook-Pro project_1 % cd ..
user@users-MacBook-Pro test_projects % cd project_2
user@users-MacBook-Pro project_2 % pyenv which python
/Users/user/.pyenv/versions/3.5.1/bin/python
user@users-MacBook-Pro project_2 % pip list
pip (7.1.2)
setuptools (18.2)
You are using pip version 7.1.2, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
user@users-MacBook-Pro project_2 %
No pyyaml
. So it seems to be that the PIP installs are per “environment” or version. Let’s test this out by setting a local Python version for project_3 that matches project_1…
user@users-MacBook-Pro project_3 % cd ..
user@users-MacBook-Pro test_projects % cd project_2
user@users-MacBook-Pro project_2 % cd ..
user@users-MacBook-Pro test_projects % cd project_3
user@users-MacBook-Pro project_3 % pyenv which python
/Users/user/.pyenv/versions/3.8.5/bin/python
user@users-MacBook-Pro project_3 % pip list
Package Version
---------- ---------
certifi 2020.6.20
chardet 3.0.4
idna 2.10
pip 20.1.1
requests 2.24.0
setuptools 47.1.0
urllib3 1.25.10
WARNING: You are using pip version 20.1.1; however, version 20.2.3 is available.
You should consider upgrading via the '/Users/user/.pyenv/versions/3.8.5/bin/python3.8 -m pip install --upgrade pip' command.
user@users-MacBook-Pro project_3 % pyenv local 2.7.18
user@users-MacBook-Pro project_3 % pip list
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package Version
---------- ---------
certifi 2020.6.20
chardet 3.0.4
idna 2.10
pip 19.2.3
PyYAML 5.3.1
setuptools 41.2.0
urllib3 1.25.10
WARNING: You are using pip version 19.2.3, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
user@users-MacBook-Pro project_3 %
We can see that in the global version (3.8.5
) we didn’t have the pyyaml
library but the instant we switched to version 2.7.18
we inherited all of the packages that came with that version. Pretty cool right? So while we have package management by version we still don’t have package management by project which is what we really want. The solution for that? Virtual environments of course! Luckily PyEnv works well with those as well and we’ll start exploring those in our next post. Stay tuned!