In my last post, we talked about PyEnv and how it can help manage your local Python environments. As it turns out it can also help you manage virtual environments as well! However – pursuing this functionality took me down a rabbit hole that was a bit deeper than expected. The way that PyEnv works causes some behaviors (and on my end assumptions) to change which made me start questioning some of the things that I’ve always just taken for granted. In other words – prepare yourself to go down the rabbit hole with me.
At first glance PyEnv promised the same sort of awesome automagically context switching craziness that we saw previously work with Python versions. However – the virtual environment management implementation with PyEnv felt rather foreign (and maybe a little clunky?) to me. Most notably, as I pointed out in my last post, the .zshrc alias provided to make the auto activation piece work slows down my terminal immensely which is why I omitted using it. A slow terminal is about the worst thing I can think of…
That said – I still think it’s worth reviewing what it can offer so you can make your own decision. But before we dive into that – I want to point out something about PyEnv that I should have mentioned in my last post. As some folks have pointed out, PyEnv is just creating another layer of misdirection on your laptop. Admittedly it took me sometime to fully grasp the “shims” concept and understand what was actually going on. However – at least for me – I was happy to trade this misdirection for what looks like a very clean Python installation and management approach. Instead of having Python versions thrown about my filesystem and trying to figure out if I had the right $PATH to actually use them – PyEnv neatly installs all of the versions I want to use in one place. Discussing this now actually dovetails quite nicely into how PyEnv handles virtual environments so let me back track for a second and let’s look at what PyEnv is doing with Python versions…
user@users-MacBook-Pro ~ % pyenv versions
system
2.7.18
3.5.1
* 3.8.5 (set by /Users/user/.pyenv/version)
user@users-MacBook-Pro ~ %
user@users-MacBook-Pro ~ % which python
/Users/user/.pyenv/shims/python
user@users-MacBook-Pro ~ % pyenv which python
/Users/user/.pyenv/versions/3.8.5/bin/python
user@users-MacBook-Pro ~ %
As we can see above – I have my system Python installed as well as 3 unique Python versions installed using PyEnv. If we use the standard which command we can see the shim as expected, but by using the pyenv which command we can see the real Python being used. Notice the file path – let’s take a look there…
user@users-MacBook-Pro ~ % cd /Users/user/.pyenv/versions
user@users-MacBook-Pro versions % ls
2.7.18 3.5.1 3.8.5
user@users-MacBook-Pro versions %
Let’s say we’ve finally decided we’re done with that 2.7.18 version and just want it off our laptop. How do we do that? Well PyEnv does have a handy uninstall command – but to prove my point let’s just delete that folder manually…
user@users-MacBook-Pro versions % rm -rf 2.7.18
user@users-MacBook-Pro versions % pyenv versions
system
3.5.1
* 3.8.5 (set by /Users/user/.pyenv/version)
user@users-MacBook-Pro versions %
Gonesies. This is perhaps one of the key features of PyEnv for me. It puts all of the things in one place. I no longer have to worry about lingering Python pieces laying around my filesystem causing issues. We can also prove the uninstall command works as well…
user@users-MacBook-Pro versions % pyenv uninstall 3.5.1
pyenv: remove /Users/user/.pyenv/versions/3.5.1? [y|N]y
pyenv: 3.5.1 uninstalled
user@users-MacBook-Pro versions % pyenv versions
system
* 3.8.5 (set by /Users/user/.pyenv/version)
user@users-MacBook-Pro versions %
Alright – so now that we’ve shown how and where the PyEnv based Python installs actually live (and why I think that’s awesome) let’s talk about how PyEnv handles venvs.
Note1: As we start talking about PyEnv and virtual environments do keep in mind what we just saw and how PyEnv handles installs and versions. It will come in handy during the rest of this blog post.
Note2: I apparently can’t stop calling them different things so please keep in mind that a venv, a virtualenv, and a virtual environment are the same thing. Not familiar with what they are and why to use them? Check this out!
PyEnv has handy built in functions to create venvs but starting to use them felt, at least for me, a little bit weird. After understanding how PyEnv was handling the venvs the model did make more sense to me – but if you’re used to working with venvs in a more traditional sense you might find this approach clunky or at the very least uncomfortable. That said let’s dive right into creating one. First Im going to reinstall some Python versions so we have some stuff to play with…
pyenv install 2.7.18
pyenv install 3.5.1
Cool. Now let’s make some folders to house our potential projects…
user@users-MacBook-Pro ~ % mkdir venv_projects
user@users-MacBook-Pro ~ % cd venv_projects
user@users-MacBook-Pro venv_projects % mkdir test_project_1
user@users-MacBook-Pro venv_projects % mkdir test_project_2
So let’s make a venv in test_project_1 with PyEnv…
user@users-MacBook-Pro venv_projects % cd test_project_1
user@users-MacBook-Pro test_project_1 %
user@users-MacBook-Pro test_project_1 % pyenv virtualenv 2.7.18 test_project_1
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting virtualenv
Using cached https://files.pythonhosted.org/packages/12/51/36c685ff2c1b2f7b4b5db29f3153159102ae0e0adaff3a26fd1448232e06/virtualenv-20.0.31-py2.py3-none-any.whl
Collecting appdirs<2,>=1.4.3 (from virtualenv)
Using cached https://files.pythonhosted.org/packages/3b/00/2344469e2084fb287c2e0b57b72910309874c3245463acd6cf5e3db69324/appdirs-1.4.4-py2.py3-none-any.whl
Collecting importlib-metadata<2,>=0.12; python_version < "3.8" (from virtualenv)
Using cached https://files.pythonhosted.org/packages/8e/58/cdea07eb51fc2b906db0968a94700866fc46249bdc75cac23f9d13168929/importlib_metadata-1.7.0-py2.py3-none-any.whl
Collecting six<2,>=1.9.0 (from virtualenv)
Using cached https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl
Collecting filelock<4,>=3.0.0 (from virtualenv)
Using cached https://files.pythonhosted.org/packages/14/ec/6ee2168387ce0154632f856d5cc5592328e9cf93127c5c9aeca92c8c16cb/filelock-3.0.12.tar.gz
Collecting importlib-resources>=1.0; python_version < "3.7" (from virtualenv)
Using cached https://files.pythonhosted.org/packages/ba/03/0f9595c0c2ef12590877f3c47e5f579759ce5caf817f8256d5dcbd8a1177/importlib_resources-3.0.0-py2.py3-none-any.whl
Collecting distlib<1,>=0.3.1 (from virtualenv)
Using cached https://files.pythonhosted.org/packages/f5/0a/490fa011d699bb5a5f3a0cf57de82237f52a6db9d40f33c53b2736c9a1f9/distlib-0.3.1-py2.py3-none-any.whl
Collecting pathlib2<3,>=2.3.3; python_version < "3.4" and sys_platform != "win32" (from virtualenv)
Using cached https://files.pythonhosted.org/packages/e9/45/9c82d3666af4ef9f221cbb954e1d77ddbb513faf552aea6df5f37f1a4859/pathlib2-2.3.5-py2.py3-none-any.whl
Collecting configparser>=3.5; python_version < "3" (from importlib-metadata<2,>=0.12; python_version < "3.8"->virtualenv)
Using cached https://files.pythonhosted.org/packages/7a/2a/95ed0501cf5d8709490b1d3a3f9b5cf340da6c433f896bbe9ce08dbe6785/configparser-4.0.2-py2.py3-none-any.whl
Collecting contextlib2; python_version < "3" (from importlib-metadata<2,>=0.12; python_version < "3.8"->virtualenv)
Using cached https://files.pythonhosted.org/packages/85/60/370352f7ef6aa96c52fb001831622f50f923c1d575427d021b8ab3311236/contextlib2-0.6.0.post1-py2.py3-none-any.whl
Collecting zipp>=0.5 (from importlib-metadata<2,>=0.12; python_version < "3.8"->virtualenv)
Using cached https://files.pythonhosted.org/packages/96/0a/67556e9b7782df7118c1f49bdc494da5e5e429c93aa77965f33e81287c8c/zipp-1.2.0-py2.py3-none-any.whl
Collecting typing; python_version < "3.5" (from importlib-resources>=1.0; python_version < "3.7"->virtualenv)
Using cached https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl
Collecting singledispatch; python_version < "3.4" (from importlib-resources>=1.0; python_version < "3.7"->virtualenv)
Using cached https://files.pythonhosted.org/packages/c5/10/369f50bcd4621b263927b0a1519987a04383d4a98fb10438042ad410cf88/singledispatch-3.4.0.3-py2.py3-none-any.whl
Collecting scandir; python_version < "3.5" (from pathlib2<3,>=2.3.3; python_version < "3.4" and sys_platform != "win32"->virtualenv)
Using cached https://files.pythonhosted.org/packages/df/f5/9c052db7bd54d0cbf1bc0bb6554362bba1012d03e5888950a4f5c5dadc4e/scandir-1.10.0.tar.gz
Installing collected packages: appdirs, configparser, contextlib2, scandir, six, pathlib2, zipp, importlib-metadata, filelock, typing, singledispatch, importlib-resources, distlib, virtualenv
Running setup.py install for scandir ... done
Running setup.py install for filelock ... done
Successfully installed appdirs-1.4.4 configparser-4.0.2 contextlib2-0.6.0.post1 distlib-0.3.1 filelock-3.0.12 importlib-metadata-1.7.0 importlib-resources-3.0.0 pathlib2-2.3.5 scandir-1.10.0 singledispatch-3.4.0.3 six-1.15.0 typing-3.7.4.3 virtualenv-20.0.31 zipp-1.2.0
WARNING: You are using pip version 19.2.3, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
created virtual environment CPython2.7.18.final.0-64 in 592ms
creator CPython2Posix(dest=/Users/user/.pyenv/versions/2.7.18/envs/test_project_1, clear=False, global=False)
seeder FromAppData(download=False, pip=bundle, wheel=bundle, setuptools=bundle, via=copy, app_data_dir=/Users/user/Library/Application Support/virtualenv)
added seed packages: pip==20.2.2, setuptools==44.1.1, wheel==0.35.1
activators PythonActivator,CShellActivator,FishActivator,PowerShellActivator,BashActivator
Looking in links: /var/folders/sb/1xk4f6tj1wq20ngdjw1cwdwm0000gn/T/tmpBBtVKE
Requirement already satisfied: setuptools in /Users/user/.pyenv/versions/2.7.18/envs/test_project_1/lib/python2.7/site-packages (44.1.1)
Requirement already satisfied: pip in /Users/user/.pyenv/versions/2.7.18/envs/test_project_1/lib/python2.7/site-packages (20.2.2)
user@users-MacBook-Pro test_project_1 %
Alright – that was a lot of output – but it sure looks like it installed all the things we need. For those of you familiar with working with venvs more traditionally, you might think to yourself that now is the time to do a source test_project_1/bin/activate. So let’s see if thats possible…
user@users-MacBook-Pro test_project_1 % ls -al
total 0
drwxr-xr-x 2 user staff 64 Sep 23 08:47 .
drwxr-xr-x 4 user staff 128 Sep 23 08:48 ..
user@users-MacBook-Pro test_project_1 %
Hrmmm. Ok – well maybe there’s some other magic going on let’s see what Python we’re using…
pyenv which python
/Users/user/.pyenv/versions/3.8.5/bin/python
user@users-MacBook-Pro test_project_1 %
More hrmmm. So it’s still using our global version. So what’s going on here? This is where things get a little different. PyEnv seems to treat venvs as just another version of Python. And when you step back and think about how they’re accomplishing this – it makes sense given the model they’ve defined. All the same, it feels different and can definitely make you think you’re doing it wrong if you don’t understand how PyEnv works. So what is going on? Let’s go and take a look at our versions directory again…
user@users-MacBook-Pro test_project_1 % cd /Users/user/.pyenv/versions
user@users-MacBook-Pro versions % ls
2.7.18 3.5.1 3.8.5 test_project_1
user@users-MacBook-Pro versions %
Aha! That looks familiar. But why is that there? Sure looks to me like the virtual environment is being created as another PyEnv version. But if we look closer…
user@users-MacBook-Pro versions % ls -al
total 0
drwxr-xr-x 6 user staff 192 Sep 23 08:49 .
drwxr-xr-x 26 user staff 832 Sep 22 10:20 ..
drwxr-xr-x 7 user staff 224 Sep 23 08:49 2.7.18
drwxr-xr-x 6 user staff 192 Sep 23 08:40 3.5.1
drwxr-xr-x 7 user staff 224 Sep 22 10:20 3.8.5
lrwxr-xr-x 1 user staff 54 Sep 23 08:49 test_project_1 -> /Users/user/.pyenv/versions/2.7.18/envs/test_project_1
user@users-MacBook-Pro versions %
AHA! Symlinks! So that file is just a symlink pointing to /Users/user/.pyenv/versions/2.7.18/envs/test_project_1. Let’s look in there…
user@users-MacBook-Pro versions % cd 2.7.18
user@users-MacBook-Pro 2.7.18 % ls
bin envs include lib share
user@users-MacBook-Pro 2.7.18 % cd envs
user@users-MacBook-Pro envs % ls
test_project_1
user@users-MacBook-Pro envs % cd test_project_1
user@users-MacBook-Pro test_project_1 % ls
bin include lib pyvenv.cfg
user@users-MacBook-Pro test_project_1 %
Interesting. So each venv looks like a discrete version of Python but is really created underneath the base version you specify. So in our case, since the venv we created uses 2.7.18 the files exist under /Users/user/.pyenv/versions/2.7.18/envs/. This is starting to look more like the setup of a “normal” virtualenv. Let’s dive into the bin directory and see what is there…
user@users-MacBook-Pro test_project_1 % cd bin
user@users-MacBook-Pro bin % ls -al
total 4248
drwxr-xr-x 23 user staff 736 Sep 23 08:49 .
drwxr-xr-x 7 user staff 224 Sep 23 08:49 ..
-rw-r--r-- 1 user staff 2243 Sep 23 08:49 activate
-rw-r--r-- 1 user staff 1462 Sep 23 08:49 activate.csh
-rw-r--r-- 1 user staff 3093 Sep 23 08:49 activate.fish
-rw-r--r-- 1 user staff 1751 Sep 23 08:49 activate.ps1
-rw-r--r-- 1 user staff 1199 Sep 23 08:49 activate_this.py
-rwxr-xr-x 1 user staff 279 Sep 23 08:49 easy_install
-rwxr-xr-x 1 user staff 279 Sep 23 08:49 easy_install-2.7
-rwxr-xr-x 1 user staff 279 Sep 23 08:49 easy_install2
-rwxr-xr-x 1 user staff 279 Sep 23 08:49 easy_install2.7
-rwxr-xr-x 1 user staff 270 Sep 23 08:49 pip
-rwxr-xr-x 1 user staff 270 Sep 23 08:49 pip-2.7
-rwxr-xr-x 1 user staff 270 Sep 23 08:49 pip2
-rwxr-xr-x 1 user staff 270 Sep 23 08:49 pip2.7
-rwxr-xr-x 1 user staff 126 Sep 23 08:49 pydoc
-rwxr-xr-x 1 user staff 2100208 Sep 23 08:49 python
lrwxr-xr-x 1 user staff 6 Sep 23 08:49 python2 -> python
lrwxr-xr-x 1 user staff 6 Sep 23 08:49 python2.7 -> python
-rwxr-xr-x 1 user staff 257 Sep 23 08:49 wheel
-rwxr-xr-x 1 user staff 257 Sep 23 08:49 wheel-2.7
-rwxr-xr-x 1 user staff 257 Sep 23 08:49 wheel2
-rwxr-xr-x 1 user staff 257 Sep 23 08:49 wheel2.7
user@users-MacBook-Pro bin %
Here we can see all of the python executables we’d expect to see for a given environment. So if we step back and think about the folder structure, and how PyEnv works, any guesses as to how we can leverage this venv? To do that, let’s go back to project directory and run this command…
user@users-MacBook-Pro test_project_1 % pyenv local test_project_1
user@users-MacBook-Pro test_project_1 % ls -al
total 8
drwxr-xr-x 3 user staff 96 Sep 23 10:39 .
drwxr-xr-x 4 user staff 128 Sep 23 08:48 ..
-rw-r--r-- 1 user staff 15 Sep 23 10:39 .python-version
user@users-MacBook-Pro test_project_1 % more .python-version
test_project_1
user@users-MacBook-Pro test_project_1 %
The first thing we need to do is set the local Python version. To do that we just say that the “local” Python version is the one in our venv. So let’s see how this looks…
user@users-MacBook-Pro test_project_1 % pyenv which python
/Users/user/.pyenv/versions/test_project_1/bin/python
user@users-MacBook-Pro test_project_1 %
Nice! So if you followed the setup from our last post and have put the following lines in your .zshrc file…
export PATH="/Users/user/.pyenv/bin:$PATH"
eval "$(pyenv init - zsh)"
Note: We are specifically omitting the eval "$(pyenv virtualenv-init - zsh)" line. We do NOT need that.
You should now be automagically using the correct version of Python…
user@users-MacBook-Pro test_project_1 % pyenv which python
/Users/user/.pyenv/versions/test_project_1/bin/python
user@users-MacBook-Pro test_project_1 % python
Python 2.7.18 (default, Sep 23 2020, 08:43:08)
[GCC 4.2.1 Compatible Apple LLVM 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
We should now be seeing the same automagic behavior we saw in the last post. Move out of this directory and we’ll be back to the global version…
user@users-MacBook-Pro test_project_1 % cd ..
user@users-MacBook-Pro venv_projects % pyenv which python
/Users/user/.pyenv/versions/3.8.5/bin/python
user@users-MacBook-Pro venv_projects %
Nice. But here is where I started questioning myself. By using the base functionality we established in the last post, we are already getting automatic switching to the version of Python that we want. If we think about virtual environments at a high level I think of them as providing two big things. The first is a specific version of Python and the second is a discrete set of Python packages managed typically by pip. It appears we’re already getting the first and we haven’t even had to “activate” our venv. Let’s test out the second…
user@users-MacBook-Pro test_project_1 % pip list
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package Version
---------- -------
pip 20.2.2
setuptools 44.1.1
wheel 0.35.1
WARNING: You are using pip version 20.2.2; however, version 20.2.3 is available.
You should consider upgrading via the '/Users/user/.pyenv/versions/2.7.18/envs/test_project_1/bin/python -m pip install --upgrade pip' command.
user@users-MacBook-Pro test_project_1 %
user@users-MacBook-Pro test_project_1 %
user@users-MacBook-Pro test_project_1 % pip install pyyaml
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Processing /Users/user/Library/Caches/pip/wheels/d1/d5/a0/3c27cdc8b0209c5fc1385afeee936cf8a71e13d885388b4be2/PyYAML-5.3.1-cp27-cp27m-macosx_10_15_x86_64.whl
Installing collected packages: pyyaml
Successfully installed pyyaml-5.3.1
WARNING: You are using pip version 20.2.2; however, version 20.2.3 is available.
You should consider upgrading via the '/Users/user/.pyenv/versions/2.7.18/envs/test_project_1/bin/python -m pip install --upgrade pip' command.
user@users-MacBook-Pro test_project_1 % pip list
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package Version
---------- -------
pip 20.2.2
PyYAML 5.3.1
setuptools 44.1.1
wheel 0.35.1
WARNING: You are using pip version 20.2.2; however, version 20.2.3 is available.
You should consider upgrading via the '/Users/user/.pyenv/versions/2.7.18/envs/test_project_1/bin/python -m pip install --upgrade pip' command.
user@users-MacBook-Pro test_project_1 % cd ..
user@users-MacBook-Pro venv_projects % pip list
Package Version
---------- ---------
certifi 2020.6.20
chardet 3.0.4
idna 2.10
pip 20.1.1
requests 2.24.0
setuptools 47.1.0
urllib3 1.25.10
WARNING: You are using pip version 20.1.1; however, version 20.2.3 is available.
You should consider upgrading via the '/Users/user/.pyenv/versions/3.8.5/bin/python3.8 -m pip install --upgrade pip' command.
user@users-MacBook-Pro venv_projects %
There’s a lot of output above – but the basic gist of what Im doing is installing pyyaml using pip from the test_project_1 directory. I then back up one directory and validate that I no longer see that package because I no longer have a local version of Python and am instead defaulting to the global Python (in other words I just switched versions of Python using PyEnv). To say this slightly differently, package management with PyEnv appears to be per “version”. Make sense? We saw some of this behavior previously. Recall at the end of the last post if we had two project folders both of which used the same local version of Python we inherited the packages between the two. Since my “global” version is different from my “local” version I no longer see the package I just installed. Now throw venvs on top of this with the understanding that each venv is also treated as a PyEnv “version”. If we create another virtual environment with PyEnv using the same version and set it as the local Python version we do NOT inherit the packages. For instance…
user@users-MacBook-Pro venv_projects % cd test_project_2
user@users-MacBook-Pro test_project_2 % pyenv virtualenv 2.7.18 test_project_2
created virtual environment CPython2.7.18.final.0-64 in 389ms
creator CPython2Posix(dest=/Users/user/.pyenv/versions/2.7.18/envs/test_project_2, clear=False, global=False)
seeder FromAppData(download=False, pip=bundle, wheel=bundle, setuptools=bundle, via=copy, app_data_dir=/Users/user/Library/Application Support/virtualenv)
added seed packages: pip==20.2.2, setuptools==44.1.1, wheel==0.35.1
activators PythonActivator,CShellActivator,FishActivator,PowerShellActivator,BashActivator
Looking in links: /var/folders/sb/1xk4f6tj1wq20ngdjw1cwdwm0000gn/T/tmpredLoM
Requirement already satisfied: setuptools in /Users/user/.pyenv/versions/2.7.18/envs/test_project_2/lib/python2.7/site-packages (44.1.1)
Requirement already satisfied: pip in /Users/user/.pyenv/versions/2.7.18/envs/test_project_2/lib/python2.7/site-packages (20.2.2)
user@users-MacBook-Pro test_project_2 % pyenv local test_project_2
user@users-MacBook-Pro test_project_2 % pip list
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package Version
---------- -------
pip 20.2.2
setuptools 44.1.1
wheel 0.35.1
WARNING: You are using pip version 20.2.2; however, version 20.2.3 is available.
You should consider upgrading via the '/Users/user/.pyenv/versions/2.7.18/envs/test_project_2/bin/python -m pip install --upgrade pip' command.
user@users-MacBook-Pro test_project_2 %
So at this point – I started questioning things. When would I ever need to actually “activate” any of these venvs? Once you grasp how PyEnv does the version switching, how versions are handled, and how PyEnv treats virtual environments as versions, you can start to see why you might not need to “activate” a virtual environment to get the same kind of behavior. If you’re still struggling with what Im getting at – this helped me visualize things (note some folders are removed to make this not huge)…
user@users-MacBook-Pro versions % tree -L 6 -d
.
├── 2.7.18
│ ├── bin
│ ├── envs
│ │ ├── test_project_1
│ │ │ ├── bin
│ │ │ ├── include
│ │ │ │ └── python2.7 -> /Users/user/.pyenv/versions/2.7.18/include/python2.7
│ │ │ └── lib
│ │ │ └── python2.7
│ │ │ ├── config
│ │ │ ├── lib-dynload -> /Users/user/.pyenv/versions/2.7.18/lib/python2.7/lib-dynload
│ │ │ └── site-packages
│ │ └── test_project_2
│ │ ├── bin
│ │ ├── include
│ │ │ └── python2.7 -> /Users/user/.pyenv/versions/2.7.18/include/python2.7
│ │ └── lib
│ │ └── python2.7
│ │ ├── config
│ │ ├── lib-dynload -> /Users/user/.pyenv/versions/2.7.18/lib/python2.7/lib-dynload
│ │ └── site-packages
│ ├── include
│ │ └── python2.7
│ ├── lib
│ │ ├── pkgconfig
│ │ └── python2.7
│ │ ├── bsddb
│ │ ├── compiler
│ │ ├── config
│ │ ├── ctypes
│ │ ├── curses
│ │ ├── distutils
│ │ ├── email
│ │ ├── encodings
│ │ ├── ensurepip
│ │ ├── hotshot
│ │ ├── idlelib
│ │ ├── importlib
│ │ ├── json
│ │ ├── lib-dynload
│ │ ├── lib-tk
│ │ ├── lib2to3
│ │ ├── logging
│ │ ├── multiprocessing
│ │ ├── plat-darwin
│ │ ├── plat-mac
│ │ ├── pydoc_data
│ │ ├── site-packages
│ │ ├── sqlite3
│ │ ├── test
│ │ ├── unittest
│ │ ├── wsgiref
│ │ └── xml
│ └── share
│ └── man
│ └── man1
├── 3.5.1
│ ├── bin
│ ├── envs
│ │ └── test_project_3
│ │ ├── bin
│ │ ├── include
│ │ └── lib
│ │ └── python3.5
│ │ └── site-packages
│ ├── include
│ │ └── python3.5m
│ ├── lib
│ │ ├── pkgconfig
│ │ └── python3.5
│ │ ├── __pycache__
│ │ ├── asyncio
│ │ ├── collections
│ │ ├── concurrent
│ │ ├── config-3.5m
│ │ ├── ctypes
│ │ ├── curses
│ │ ├── dbm
│ │ ├── distutils
│ │ ├── email
│ │ ├── encodings
│ │ ├── ensurepip
│ │ ├── html
│ │ ├── http
│ │ ├── idlelib
│ │ ├── importlib
│ │ ├── json
│ │ ├── lib-dynload
│ │ ├── lib2to3
│ │ ├── logging
│ │ ├── multiprocessing
│ │ ├── plat-darwin
│ │ ├── pydoc_data
│ │ ├── site-packages
│ │ ├── sqlite3
│ │ ├── test
│ │ ├── tkinter
│ │ ├── turtledemo
│ │ ├── unittest
│ │ ├── urllib
│ │ ├── venv
│ │ ├── wsgiref
│ │ ├── xml
│ │ └── xmlrpc
│ └── share
│ └── man
│ └── man1
├── 3.8.5
│ ├── bin
│ ├── envs
│ ├── include
│ │ └── python3.8
│ │ ├── cpython
│ │ └── internal
│ ├── lib
│ │ ├── pkgconfig
│ │ └── python3.8
│ │ ├── __pycache__
│ │ ├── asyncio
│ │ ├── collections
│ │ ├── concurrent
│ │ ├── config-3.8-darwin
│ │ ├── ctypes
│ │ ├── curses
│ │ ├── dbm
│ │ ├── distutils
│ │ ├── email
│ │ ├── encodings
│ │ ├── ensurepip
│ │ ├── html
│ │ ├── http
│ │ ├── idlelib
│ │ ├── importlib
│ │ ├── json
│ │ ├── lib-dynload
│ │ ├── lib2to3
│ │ ├── logging
│ │ ├── multiprocessing
│ │ ├── pydoc_data
│ │ ├── site-packages
│ │ ├── sqlite3
│ │ ├── test
│ │ ├── tkinter
│ │ ├── turtledemo
│ │ ├── unittest
│ │ ├── urllib
│ │ ├── venv
│ │ ├── wsgiref
│ │ ├── xml
│ │ └── xmlrpc
│ └── share
│ └── man
│ └── man1
├── test_project_1 -> /Users/user/.pyenv/versions/2.7.18/envs/test_project_1
├── test_project_2 -> /Users/user/.pyenv/versions/2.7.18/envs/test_project_2
└── test_project_3 -> /Users/user/.pyenv/versions/3.5.1/envs/test_project_3
555 directories
user@users-MacBook-Pro versions %
As you can see – each “version” has it’s own site-packages directory. At this point, it seems clear to me that each “version” (I keep putting it in quotes since venvs are clearly treated as versions) has it’s own package directory. So it seems pretty clear to me that without doing an activation I am getting a virtual environment like experience.
I’ll remind folks at this point that I am NOT using the automatic venv activation scripts that PyEnv suggests you use ($(pyenv virtualenv-init - zsh)).
What’s left at this point is to figure out how each version of Python is being instructed to use it’s local site-packages directory. After much digging, I found that it really came down to the site Python module. Turns out you can take a look at what’s there pretty easily…
user@users-MacBook-Pro test_project_1 % python -m site
sys.path = [
'/Users/user/venv_projects/test_project_1',
'/Users/user/.pyenv/versions/2.7.18/lib/python27.zip',
'/Users/user/.pyenv/versions/2.7.18/lib/python2.7',
'/Users/user/.pyenv/versions/2.7.18/lib/python2.7/plat-darwin',
'/Users/user/.pyenv/versions/2.7.18/lib/python2.7/plat-mac',
'/Users/user/.pyenv/versions/2.7.18/lib/python2.7/plat-mac/lib-scriptpackages',
'/Users/user/.pyenv/versions/2.7.18/lib/python2.7/lib-tk',
'/Users/user/.pyenv/versions/2.7.18/lib/python2.7/lib-old',
'/Users/user/.pyenv/versions/2.7.18/lib/python2.7/lib-dynload',
'/Users/user/.pyenv/versions/test_project_1/lib/python2.7/site-packages',
]
USER_BASE: '/Users/user/.local' (doesn't exist)
USER_SITE: '/Users/user/.local/lib/python2.7/site-packages' (doesn't exist)
ENABLE_USER_SITE: False
user@users-MacBook-Pro test_project_1 %
The really interesting output is the sys.path list. This list is used to search for Python packages and modules and is evaluated in order. Looking at the first entry we can see this makes sense. If you have other Python packages locally it will first search there. Then we have lines 4-11 which from what I can discern are created by the sites module based on platform and other information. But it’s the last line that really interested me. This line specifies the “global” site packages directory. Which you can find yourself by running this command…
python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
which I found here. If we run that in our project folder we’ll see that we get that last line there…
user@users-MacBook-Pro test_project_1 % python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
/Users/user/.pyenv/versions/test_project_1/lib/python2.7/site-packages
user@users-MacBook-Pro test_project_1 %
So this “global sites-packages” folder is where our Python packages are for a given project. Notice that this value changes as we switch versions of Python…
user@users-MacBook-Pro test_project_1 % cd ..
user@users-MacBook-Pro venv_projects % python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
/Users/user/.pyenv/versions/3.8.5/lib/python3.8/site-packages
user@users-MacBook-Pro venv_projects % mkdir test_project_3
user@users-MacBook-Pro venv_projects % cd test_project_3
user@users-MacBook-Pro test_project_3 % pyenv local 3.5.1
user@users-MacBook-Pro test_project_3 % python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
/Users/user/.pyenv/versions/3.5.1/lib/python3.5/site-packages
user@users-MacBook-Pro test_project_3 % pyenv virtualenv 3.5.1 test_project_3
Ignoring indexes: https://pypi.python.org/simple
Requirement already satisfied (use --upgrade to upgrade): setuptools in /Users/user/.pyenv/versions/3.5.1/envs/test_project_3/lib/python3.5/site-packages
Requirement already satisfied (use --upgrade to upgrade): pip in /Users/user/.pyenv/versions/3.5.1/envs/test_project_3/lib/python3.5/site-packages
user@users-MacBook-Pro test_project_3 % pyenv local test_project_3
user@users-MacBook-Pro test_project_3 % python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
/Users/user/.pyenv/versions/test_project_3/lib/python3.5/site-packages
user@users-MacBook-Pro test_project_3 %
On line 3 we see that since we’re out of the project directory we are once again using the global version we have set of 3.8.5. Now we make a new project directory called test_project_3 and inside of it we set the local version to 3.5.1. When we check again on line 8 – we see that we’re using the site packages folder that’s part of the base 3.5.1 version directory. Then we create a venv with PyEnv and set that as the local Python “version”. In doing so we can see on line 15, we are now using that site-packages folder.
So at this point – I think we have close to a full picture of how this is working. The bulk of the magic is in how PyEnv manages versions and the actual package management piece is a function of Python and just works the way that it does because of the PyEnv folder structure. My hypothesis is that by using PyEnv version management (and the shim model) you’re getting “virtual environment like” behavior without having to actually “activate” a virtual environment.
Now that said – there are some limitations to this. While discussing this on the Network to Code slack Jacob McGill (thank you for the comments!) pointed out one limitation I hadn’t considered. My model of laying out projects fit wells into PyEnv since I am typically working on files for a given project in that given project folder. But without “activating” the virtual environment, you HAVE to be in the project directory in order for any of this to work. In other words, once I leave the project folder (for instance test_project_1) I lose all of this. Now in my current work flow, this isn’t a problem since I tightly couple the project folder to the work Im doing. But I can see this being a drawback for others. As an example….
user@users-MacBook-Pro test_project_1 % pyenv activate test_project_1
pyenv-virtualenv: prompt changing will be removed from future release. configure `export PYENV_VIRTUALENV_DISABLE_PROMPT=1' to simulate the behavior.
(test_project_1) user@users-MacBook-Pro test_project_1 %
(test_project_1) user@users-MacBook-Pro test_project_1 % cd ..
(test_project_1) user@users-MacBook-Pro venv_projects % pip list
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package Version
---------- -------
pip 20.2.2
PyYAML 5.3.1
setuptools 44.1.1
wheel 0.35.1
WARNING: You are using pip version 20.2.2; however, version 20.2.3 is available.
You should consider upgrading via the '/Users/user/.pyenv/versions/2.7.18/envs/test_project_1/bin/python -m pip install --upgrade pip' command.
(test_project_1) user@users-MacBook-Pro venv_projects % cd test_project_3
(test_project_1) user@users-MacBook-Pro test_project_3 % pip list
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package Version
---------- -------
pip 20.2.2
PyYAML 5.3.1
setuptools 44.1.1
wheel 0.35.1
WARNING: You are using pip version 20.2.2; however, version 20.2.3 is available.
You should consider upgrading via the '/Users/user/.pyenv/versions/2.7.18/envs/test_project_1/bin/python -m pip install --upgrade pip' command.
(test_project_1) user@users-MacBook-Pro test_project_3 %
You can see that if I do “activate” the venv then I can move out of the project folder and still have my Python version and packages that I expect. I can even move into another project folder and I’m still using the activated virtual environment.
This all said – I would love to hear from Python/PyEnv experts out there on if my hypothesis is true. Please comment and let me know your thoughts! At this point, if I do end up using PyEnv for virtual environment management, I don’t think I’ll be doing any virtual environment activations (unless someone can point out a good reason to do so!).
Do you see any clear advantage of pyenv over viirtualenv or venv ? I don’t.
https://stackoverflow.com/questions/41573587/what-is-the-difference-between-venv-pyvenv-pyenv-virtualenv-virtualenvwrappe
What’s more, pyenv was deprecated from the standard library as of 3.6 https://docs.python.org/3/library/venv.html
Thanks for the comment! At this stage in the game Im just taking a look at a series of tools to see which ones I might work into my own pipeline. To be completely honest, I didn’t expect to see the behavior I experienced with PyEnv and virtual environments but I sure thought it was interesting. It certainly made me wonder why folks would even need to use virtual environments given the way that PyEnv handles versions and their use. Do you agree that it seems that PyEnvs base behavior is providing a virtual environment like experience without actually using them?
Thanks for the links! The first one was a good read for sure and I’ll admit that I for sure need to spend more time looking at virtual environment tooling especially as it relates to later versions of 3.x Python. Expect more posts on the topic shortly 🙂 What are your thoughts on the tools? Which ones do you use?
In regards to your last comment, do you mean that `pyvenv` was deprecated? (you said pyenv so just want to make sure). I’ll for sure spend some more time looking into other virtual environment tooling and let you know what I think (likely in the form of another blog post). Stay tuned!
Thanks for the reply.
According to the 2nd link above from python.org, PyEnv was indeed Deprecated:
> pyvenv was the recommended tool for creating virtual environments for Python 3.3 and 3.4, and is deprecated in Python 3.6.
Never tried it, hence cannot comment on it.
virtualenv is my current small python project development “isolation” goto solution. Also keeping an eye on the venv module for plan B.
For larger projects, lots of folks prefer containers. I agree with those folks.
My 2 cents. Python package management seems to take more work compared to other languages, eg java and javascript.
You are mixing pyEnv with pyVenv… pyVenv was incorporated into the standard library and is now called simply “venv”.
venv is very similar to virtualenv, it allows you to have a isolated site-packages environment for each project so you don’t have to worry with conflicting libraries. Imagine you are building a project using Django and another using Flask, and both require Requests.
Now imagine a new version of Requests have been release which breaks compatibility with the previous version, and Flask updated their code to work with the new version, but Django hasn’t yet.
You need both versions of Requests, one for each project, but you can’t have both on the global site-packages because they use the same module name.
There are other advantages of using a Python virtual environment too, but this is the simplest to grasp.
pyEnv allows you to manage several Python versions on your system.
Say you need to test your library on Python 3.6, 3.7 and 3.8. Most package management tools will only allow you to have a single version of the python3 package, pyenv manages that for you by deploying individual python distributions isolated from the main system python.
There is also pipx, which allows you to have a virtualenv for each python application you want to run, so you don’t have to “pip install ansible” and end up polluting your system site-packages.
It’s a bit confusing, and other languages manage this arguably better (e.g. Node), but once you get your head around it, it’s actually quite useful.
Obligatory XKCD reference:
https://xkcd.com/1987/
Gotcha.
R.I.P. pyVenv and long live the venv standard for Virtual Environment Management
OTOH, pyEnv does for python, what nvm does for js, what jdkman does for java: SDK Version Management.