Issue 32444: python -m venv symlink dependency on how python binary is called is not documented (original) (raw)

The venv module does not evenly create the necessary binaries/symlinks when building virtualenvs. If you call venv like this:

seliger@core:~/venvs$ python3 -m venv venvA

seliger@core:~/venvs$ ls -l venvA/bin/ total 32 -rw-r--r-- 1 seliger seliger 2143 Dec 28 21:29 activate -rw-r--r-- 1 seliger seliger 1259 Dec 28 21:29 activate.csh -rw-r--r-- 1 seliger seliger 2397 Dec 28 21:29 activate.fish -rwxrwxr-x 1 seliger seliger 254 Dec 28 21:29 easy_install -rwxrwxr-x 1 seliger seliger 254 Dec 28 21:29 easy_install-3.5 -rwxrwxr-x 1 seliger seliger 226 Dec 28 21:29 pip -rwxrwxr-x 1 seliger seliger 226 Dec 28 21:29 pip3 -rwxrwxr-x 1 seliger seliger 226 Dec 28 21:29 pip3.5 lrwxrwxrwx 1 seliger seliger 7 Dec 28 21:29 python -> python3 lrwxrwxrwx 1 seliger seliger 16 Dec 28 21:29 python3 -> /usr/bin/python3

...you do not end up with a "python3.5" binary.

However, if you call venv like this:

seliger@core:~/venvs$ python3.5 -m venv venvB

seliger@core:~/venvs$ ls -l venvB/bin total 32 -rw-r--r-- 1 seliger seliger 2143 Dec 28 21:29 activate -rw-r--r-- 1 seliger seliger 1259 Dec 28 21:29 activate.csh -rw-r--r-- 1 seliger seliger 2397 Dec 28 21:29 activate.fish -rwxrwxr-x 1 seliger seliger 256 Dec 28 21:29 easy_install -rwxrwxr-x 1 seliger seliger 256 Dec 28 21:29 easy_install-3.5 -rwxrwxr-x 1 seliger seliger 228 Dec 28 21:29 pip -rwxrwxr-x 1 seliger seliger 228 Dec 28 21:29 pip3 -rwxrwxr-x 1 seliger seliger 228 Dec 28 21:29 pip3.5 lrwxrwxrwx 1 seliger seliger 9 Dec 28 21:29 python -> python3.5 lrwxrwxrwx 1 seliger seliger 9 Dec 28 21:29 python3 -> python3.5 lrwxrwxrwx 1 seliger seliger 18 Dec 28 21:29 python3.5 -> /usr/bin/python3.5

...you DO get the necessary python3.5 binary. Some vendors are making it a requirement to call a specific pythonX.Y binary to ensure compatibility. One such example is the serverless framework when deploying to Amazon Web Services. Another example is the useful pyenv utility that manages full Python builds and virtualenvs. When it depends upon venv, it exhibits the same behavior. I submitted a patch workaround to force calling venv using pythonX.Y, but this really seems like an issue with the venv module itself.

The expected behavior should be that venv generates all three binaries (python, python3, and python3.5) regardless of how the python command was invoked.

I am able to reproduce this on Python 3.5 and 3.6. I could not find any other similar references in searching the bug system.

Here is the write-up I posted over on the pyenv issues page:

This is apparently a behavior issue in the way venv works. Not only does this impact 3.6.x, I also confirmed that the same issue occurs with venv in general using the stock 3.5 Python that comes with Ubuntu 16.04.

The trouble is at Lib/venv/init.py:111. Starting at line 111:

    if sys.platform == 'darwin' and '__PYVENV_LAUNCHER__' in env:
        executable = os.environ['__PYVENV_LAUNCHER__']
    else:
        executable = sys.executable
    dirname, exename = os.path.split(os.path.abspath(executable))
    context.executable = executable

This carries down to the setup_python method starting at line 187 (Lib/venv/init.py:187). It takes whatever executable that is called as the source, and then creates the others based on a hard coded list (e.g. python and python3).

def setup_python(self, context):
    """
    Set up a Python executable in the environment.
    :param context: The information for the environment creation request
                    being processed.
    """
    binpath = context.bin_path
    path = context.env_exe
    copier = self.symlink_or_copy
    copier(context.executable, path)
    dirname = context.python_dir
    if os.name != 'nt':
        if not os.path.islink(path):
            os.chmod(path, 0o755)
        for suffix in ('python', 'python3'):
            path = os.path.join(binpath, suffix)
            if not os.path.exists(path):
                # [Issue 18807](issue18807 "[closed] Allow venv to create copies, even when symlinks are supported"): make copies if
                # symlinks are not wanted
                copier(context.env_exe, path, relative_symlinks_ok=True)
                if not os.path.islink(path):
                    os.chmod(path, 0o755)
    else:
        subdir = 'DLLs'
        include = self.include_binary
        files = [f for f in os.listdir(dirname) if include(f)]
        for f in files:
            src = os.path.join(dirname, f)
            dst = os.path.join(binpath, f)
            if dst != context.env_exe:  # already done, above
                copier(src, dst)
        dirname = os.path.join(dirname, subdir)
        if os.path.isdir(dirname):
            files = [f for f in os.listdir(dirname) if include(f)]
            for f in files:
                src = os.path.join(dirname, f)
                dst = os.path.join(binpath, f)
                copier(src, dst)
        # copy init.tcl over
        for root, dirs, files in os.walk(context.python_dir):
            if 'init.tcl' in files:
                tcldir = os.path.basename(root)
                tcldir = os.path.join(context.env_dir, 'Lib', tcldir)
                if not os.path.exists(tcldir):
                    os.makedirs(tcldir)
                src = os.path.join(root, 'init.tcl')
                dst = os.path.join(tcldir, 'init.tcl')
                shutil.copyfile(src, dst)
                break

If you don't call -m venv as python3.X, you will never get that binary in the bin directory.

This is a feature that actually supports your use case, as well as the use cases of those who don't want to strap the python version: you get what you ask for. If you call venv with 'python3', you get a venv that will use your new python if you upgrade your python, but if you call it with python3.x, you will get a venv that will not upgrade, which is exactly what you want for your use case.