3.11 support

[vfazio]

Issue

On python 3.11+, virtualenv returns an invalid system_executable for python binaries out of virtual environments made with --copies

See:

python-poetry/poetry#6940 (comment)

Part of this is due to a change in 3.11 that changes the value of sys._base_executable to substitute the value of the "home" key from pyvenv.cfg.

https://bugs.python.org/issue46028
python/cpython#29041
python/cpython#30144

On it's own, Virtualenv is setting this to a path that does not actually include the python binary so the system_executable is "wrong"

root@vfazio2:/tmp/tmp.U2ZVtBvloJ# python3 -m virtualenv --copies .venv2
created virtual environment CPython3.11.0.final.0-64 in 1164ms
  creator CPython3Posix(dest=/tmp/tmp.U2ZVtBvloJ/.venv2, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==22.3, setuptools==65.5.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

root@vfazio2:/tmp/tmp.U2ZVtBvloJ# ./.venv2/bin/python3 /usr/local/lib/python3.11/site-packages/virtualenv/discovery/py_info.py | jq .system_executable
"/usr/local/python3"

root@vfazio2:/tmp/tmp.U2ZVtBvloJ# test -e /usr/local/python3; echo $?
1

root@vfazio2:/tmp/tmp.U2ZVtBvloJ# ./.venv2/bin/python /usr/local/lib/python3.11/site-packages/virtualenv/discovery/py_info.py | jq .system_executable
"/usr/local/python"
root@vfazio2:/tmp/tmp.U2ZVtBvloJ# test -e /usr/local/python ; echo $?
1

For virtual environments created from a venv parent, subsequent virtualenvs continue to return a binary that does not exist. Poetry is a good example:

root@vfazio2:/tmp/tmp.U2ZVtBvloJ# cat /opt/poetry/venv/pyvenv.cfg 
home = /usr/local/bin
include-system-site-packages = false
version = 3.11.0
executable = /usr/local/bin/python3.11
command = /usr/local/bin/python3 -m venv --copies --clear /opt/poetry/venv

root@vfazio2:/tmp/tmp.U2ZVtBvloJ# /opt/poetry/venv/bin/python /usr/local/lib/python3.11/site-packages/virtualenv/discovery/py_info.py | jq .system_executable
"/usr/local/bin/python"

root@vfazio2:/tmp/tmp.U2ZVtBvloJ# test -e /usr/local/bin/python; echo $?
1

This causes virtualenv to fail to create virtual environments due to not being able to find the system executable

  RuntimeError

  failed to query /usr/local/bin/python with code 2 err: 'No such file or directory'

  at /opt/poetry/venv/lib/python3.11/site-packages/virtualenv/discovery/cached_py_info.py:31 in from_exe
       27│     env = os.environ if env is None else env
       28│     result = _get_from_cache(cls, app_data, exe, env, ignore_cache=ignore_cache)
       29│     if isinstance(result, Exception):
       30│         if raise_on_error:
    →  31│             raise result
       32│         else:
       33│             logging.info("%s", result)
       34│         result = None
       35│     return result

Stack:

root@vfazio2:/tmp/tmp.U2ZVtBvloJ# /opt/poetry/venv/bin/python
Python 3.11.0 (main, Nov  2 2022, 16:46:02) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import virtualenv
>>> virtualenv.cli_run("tempvenv")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/poetry/venv/lib/python3.11/site-packages/virtualenv/run/__init__.py", line 28, in cli_run
    of_session = session_via_cli(args, options, setup_logging, env)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/poetry/venv/lib/python3.11/site-packages/virtualenv/run/__init__.py", line 46, in session_via_cli
    parser, elements = build_parser(args, options, setup_logging, env)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/poetry/venv/lib/python3.11/site-packages/virtualenv/run/__init__.py", line 68, in build_parser
    parser._interpreter = interpreter = discover.interpreter
                                        ^^^^^^^^^^^^^^^^^^^^
  File "/opt/poetry/venv/lib/python3.11/site-packages/virtualenv/discovery/discover.py", line 38, in interpreter
    self._interpreter = self.run()
                        ^^^^^^^^^^
  File "/opt/poetry/venv/lib/python3.11/site-packages/virtualenv/discovery/builtin.py", line 44, in run
    result = get_interpreter(python_spec, self.try_first_with, self.app_data, self._env)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/poetry/venv/lib/python3.11/site-packages/virtualenv/discovery/builtin.py", line 59, in get_interpreter
    for interpreter, impl_must_match in propose_interpreters(spec, try_first_with, app_data, env):
  File "/opt/poetry/venv/lib/python3.11/site-packages/virtualenv/discovery/builtin.py", line 90, in propose_interpreters
    yield PythonInfo.from_exe(os.path.abspath(spec.path), app_data, env=env), True
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/poetry/venv/lib/python3.11/site-packages/virtualenv/discovery/py_info.py", line 372, in from_exe
    raise exception
  File "/opt/poetry/venv/lib/python3.11/site-packages/virtualenv/discovery/py_info.py", line 369, in from_exe
    proposed = proposed._resolve_to_system(app_data, proposed)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/poetry/venv/lib/python3.11/site-packages/virtualenv/discovery/py_info.py", line 409, in _resolve_to_system
    target = cls.from_exe(target.system_executable, app_data)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/poetry/venv/lib/python3.11/site-packages/virtualenv/discovery/py_info.py", line 365, in from_exe
    proposed = from_exe(cls, app_data, exe, env=env, raise_on_error=raise_on_error, ignore_cache=ignore_cache)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/poetry/venv/lib/python3.11/site-packages/virtualenv/discovery/cached_py_info.py", line 31, in from_exe
    raise result
RuntimeError: failed to query /usr/local/bin/python with code 2 err: 'No such file or directory'

I've proposed a fix in upstream cpython to try to paper over this discrepancy between binaries available in a venv and what's available in the system install path. I don't expect that to move quickly. Virtualenv could, in the interim, catch this known problematic scenario and return a system python that does exist.

[gaborbernat]

PR welcome.

[vfazio]

I'm not totally sure virtualenv is doing it wrong seeing as how this has been consistent for previous 3.x versions up until 3.11 and the reworking of Modules/getpath.py

I'm not well versed enough in virtualenv to make real suggestions,.. I assume we're intentionally searching for the executable one level higher than current?

[vfazio]

@gaborbernat One option i was considering was tweaking _fast_get_system_executable. I have a working proof-of-concept below:

diff --git a/src/virtualenv/discovery/py_info.py b/src/virtualenv/discovery/py_info.py
index 7c680ee..755773c 100644
--- a/src/virtualenv/discovery/py_info.py
+++ b/src/virtualenv/discovery/py_info.py
@@ -138,6 +138,19 @@ class PythonInfo(object):
                 base_executable = getattr(sys, "_base_executable", None)  # some platforms may set this to help us
                 if base_executable is not None:  # use the saved system executable if present
                     if sys.executable != base_executable:  # we know we're in a virtual environment, cannot be us
+                        # Do light validation for non-Windows virtual environments since some installs/distributions
+                        # do not provide a version-less "python" binary, so try to fallback to an alternative
+                        # This should still be relatively fast since it's just a couple of `stat` calls
+                        if self.os != "nt" and (self.version_info.major, self.version_info.minor) >= (3, 11):
+                            if not os.path.exists(base_executable):
+                                major, minor = self.version_info.major, self.version_info.minor
+                                base_dir = os.path.dirname(base_executable)
+                                for candidate in [
+                                    os.path.join(base_dir, f"python{v}") for v in (major, f"{major}.{minor}")
+                                ]:
+                                    if os.path.exists(candidate):
+                                        base_executable = candidate
+                                        break
                         return base_executable

This essentially mimics the logic i put in my PR to cpython to fix this issue, but the chances of that getting accepted seems low and if it was, the timeline is unknown. Even if it is, there's a gap for 3.11.0 where a non-existent binary will be returned from system_executable because venv and virtualenv provide "python" but installations using make install and debian based distros don't provide "python" in the install path by default (PEP 394).

running a couple of os.stat calls seems still fast to me and this is still conditionalized for the affected platform and versions so it shouldn't be hit too often.

Does this seem like a reasonable interim fix until there's something in cpython?

[gaborbernat]

Can you show what the diff would be to the current state?

[vfazio]

Can you show what the diff would be to the current state?

I've updated the snippet to be a diff

In terms of results, here's what it would look like:

installed python from sources:

(virtualenv-3.10) vfazio@vfazio2 ~/development/virtualenv $ ls -la /tmp/tmp.DWGY1NYz88/usr/local/bin/python*
lrwxrwxrwx 1 vfazio domain users       10 Nov  7 12:04 /tmp/tmp.DWGY1NYz88/usr/local/bin/python3 -> python3.12
lrwxrwxrwx 1 vfazio domain users       17 Nov  7 12:04 /tmp/tmp.DWGY1NYz88/usr/local/bin/python3-config -> python3.12-config
-rwxr-xr-x 1 vfazio domain users 35408008 Nov  7 12:04 /tmp/tmp.DWGY1NYz88/usr/local/bin/python3.12
-rwxr-xr-x 1 vfazio domain users     3047 Nov  7 12:04 /tmp/tmp.DWGY1NYz88/usr/local/bin/python3.12-config

make a venv:

/tmp/tmp.DWGY1NYz88/usr/local/bin/python3 -m venv --copies /tmp/tmp.IM5F9abIoz/

before the fix to py_info:

(virtualenv-3.10) vfazio@vfazio2 ~/development/virtualenv $ /tmp/tmp.IM5F9abIoz/bin/python /mnt/development/virtualenv/src/virtualenv/discovery/py_info.py | jq .system_executable
"/tmp/tmp.DWGY1NYz88/usr/local/bin/python"

Note that "python" is not available in the system installation.

after fix:

(virtualenv-3.10) vfazio@vfazio2 ~/development/virtualenv $ /tmp/tmp.IM5F9abIoz/bin/python /mnt/development/virtualenv/src/virtualenv/discovery/py_info.py | jq .system_executable
"/tmp/tmp.DWGY1NYz88/usr/local/bin/python3"

The result now maps to a legitimate binary available in the directory pointed at by "home" from pyvenv.cfg.

For situations where "home" is incorrect (fixed in my PR), there will be no change in behavior, there are just 3 extra os.stat calls

[gaborbernat]

Fill in the PR seems reasonable, with some small changes I can comment on the PR.

[vfazio]

Fill in the PR seems reasonable, with some small changes I can comment on the PR.

#2442

[vfazio]

Upstream is considering my PR for the fix in main. I have no idea if it will be backported to 3.11 at this point.

[vfazio]

@gaborbernat thanks!

is there a way to get releases pushed to bootstrap? https://bootstrap.pypa.io/virtualenv/