pip imports the deprecated distutils module

[vstinner]

Description

The pip/_internal/locations/_distutils.py file imports the distutils module:

pip/src/pip/_internal/locations/_distutils.py

Lines 9 to 12 in cb24fb4

	from distutils.cmd import Command as DistutilsCommand
	from distutils.command.install import SCHEME_KEYS
	from distutils.command.install import install as distutils_install_command
	from distutils.sysconfig import get_python_lib

This module is deprecated since Python 3.10 and is going to be removed in Python 3.12: python/cpython#92584 (comment)

Expected behavior

No response

pip version

22.0.4

Python version

3.12.0a0

OS

Linux

How to Reproduce

1. Build Python 3.12 with [WIP] gh-92584: Rename the distutils package to _distutils python/cpython#92585 PR
2. Create a venv without pip: ./python -m venv env
3. Install pip: ./env/bin/python -m ensurepip -v

Output

vstinner@apu$ ./env/bin/python -m ensurepip -v
Traceback (most recent call last):
  File "<string>", line 6, in <module>
  File "/home/vstinner/python/main/Lib/runpy.py", line 226, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/runpy.py", line 98, in _run_module_code
    _run_code(code, mod_globals, init_globals,
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/runpy.py", line 88, in _run_code
    exec(code, run_globals)
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmps2pazori/pip-22.0.4-py3-none-any.whl/pip/__main__.py", line 29, in <module>
  File "/tmp/tmps2pazori/pip-22.0.4-py3-none-any.whl/pip/_internal/cli/main.py", line 9, in <module>
  File "/tmp/tmps2pazori/pip-22.0.4-py3-none-any.whl/pip/_internal/cli/autocompletion.py", line 10, in <module>
  File "/tmp/tmps2pazori/pip-22.0.4-py3-none-any.whl/pip/_internal/cli/main_parser.py", line 8, in <module>
  File "/tmp/tmps2pazori/pip-22.0.4-py3-none-any.whl/pip/_internal/cli/cmdoptions.py", line 23, in <module>
  File "/tmp/tmps2pazori/pip-22.0.4-py3-none-any.whl/pip/_internal/cli/parser.py", line 12, in <module>
  File "/tmp/tmps2pazori/pip-22.0.4-py3-none-any.whl/pip/_internal/configuration.py", line 26, in <module>
  File "/tmp/tmps2pazori/pip-22.0.4-py3-none-any.whl/pip/_internal/utils/logging.py", line 27, in <module>
  File "/tmp/tmps2pazori/pip-22.0.4-py3-none-any.whl/pip/_internal/utils/misc.py", line 39, in <module>
  File "/tmp/tmps2pazori/pip-22.0.4-py3-none-any.whl/pip/_internal/locations/__init__.py", line 14, in <module>
  File "/tmp/tmps2pazori/pip-22.0.4-py3-none-any.whl/pip/_internal/locations/_distutils.py", line 9, in <module>
ModuleNotFoundError: No module named 'distutils'
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/runpy.py", line 88, in _run_code
    exec(code, run_globals)
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/ensurepip/__main__.py", line 5, in <module>
    sys.exit(ensurepip._main())
             ^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/ensurepip/__init__.py", line 276, in _main
    return _bootstrap(
           ^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/ensurepip/__init__.py", line 192, in _bootstrap
    return _run_pip([*args, *_PACKAGE_NAMES], additional_paths)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/ensurepip/__init__.py", line 92, in _run_pip
    return subprocess.run([sys.executable, '-W', 'ignore::DeprecationWarning',
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/subprocess.py", line 558, in run
    raise CalledProcessError(retcode, process.args,
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
subprocess.CalledProcessError: Command '['/home/vstinner/python/main/env/bin/python', '-W', 'ignore::DeprecationWarning', '-c', '\nimport runpy\nimport sys\nsys.path = [\'/tmp/tmps2pazori/setuptools-58.1.0-py3-none-any.whl\', \'/tmp/tmps2pazori/pip-22.0.4-py3-none-any.whl\'] + sys.path\nsys.argv[1:] = [\'install\', \'--no-cache-dir\', \'--no-index\', \'--find-links\', \'/tmp/tmps2pazori\', \'-v\', \'setuptools\', \'pip\']\nrunpy.run_module("pip", run_name="__main__", alter_sys=True)\n']' returned non-zero exit status 1.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[uranusjr]

This module is provided for backward compatibility; on 3.10 it should never be used. So I guess (assuming there's no real bug where we still use it) we simply need to do conditional imports better.

[vstinner]

Maybe only import locations/_distutils.py in locations/__init__.py if _USE_SYSCONFIG is false?

[uranusjr]

I looked at the code and pip in fact still relies on some distutils internals beside the locations module. I’ll try to either remove them, or at least localise so distutils is imported only when relevant functionalities are accessed (the most significant usage of it is to parse --install-option).

[vstinner]

Hi, is there an update on this issue? How can I help?

[pradyunsg]

No updates (since those would be reflected on this issue).

Looking at this now... There's 5 major files that this needs changes for:

• src/pip/_internal/locations/init.py: this needs a conditional import of pip._internal.locations._distutils
• src/pip/_internal/locations/_distutils.py: well... this should be imported conditionally and never imported on 3.12+.
• src/pip/_internal/locations/_sysconfig.py: this needs change_root from distutils.util
• src/pip/_internal/operations/install/legacy.py: this needs change_root from distutils.util
• src/pip/_internal/utils/distutils_args.py: this uses FancyGetopt from distutils to parse out distutils arguments.

Help would be appreciated with all of these. There's one import/use of distutils in a vendored package -- for distilb.get_archive_formats -- and that code path shouldn't be executed in the real world (that function is not used by pip, AFAIK).