Sysconfig merge #2

frenzymadness · 2020-10-20T08:07:01Z

No description provided.

frenzymadness · 2020-10-21T14:41:08Z

Well, this seems to be almost complete and I'd like to merge it before I get to other parts of distutils we need to merge to sysconfig.

In points:

A lot of functions or global variables are the same so an import from sysconfig to distutils.sysconfig is all we need.
Sometimes, a function has a different name or slightly different API which is also solved without big issues.
One function (_parse_makefile) has a small difference so I've implemented a compatibility switch.

My plan was to not touch tests at all to prove that the change of the deprecated module has no impact on its functionality but this plan failed. The main reason is that sysconfig uses _CONFIG_VARS global dictionary and distutils.sysconfig has _config_vars. The different names are not an issue here. The issue is that sysconfig._CONFIG_VARS is guarded by saved_test_environment but distutils.sysconfig._config_vars is not. If a test previously left some changes in distutils.sysconfig._config_vars, it had no effect on the result but now after the merge, it caused a lot of troubles. As a result, I had to implement or adjust setUp/tearDown methods for the tests which are (in)directly manipulating with the global dictionary. I am not always using the same names in those methods to make my modifications look similar to existing ones.

Another change in a test (test_venv) is related to a fact that pip uses distutils.sysconfig so we have to ignore deprecation warnings where we run ensurepip via subprocess. Later, I'll send a PR to pip to fix this.

What I think is left to be done:

document somehow that distutils.sysconfig is deprecated
add documentation for new functions in sysconfig module - copy/move it from distutils.sysconfig docs
document new kw argument in _parse_makefile.

Cc: @vstinner @encukou @hroncok

hroncok · 2020-10-21T15:32:56Z

The description is awesome. Plese let the PEP discussion know that you are working on this (even if you are not yet ready to share the code).

Lib/sysconfig.py

zooba

Thanks so much for working on this!

Lib/sysconfig.py

Lib/test/support/__init__.py

Lib/distutils/sysconfig.py

zooba · 2020-10-22T09:52:10Z

Lib/distutils/sysconfig.py

+from sysconfig import get_python_version
+from sysconfig import get_python_lib
+
+if os.name == "nt":


Strictly, sys.platform == "win32" is the correct check for this case (sys.platform is "how it was built", while os.name is "which POSIX emulation are we using" - they just happen to line up right now ;) )

The functions _fix_pcbuild is defined in sysconfig only if os.name == 'nt' so I can either keep the condition as is or change it in both modules. What do you think would be better?

Long term I'd rather it was changed in sysconfig. I don't mind what happens in distutils.sysconfig.

os.name == "nt" is used in sysconfig in many places so I'd rather keep it as TODO.

merwok · 2020-10-22T15:00:00Z

Lib/sysconfig.py

    """Parse a Makefile-style file.

    A dictionary containing name/value pairs is returned.  If an
    optional dictionary is passed in as the second argument, it is
    used instead of a new dictionary.
    """
-    # Regexes needed for parsing Makefile (and similar syntaxes,
-    # like old-style Setup files).
-    import re


It’s possible that re was imported in this function rather than at module scope because building CPython (and the _sre modules) needs sysconfig.

Ok, I'll add try/except to handle situation when re module won't be available. Or do you have a better idea?

That may not work if the problem is circular imports. Better to just not use re until it's needed (and there's probably nothing gained from re.compile here anyway; the regular re methods will cache well enough).

That would mean import re and define some of the regexes in three places:

expand_makefile_vars function in sysconfig

_parse_makefile function in sysconfig

globally in distutils.sysconfig because they're imported from there in read_setup_file function in distutils.extension (not future proof)

I'd rather keep them in one place. I've checked and there seems to be no circular import. Do you think there might be a problem which the fresh builds in CI cannot spot?

merwok · 2020-10-22T20:37:56Z

FTR I found some tickets about differences between the two modules:

frenzymadness · 2020-10-23T11:20:10Z

FTR I found some tickets about differences between the two modules:
* https://bugs.python.org/issue39825

* https://bugs.python.org/issue24908

Thank you. Both seem to be better fixed in sysconfig and the first one already has a PR which does exactly that. I'll follow both of them.

frenzymadness · 2020-10-23T11:42:47Z

Doc status:

distutils.sysconfig is marked as deprecated
docs for get_python_inc and get_python_lib is copied to sysconfig docs
there is still unresolved discussion about what to do with customize_compiler function so I'm keeping its doc where it is now.
the function with changed API is private so there is no documentation for it.

frenzymadness · 2020-11-03T11:09:09Z

My plan is to get back to this now and open an upstream PR once I solve the problem on macOS.

…ONFIG_VARS might be modified

This reverts commit fef432d1b75d98bb68ec98dd92d3de1aacca437c.

…fig to sysconfig

…edtogether with the module.

frenzymadness · 2020-11-04T11:29:53Z

Upstream PR: python#23142

frenzymadness force-pushed the sysconfig_merge branch 7 times, most recently from 0538371 to da88e63 Compare October 21, 2020 12:11

mattip reviewed Oct 22, 2020

View reviewed changes

Lib/sysconfig.py Show resolved Hide resolved

zooba reviewed Oct 22, 2020

View reviewed changes

merwok reviewed Oct 22, 2020

View reviewed changes

frenzymadness force-pushed the sysconfig_merge branch 2 times, most recently from d9f9d92 to 182712e Compare October 23, 2020 11:10

frenzymadness added 12 commits November 3, 2020 14:25

Mark distutils.sysconfig as deprecated

4b481f5

Import global paths from sysconfig module

19a1b57

Move some similar functions from distutils.sysconfig -> sysconfig

2843e92

Move the rest of global variables

e623edd

Move the rest of functions to sysconfig

248599e

Import regexes used in tests

524484e

Implement compatibility option for parse_makefile

f8ecf9a

Do not use assignment in teardown to not lose reference to global dict

59b37be

Ignore DeprecationWarning in pip code in ensurepip module

ece8382

Improve setUp/tearDown of UnixCCompilerTestCase

9c396d7

Fix test.support to use sysconfig directly

4e67344

Improve or create setUp/tearDown methods for tests where sysconfig._C…

8f7c1c1

…ONFIG_VARS might be modified

frenzymadness added 9 commits November 3, 2020 14:25

fixup! Improve setUp/tearDown of UnixCCompilerTestCase

a6f918f

Rename distutils_compat to keep_unresolved and reverse the logic

66e3b96

Revert "Fix test.support to use sysconfig directly"

357e091

This reverts commit fef432d1b75d98bb68ec98dd92d3de1aacca437c.

Merge all import from sysconfig to one statement

463dc78

Allow import of sysconfig when re module is not available

156c05e

Add a TODO comment about PEP 632

bcefaa6

Mark distutils.sysconfig as deprecated in docs

fc86499

Copy docs for get_python_inc and get_python_lib from distutils.syscon…

a3e6497

…fig to sysconfig

Move customize_compiler back to distutils.sysconfig as it'll be remov…

baa8258

…edtogether with the module.

frenzymadness force-pushed the sysconfig_merge branch 2 times, most recently from 74ff716 to 6d29b50 Compare November 3, 2020 13:34

Fix import of not-yet-initialized _CONFIG_VARS

a2d8e4d

frenzymadness force-pushed the sysconfig_merge branch from 6d29b50 to a2d8e4d Compare November 3, 2020 13:44

frenzymadness mentioned this pull request Nov 4, 2020

bpo-41282: (PEP 632) Deprecate distutils.sysconfig (partial implementation of the PEP) python/cpython#23142

Merged

frenzymadness closed this Nov 4, 2020

Sysconfig merge #2

Sysconfig merge #2

Uh oh!

Conversation

frenzymadness commented Oct 20, 2020

Uh oh!

frenzymadness commented Oct 21, 2020

Uh oh!

hroncok commented Oct 21, 2020

Uh oh!

Uh oh!

zooba left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

merwok commented Oct 22, 2020

Uh oh!

frenzymadness commented Oct 23, 2020

Uh oh!

frenzymadness commented Oct 23, 2020

Uh oh!

frenzymadness commented Nov 3, 2020

Uh oh!

frenzymadness commented Nov 4, 2020

Uh oh!

Uh oh!