From 5e9e27461685738b237bd47538974765f9c7bbba Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 20 Mar 2024 15:48:13 -0700 Subject: [PATCH 01/55] Write up to the specification --- peps/pep-9999.rst | 158 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 158 insertions(+) create mode 100644 peps/pep-9999.rst diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst new file mode 100644 index 00000000000..1c41df63cf8 --- /dev/null +++ b/peps/pep-9999.rst @@ -0,0 +1,158 @@ +PEP: +Title: A file format to list Python dependencies for installation reproducibility +Author: Brett Cannon +PEP-Delegate: +Discussions-To: +Status: Draft +Type: Standards Track +Topic: Packaging +Created: +Post-History: +Replaces: 665 + +Abstract +======== + +[A short (~200 word) description of the technical issue being addressed.] + +This PEP proposes a new file format for specifying what dependencies to +(possibly) install into a Python environment for consitent installation +repoducibility. The format is designed to be human-readable but +machine-generated. Installers consuming the file should also not need to resolve +which listed dependencies to install, but instead evaluate each dependency in +question in isolation. + + +Motivation +========== + +Currently, there is no standard way to specify what top-level dependencies one +would like to see installed into a Python environment and then subsequently +record what war eventually installed in a file (called a *lock file*). +Considering there are four well-known solutions to this problem in the +community (``pip freeze``, pip-tools_, Poetry_, and PDM_), there seems to be an +appetite for lock files in general. + +Those tools also vary in what locking scenarios they approach. For instance, +``pip freeze`` and pip-tools only generate lock files for the current +environment while PDM and Poetry try to lock for *any* environment to some +degree. And none of them directly support locking to specific files to install +which can be important for some workflows. There's also concerns around secure +defaults in the face of supply chain attacks. Finally, not all the formats are +easy to audit to determine what what would be installed into an environment +ahead of time. + +Ghe lack of a standard also has some drawbacks. For instance, any tooling that +wants to work with lock files must choose which format to support, potentially +leaving users unsupported (e.g. if Dependabot_ chose not to support PDM. +support by cloud providers who can do dependency installations on your behalf, +etc.). + + +Rationale +========= + +[Describe why particular design decisions were made.] + +The file format is designed to be human-readable. This is +so that the contents of the file can be audited by a human to make sure no +undesired dependencies end up being included in the lock file. It is also to +facilitate understanding what would be installed if the lock file were to be +used without necessitating running a tool, once again to help with auditing. + +The file format is also designed to not require a resolver at install time. Being +able to analyze dependencies in isolation from one another when listed in a lock +file provides a few benefits. First, it supports auditing by making it easy to +figure out if a certain dependency would be installed for a certain environment. +It should also lead to faster installs which are much more frequent than +creating a lock file. Finally, the four tools mentioned in the Motivation +section either already implement this approach of evaluating dependencies in +isolation or have suggested they could in +`Poetry's case `__. + +The lock file format is designed to support two locking scenarios. The first is +locking to specific files for an environment. This is to support workflows where +the environment(s) are known ahead of time and specifying what exactly to +install is important. This is similar to what ``pip freeze`` and pip-tools_ +support, but with more strictness of the exact files as well as incorporating +into the file format the ability to specify the locked files for multiple +environments. + +The second locking scenario is to lock to package versions. This is the approach +taken by PDM_ (and indirectly by Poetry_). This is to support workflows where +the environment(s) are not known ahead of time. This can happen in open source +projects where contributors can have a myriad of different environments but +there's a desire to lock the packages used for e.g. building the project's +documentation. Since trying to lock the files for all possible environments +could be tedious at best and very expensive at worst, locking to the package +acts as a pragmatic compromise. + + +Specification +============= + +[Describe the syntax and semantics of any new language feature.] + +XXX core metadata 2.4 + +XXX file name + +XXX file format + +XXX expectations for lockers + +XXX expectations for installers + +Backwards Compatibility +======================= + +[Describe potential impact and severity on pre-existing code.] + + +Security Implications +===================== + +[How could a malicious user take advantage of this new feature?] + + +How to Teach This +================= + +[How to teach users, new and experienced, how to apply the PEP to their work.] + + +Reference Implementation +======================== + +[Link to any existing implementation and details about its state, e.g. proof-of-concept.] + + +Rejected Ideas +============== + +[Why certain ideas that were brought while discussing this PEP were not ultimately pursued.] + + +Open Issues +=========== + +[Any points that are still being decided/discussed.] + + +Footnotes +========= + +[A collection of footnotes cited in the PEP, and a place to list non-inline hyperlink targets.] + + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. + + +_Dependabot: https://docs.github.com/en/code-security/dependabot +_PDM: https://pypi.org/project/pdm/ +_pip-tools: https://pypi.org/project/pip-tools/ +_Poetry: https://python-poetry.org/ From f4fc7284c521f2a981d0e65ec5396d78914e090b Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Thu, 21 Mar 2024 13:29:37 -0700 Subject: [PATCH 02/55] Tweak rationale --- peps/pep-9999.rst | 68 ++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 56 insertions(+), 12 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 1c41df63cf8..85d5c23dfdd 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -70,22 +70,66 @@ section either already implement this approach of evaluating dependencies in isolation or have suggested they could in `Poetry's case `__. -The lock file format is designed to support two locking scenarios. The first is -locking to specific files for an environment. This is to support workflows where -the environment(s) are known ahead of time and specifying what exactly to -install is important. This is similar to what ``pip freeze`` and pip-tools_ + +Locking Scenarios +----------------- + +The lock file format is designed to support two locking scenarios. + + +Per-file Locking +~~~~~~~~~~~~~~~~ + +*Per-file locking* operates under the premise that one wants to install exact +same files in matching environments. As such, the lock file specifies the +requirements for an environment and then matches that environment requirements +with the files to install. There can be multiple environments specified in a +single file, each with their own set of files to install. By specifying the +exact files to install avoids needs performing a resolution to decide what to +install. + +The motivation for this approach to locking is for those who have controlled +environments that they work with. For instance, if you have specific, controlled +development and production environments then you can use per-file locking to +make sure the **same** files are installed in either environment for everyone. + +This is similar to what ``pip freeze`` and pip-tools_ support, but with more strictness of the exact files as well as incorporating into the file format the ability to specify the locked files for multiple environments. -The second locking scenario is to lock to package versions. This is the approach -taken by PDM_ (and indirectly by Poetry_). This is to support workflows where -the environment(s) are not known ahead of time. This can happen in open source -projects where contributors can have a myriad of different environments but -there's a desire to lock the packages used for e.g. building the project's -documentation. Since trying to lock the files for all possible environments -could be tedious at best and very expensive at worst, locking to the package -acts as a pragmatic compromise. + +Package Locking +~~~~~~~~~~~~~~~ + +*Package locking* lists the packages and their versions that *may* apply to any +environment being installed for. The of packages and their version are evaluated +individually and independently from any other packages and versions listed in +the file. This allows installation to be linear -- read each package and version +and make an isolated decision as to whether it should be installed -- and thus +not require the installer to perform a resolution to decide what to install. + +The motivation of this approach comes from +`PDM lock files `__. By listing the +potential packages and versions that may be installed, what packages may be +installed is controlled for in a way that's easy to reason about. This also +allows for not specifying the exact environments that would be supported by the +lock file so there's more flexibility and potentially easier +-- and thus faster -- lock generation. This approach supports scenarios like +open source projects that want to lock what people should use to build the +documentation as open source projects do not necessarily know upfront what +environments their contributors are working from. + +One issue with this approach has with current packaging standards is that to be +wholly accurate with what dependencies could be installed, every distribution +file would need to be downloaded and analyzed. This is is not always practical +if a `project has a lot of distribution files `__, +especially when varying metadata between distribution files is considered rare. +As such, this PEP includes a proposal of a new `core metadata`_ field to specify +if metadata varies in any way between distribution files for a package version. + +As already mentioned, this approach is supported by PDM_. Poetry_ has +`shown some interest `__. Specification From 68a6dcd8e7f058007c7166f420aac017642afa09 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Thu, 21 Mar 2024 15:32:59 -0700 Subject: [PATCH 03/55] Write down the file format --- peps/pep-9999.rst | 298 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 292 insertions(+), 6 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 85d5c23dfdd..be6f425ab6c 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -78,7 +78,7 @@ The lock file format is designed to support two locking scenarios. Per-file Locking -~~~~~~~~~~~~~~~~ +'''''''''''''''' *Per-file locking* operates under the premise that one wants to install exact same files in matching environments. As such, the lock file specifies the @@ -100,7 +100,7 @@ environments. Package Locking -~~~~~~~~~~~~~~~ +''''''''''''''' *Package locking* lists the packages and their versions that *may* apply to any environment being installed for. The of packages and their version are evaluated @@ -135,18 +135,296 @@ As already mentioned, this approach is supported by PDM_. Poetry_ has Specification ============= -[Describe the syntax and semantics of any new language feature.] +Core Metadata 2.4 +----------------- + +A new core metadata version of 2.4 is to be introduced. This new version will +introduce a new field named `Differing-Metadata`. The absence of the field +indicates that the metadata is consistent across all distribution files for a +certain version. A value of ``all`` signifies **any** distribution file for a +specific version *may* have differing metadata. A value of ``sdist`` signifies +that only source distribution file for a specific version *may* have differing +metadata. + +This should allow for the reading of the metadata from any distribution file to +know whether examining the other distribution files have the same or differing +metadata. + + +File Name +--------- + +A lock file MUST be named ``pylock.toml`` or match the regular expression +``f"pylock\.([A-Z0-9]|[A-Z0-9][A-Z0-9._-]*[A-Z0-9])\.toml"`` (the part between +``pylock.`` and ``.toml`` comes from +`valid non-normalized names `__ +and their usage in :pep:`735`). The use of the ``.toml`` extension is to make +syntax highlighting in editors easier and to reenforce the fact that the file +format is meant to be human-readable. + + +File Format +----------- + +The format of the file is TOML_. + +All keys listed below are required unless otherwise noted. If two keys are +mutually exclusive to one another, then one of the keys is required while the +other is disallowed. + + +``version`` +''''''''''' + +- String +- The version of the lock file format +- This PEP specifies the initial version -- and only valid vailue until future + updates to the standard -- as ``"1.0"`` + + +``dependencies`` +''''''''''''''' + +- Array of strings +- A listing the `dependency specifiers`_ that act as the input to the lock file, + representing the direct, top-level dependencies to be installed + + +``[[file-lock]]`` +''''''''''''''''' + +- Mutually exclusive with ``[package-lock]`` +- Array of tables +- The table's existence infers the use of the per-file lockinng approach +- An environment that meets all of the specified criteria in the table will be + considered compatible with the environment that was locked for + + +``file-lock.name`` +'''''''''''''''''' + +- String +- A unique name for the environment this table represents + + +``[file-lock.marker-values]`` +''''''''''''''''''''''''''''' + +- Optional +- Table of strings +- The keys represent the names of `environment markers`_ and the values are the + values for those markers +- Compatibility is defined by the environment's values matching what is in the + table +- Lockers SHOULD sort the keys lexicographically to minimize changes when + updating the file + + +``file-lock.wheel-tags`` +'''''''''''''''''''''''' + +- Optional +- Array of strings +- An unordered list of `wheel tags`_ which must be supported by the environment +- Lockers SHOULD sort the keys lexicographically to minimize changes when + updating the file +- Lockers MUST NOT include + `compressed tag sets `__ + or duplicate tags for consistency across lockers and to simplify checking for + compatibility + + +``[package-lock]`` +'''''''''''''''''' + +- Table +- Mutually exclusive with ``[[file-lock]]`` +- Signifies + + +``package-lock.requires-python`` +'''''''''''''''''''''''''''''''' + +- String +- Holds the `version specifiers`_ for Python version compatibility + + +``[[package]]`` +''''''''''''''' + +- Array of tables +- Contains all data on the locked package versions +- Lockers MUST NOT list the same package and version more than once +- Lockers SHOULD record packages in order by ``package.name`` lexicographically + and ``package.version`` by the sort order for `version specifiers`_ +- Lockers SHOULD record keys in the same order as written in this PEP to + minimmize changes when updating + + +``package.name`` +'''''''''''''''' + +- String +- The `normalized name`_ of the pacakage + + +``package.version`` +''''''''''''''''''' + +- String +- The version of the package + -XXX core metadata 2.4 +``package.project-details-url`` +''''''''''''''''''''''' -XXX file name +- Optional +- String +- The `project detail` URL +- Useful for generating Packaging URLs (aka *PURLs*) + + +``package.marker`` +'''''''''''''''''' + +- Optional +- String +- The `environment markers`_ expression which specifies whether this package and + version applies to the environment +- Only applicable via ``[package-lock]`` and the package locking scenario +- The lack of this key means this package and version is required to be + installed + + +``package.requires-python`` +''''''''''''''''''''''''''' + +- Optional +- String +- Holds the `version specifiers`_ for Python version compatibility for the + package and version +- Useful for documenting why this package and version was included in the file +- It should not provide useful information for installers as it would be + captured by `package-lock.requires-python` or isn't relevant when + ``[[file-lock]]`` is used + + +``package.dependencies`` +'''''''''''''''''''''''' + +- Optional +- Array of strings +- record the dependencies of the package and version +- This is useful in analyzing why a package happens to be listed in the file +- This does not provide information which influences the installer as + ``[[file-lock]]`` specifies the exact files to use and ``[package-lock]`` + applicability is determined by ``package.marker`` + + +``[[package.files]]`` +''''''''''''''''''''' + +- Must be specified if ``[package.vcs]`` is not +- Array of tables +- Tables can be written inline + + +``package.files.name`` +'''''''''''''''''''''''''' + +- String +- File name + + +``package.files.hash`` +'''''''''''''''''''''' + +- String +- The hash of the file contents +- The format is ``f"{hashname}={hashvalue}"`` which is the same as the used by + the `Simple Repository API`_ and its HTML form +- Only a single hash value is used to allow the table to be written inline +- Using a single string to store both the hash algorithm and value instead of + separate keys for the two values is to make the inline table shorter + + +``package.files.origin`` +'''''''''''''''''''''''' + +- Optional +- String +- URI where the file was found when the lock file was generated + + +``package.files.file-lock`` +''''''''''''''''''''''''''' + +- Required when ``[[file-lock]]`` is used +- Array of strings +- An array of ``file-lock.name`` values which signify that the file is to be + installed when the corresponding ``[[file-lock]]`` table applies to the + environment + + +``[package.vcs]`` +''''''''''''''''' + +- Must be specified if ``[[package.files]]`` is not +- Table representing the version control system containing the package and + version + + +``package.vcs.type`` +'''''''''''''''''''' + +- String +- The type of version control system used +- The valid values are specified by the + `registered VCSs `__ + of the direct URL data structure + + +``package.vcs.origin`` +'''''''''''''''''''''' + +- String +- The URI of where the repository was located when the lock file was generated + + +``package.vcs.commit`` +'''''''''''''''''''''' + +- String +- The commit ID for the repository which represents the package and version + + +``package.vcs.file-lock`` +''''''''''''''''''''''''' + +- Required when ``[[file-lock]]`` is used +- An array of strings +- An array of ``file-lock.name`` values which signify that the repository at the + specified commit is to be installed when the corresponding ``[[file-lock]]`` + table applies to the environment + + +``[[package.build-requires]]`` +'''''''''''''''''''''''''''''' + +- Optional +- An array of tables whose structure matches that of ``[[package]]`` +- Each entry represents a package and version to use when building the + enclosing package and version +- Selection of which entries to use for an environment as the same as + ``[[package]]`` itself, albeit only applying when installing build back-end + dependencies -XXX file format XXX expectations for lockers XXX expectations for installers + Backwards Compatibility ======================= @@ -196,7 +474,15 @@ This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. +_core metadata: https://packaging.python.org/en/latest/specifications/core-metadata/ _Dependabot: https://docs.github.com/en/code-security/dependabot +_dependency specifiers: https://packaging.python.org/en/latest/specifications/dependency-specifiers/ +_environment markers: https://packaging.python.org/en/latest/specifications/dependency-specifiers/#environment-markers +_normalized name: https://packaging.python.org/en/latest/specifications/name-normalization/#name-normalization _PDM: https://pypi.org/project/pdm/ _pip-tools: https://pypi.org/project/pip-tools/ _Poetry: https://python-poetry.org/ +_project detail: https://packaging.python.org/en/latest/specifications/simple-repository-api/#project-detail +_Simple Repository API: https://packaging.python.org/en/latest/specifications/simple-repository-api/ +_version specifiers: https://packaging.python.org/en/latest/specifications/version-specifiers/ +_wheel tags: https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/ From 761922ea83650930ee424a32ffae8dc5a2bcb25c Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Thu, 21 Mar 2024 15:58:43 -0700 Subject: [PATCH 04/55] Add in the `[[tool]]` table array --- peps/pep-9999.rst | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index be6f425ab6c..5a8c4e3875c 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -420,7 +420,13 @@ other is disallowed. dependencies -XXX expectations for lockers +``[tool]`` +'''''''''''' + +- Optional +- Same usage as that of the equivalent table from the + `pyproject.toml specification`_ + XXX expectations for installers From 61a72f522b16baca2fb4ffa5c96093ac24f22ced Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Thu, 21 Mar 2024 15:59:02 -0700 Subject: [PATCH 05/55] Add expectations for lockers --- peps/pep-9999.rst | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 5a8c4e3875c..6ebcc171bb8 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -428,7 +428,20 @@ other is disallowed. `pyproject.toml specification`_ -XXX expectations for installers +Expectations for Lockers +------------------------ + +- When creating a lock file for ``[package-lock]`` and a package and version are + not using core metadata 2.4 as proposed by this PEP, the locker SHOULD read + the metadata of all files listed in ``[[package.files]]`` to make sure all + potential metadata cases are covered +- If a locker chooses not to check every file, the tool MUST either provide the + user with the option to have all files checked (whether that is opt-in or out + is left up to the tool), or the user is somehow notified that such a + standards-violating shortcut is being taken (whether this is by documentation + or at runtime is left to the tool) + + Backwards Compatibility From 86e44fae2728b9fbf0b3c6db626de30087bd2f1e Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Thu, 21 Mar 2024 15:59:42 -0700 Subject: [PATCH 06/55] Add expectations for installers for per-file locking --- peps/pep-9999.rst | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 6ebcc171bb8..17c4a5fde96 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -442,6 +442,26 @@ Expectations for Lockers or at runtime is left to the tool) +Expectations for Installers +--------------------------- + +Installing for per-file locking +''''''''''''''''''''''''''''''' + +- Iterate through each ``[[file-lock]]`` table to find the one that applies to + the environment being installed for +- If no compatible environment is found an error MUST be raised +- For the compatible environment, iterate through each entry in ``[[package]]`` +- For each ``[[package]]`` entry, iterate through ``[[package.files]]`` to look +- for any files with ``file-lock.name`` listed in ``package.files.lock`` +- If a file is found, install it and move on to the next ``[[package]]`` entry +- If no file is found then check if ``package.vcs.lock`` contains a match (no + match is acceptable) +- If a ``[[package.files]]`` contains multiple matching entries an error MUST + be raised due to ambiguity for what is to be installed +- If multiple ``[[package]]`` entries for the same package have matching files + an error MUST be raised due to ambiguity for what is to be installed + Backwards Compatibility From a960f76ab705674ca96be645e8067c2d55b6fb31 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Thu, 21 Mar 2024 16:14:11 -0700 Subject: [PATCH 07/55] Add some clarifications --- peps/pep-9999.rst | 59 ++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 51 insertions(+), 8 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 17c4a5fde96..947140b33f7 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -52,8 +52,6 @@ etc.). Rationale ========= -[Describe why particular design decisions were made.] - The file format is designed to be human-readable. This is so that the contents of the file can be audited by a human to make sure no undesired dependencies end up being included in the lock file. It is also to @@ -70,6 +68,12 @@ section either already implement this approach of evaluating dependencies in isolation or have suggested they could in `Poetry's case `__. +The format is also designed so that a *locker* which produces the lock file +and an *installer* which consumes the lock file can be separate tools. This +allows for situations such as cloud hosting providers to use their own installer +that's optimized for their system which is independent of what locker the user +used to create their lock file. + Locking Scenarios ----------------- @@ -158,9 +162,9 @@ A lock file MUST be named ``pylock.toml`` or match the regular expression ``f"pylock\.([A-Z0-9]|[A-Z0-9][A-Z0-9._-]*[A-Z0-9])\.toml"`` (the part between ``pylock.`` and ``.toml`` comes from `valid non-normalized names `__ -and their usage in :pep:`735`). The use of the ``.toml`` extension is to make -syntax highlighting in editors easier and to reenforce the fact that the file -format is meant to be human-readable. +and their usage in :pep:`735`). The use of the ``.toml`` file extension is to +make syntax highlighting in editors easier and to reinforce the fact that the +file format is meant to be human-readable. File Format @@ -198,6 +202,8 @@ other is disallowed. - The table's existence infers the use of the per-file lockinng approach - An environment that meets all of the specified criteria in the table will be considered compatible with the environment that was locked for +- Lockers MUST NOT generate multiple ``[file-lock]`` tables which would be + considered compatible for the same environment ``file-lock.name`` @@ -225,7 +231,10 @@ other is disallowed. - Optional - Array of strings -- An unordered list of `wheel tags`_ which must be supported by the environment +- An unordered array of `wheel tags`_ which must be supported by the environment +- The array MAY not be exhaustive to allow for a smaller array as well as to + help prevent multiple ``[[file-lock]]`` tables being compatible with the + same environment - Lockers SHOULD sort the keys lexicographically to minimize changes when updating the file - Lockers MUST NOT include @@ -356,7 +365,7 @@ other is disallowed. - URI where the file was found when the lock file was generated -``package.files.file-lock`` +``package.files.lock`` ''''''''''''''''''''''''''' - Required when ``[[file-lock]]`` is used @@ -364,6 +373,8 @@ other is disallowed. - An array of ``file-lock.name`` values which signify that the file is to be installed when the corresponding ``[[file-lock]]`` table applies to the environment +- There MUST only be a single file with any one ``file-lock.name`` entry per + package, regardless of version ``[package.vcs]`` @@ -398,7 +409,7 @@ other is disallowed. - The commit ID for the repository which represents the package and version -``package.vcs.file-lock`` +``package.vcs.lock`` ''''''''''''''''''''''''' - Required when ``[[file-lock]]`` is used @@ -406,6 +417,9 @@ other is disallowed. - An array of ``file-lock.name`` values which signify that the repository at the specified commit is to be installed when the corresponding ``[[file-lock]]`` table applies to the environment +- A name in the array may only appear if no file listed in + ``package.files.lock`` contains the name for the same package, regardless of + version ``[[package.build-requires]]`` @@ -418,6 +432,9 @@ other is disallowed. - Selection of which entries to use for an environment as the same as ``[[package]]`` itself, albeit only applying when installing build back-end dependencies +- This helps with reproducibility of the building of a package by recording + either what was or would have been used if the locker needed to build the + package ``[tool]`` @@ -440,6 +457,19 @@ Expectations for Lockers is left up to the tool), or the user is somehow notified that such a standards-violating shortcut is being taken (whether this is by documentation or at runtime is left to the tool) +- Lockers MAY want to provide a way to let users provide the information + necessary to install for multiple environments at once when doing per-file + locking, e.g. supporting a JSON file format which specifies wheel tags and + marker values much like in ``[[file-lock]]`` for which multiple files can be + specified, which could then be directly recorded in the corresponding + ``[[file-lock]]`` table + +.. code-block:: JSON + + { + "marker-values": {...} + "wheel-tags": [...] + } Expectations for Installers @@ -448,6 +478,8 @@ Expectations for Installers Installing for per-file locking ''''''''''''''''''''''''''''''' +An example workflow is: + - Iterate through each ``[[file-lock]]`` table to find the one that applies to the environment being installed for - If no compatible environment is found an error MUST be raised @@ -461,7 +493,17 @@ Installing for per-file locking be raised due to ambiguity for what is to be installed - If multiple ``[[package]]`` entries for the same package have matching files an error MUST be raised due to ambiguity for what is to be installed +- Find and verify the selected files and/or CVS entries based on their hash or + commit ID as appropriate +- If a source distribution or VCS was selected and + ``[[package.build-requires]]`` exists, then repeat the above process to + install the build dependencies necessary to build the package + + +Installing for package locking +'''''''''''''''''''''''''''''' +XXX Backwards Compatibility @@ -522,6 +564,7 @@ _PDM: https://pypi.org/project/pdm/ _pip-tools: https://pypi.org/project/pip-tools/ _Poetry: https://python-poetry.org/ _project detail: https://packaging.python.org/en/latest/specifications/simple-repository-api/#project-detail +_pyproject.toml specification: https://packaging.python.org/en/latest/specifications/pyproject-toml/#pyproject-toml-specification _Simple Repository API: https://packaging.python.org/en/latest/specifications/simple-repository-api/ _version specifiers: https://packaging.python.org/en/latest/specifications/version-specifiers/ _wheel tags: https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/ From 08f59d360f6e2ef9596213d054083ffb2f620993 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Thu, 21 Mar 2024 17:06:11 -0700 Subject: [PATCH 08/55] Example workflow for package locking --- peps/pep-9999.rst | 38 +++++++++++++++++++++++++++++++------- 1 file changed, 31 insertions(+), 7 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 947140b33f7..d6f362bb2b4 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -42,7 +42,7 @@ defaults in the face of supply chain attacks. Finally, not all the formats are easy to audit to determine what what would be installed into an environment ahead of time. -Ghe lack of a standard also has some drawbacks. For instance, any tooling that +The lack of a standard also has some drawbacks. For instance, any tooling that wants to work with lock files must choose which format to support, potentially leaving users unsupported (e.g. if Dependabot_ chose not to support PDM. support by cloud providers who can do dependency installations on your behalf, @@ -285,7 +285,7 @@ other is disallowed. ``package.project-details-url`` -''''''''''''''''''''''' +''''''''''''''''''''''''''''''' - Optional - String @@ -339,7 +339,7 @@ other is disallowed. ``package.files.name`` -'''''''''''''''''''''''''' +'''''''''''''''''''''' - String - File name @@ -438,7 +438,7 @@ other is disallowed. ``[tool]`` -'''''''''''' +'''''''''' - Optional - Same usage as that of the equivalent table from the @@ -475,6 +475,14 @@ Expectations for Lockers Expectations for Installers --------------------------- +- Installers MAY support installation of non-binary files + (i.e. source distributions and VCS) +- Installers MUST provide a way to avoid non-binary file installation for + reproducibility and security +- Installers SHOULD make it opt-in for using non-binary file installation for a + secure-by-default approach + + Installing for per-file locking ''''''''''''''''''''''''''''''' @@ -496,14 +504,30 @@ An example workflow is: - Find and verify the selected files and/or CVS entries based on their hash or commit ID as appropriate - If a source distribution or VCS was selected and - ``[[package.build-requires]]`` exists, then repeat the above process to - install the build dependencies necessary to build the package + ``[[package.build-requires]]`` exists, then repeat the above process as + appropriate to install the build dependencies necessary to build the package Installing for package locking '''''''''''''''''''''''''''''' -XXX +An example workflow is: + +- Verify that the environment is compatible with `package-lock.requires-python`; + if it isn't an error MUST be raised +- Iterate through each entry in ``[package]]`` +- For each entry, if there's a ``package.marker`` key, evaluate the expression + - If the expression is false, then move on + - Otherwise the package entry must be installed +- Iterate through the files listed in ``[[package.files]]``, looking for the + "best" file to install +- If no file is found, check for ``[package.vcs]`` +- If not match is found, an error MUST be raised +- Find and verify the selected files and/or CVS entries based on their hash or + commit ID as appropriate +- If the match is a source distribution or VCS and + ``[[package.build-requires]]`` is provided, repeat the above as appropriate to + build the package Backwards Compatibility From 82393b2abdb9356b0b56c39b899eb4c5fe1fb0a1 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Thu, 21 Mar 2024 18:06:43 -0700 Subject: [PATCH 09/55] Bit of a clarification --- peps/pep-9999.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index d6f362bb2b4..dc919a4a13e 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -148,7 +148,7 @@ indicates that the metadata is consistent across all distribution files for a certain version. A value of ``all`` signifies **any** distribution file for a specific version *may* have differing metadata. A value of ``sdist`` signifies that only source distribution file for a specific version *may* have differing -metadata. +metadata while all other files have the same metadata. This should allow for the reading of the metadata from any distribution file to know whether examining the other distribution files have the same or differing From 8c6801df1d649c076117535f075186552c088fb3 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Fri, 22 Mar 2024 18:04:06 -0700 Subject: [PATCH 10/55] Add `package.tool` --- peps/pep-9999.rst | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index dc919a4a13e..04f821ab006 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -437,6 +437,14 @@ other is disallowed. package +``[package.tool]`` +'''''''''''''''''' + +- Optional +- Same usage as that of the equivalent table from the + `pyproject.toml specification`_ + + ``[tool]`` '''''''''' From 6b188bad897810cff1579c8f8efa953e86ff4f9a Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Fri, 22 Mar 2024 21:21:04 -0700 Subject: [PATCH 11/55] Add `package.dependents` --- peps/pep-9999.rst | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 04f821ab006..0176dfdeaa3 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -318,13 +318,25 @@ other is disallowed. ``[[file-lock]]`` is used +``package.dependents`` +'''''''''''''''''''''' + +- Optional +- Array of strings +- A record of the packages that depend on this package and version +- This is useful inn analyzing why a package happens to be listed in the file + for auditing purposes +- This does not provide information which influences installers + + ``package.dependencies`` '''''''''''''''''''''''' - Optional - Array of strings -- record the dependencies of the package and version +- A record the dependencies of the package and version - This is useful in analyzing why a package happens to be listed in the file + for auditing purposes - This does not provide information which influences the installer as ``[[file-lock]]`` specifies the exact files to use and ``[package-lock]`` applicability is determined by ``package.marker`` From 6cced992e27d9a5e92b6cca57ce334265cc99a39 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Fri, 22 Mar 2024 22:14:41 -0700 Subject: [PATCH 12/55] Fix a typo --- peps/pep-9999.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 0176dfdeaa3..f2c5adcb72a 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -274,7 +274,7 @@ other is disallowed. '''''''''''''''' - String -- The `normalized name`_ of the pacakage +- The `normalized name`_ of the package ``package.version`` From 0080fc201b272704b69ff21292689b67aa27a5e3 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Fri, 22 Mar 2024 22:47:01 -0700 Subject: [PATCH 13/55] Add `package.description` Along the way also add more explanation as to why some things are included --- peps/pep-9999.rst | 45 +++++++++++++++++++++++++++++++++------------ 1 file changed, 33 insertions(+), 12 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index f2c5adcb72a..2da68d22357 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -275,6 +275,7 @@ other is disallowed. - String - The `normalized name`_ of the package +- Part of what's required to uniquely identify this entry ``package.version`` @@ -282,6 +283,7 @@ other is disallowed. - String - The version of the package +- Part of what's required to uniquely identify this entry ``package.project-details-url`` @@ -293,6 +295,15 @@ other is disallowed. - Useful for generating Packaging URLs (aka *PURLs*) +``package.description`` +''''''''''''''''''''''' + +- Optional +- String +- The package's ``Summary`` from its `core metadata`_ +- Useful to help understand why a package was included in the file + + ``package.marker`` '''''''''''''''''' @@ -313,6 +324,8 @@ other is disallowed. - Holds the `version specifiers`_ for Python version compatibility for the package and version - Useful for documenting why this package and version was included in the file +- Also helps document why the version restriction in + ``package-lock.requires-python`` was chosen - It should not provide useful information for installers as it would be captured by `package-lock.requires-python` or isn't relevant when ``[[file-lock]]`` is used @@ -324,7 +337,7 @@ other is disallowed. - Optional - Array of strings - A record of the packages that depend on this package and version -- This is useful inn analyzing why a package happens to be listed in the file +- Useful inn analyzing why a package happens to be listed in the file for auditing purposes - This does not provide information which influences installers @@ -335,7 +348,7 @@ other is disallowed. - Optional - Array of strings - A record the dependencies of the package and version -- This is useful in analyzing why a package happens to be listed in the file +- Useful in analyzing why a package happens to be listed in the file for auditing purposes - This does not provide information which influences the installer as ``[[file-lock]]`` specifies the exact files to use and ``[package-lock]`` @@ -354,7 +367,8 @@ other is disallowed. '''''''''''''''''''''' - String -- File name +- The file name +- Necessary for installers to decide what to install when using package locking ``package.files.hash`` @@ -367,6 +381,8 @@ other is disallowed. - Only a single hash value is used to allow the table to be written inline - Using a single string to store both the hash algorithm and value instead of separate keys for the two values is to make the inline table shorter +- Used by installers to verify the file contents match what the locker worked + with ``package.files.origin`` @@ -375,10 +391,12 @@ other is disallowed. - Optional - String - URI where the file was found when the lock file was generated +- Useful for documenting where the file came from and potentially where to look + for the file if not already downloaded/available ``package.files.lock`` -''''''''''''''''''''''''''' +'''''''''''''''''''''' - Required when ``[[file-lock]]`` is used - Array of strings @@ -442,11 +460,14 @@ other is disallowed. - Each entry represents a package and version to use when building the enclosing package and version - Selection of which entries to use for an environment as the same as - ``[[package]]`` itself, albeit only applying when installing build back-end - dependencies + ``[[package]]`` itself, albeit only applying when installing the build + back-end and its dependencies - This helps with reproducibility of the building of a package by recording either what was or would have been used if the locker needed to build the package +- If the installer and user choose to install from source and this array is + missing then the installer MAY choose to resolve what to install for building + at install time, otherwise the installer MUST raise an error ``[package.tool]`` @@ -472,11 +493,11 @@ Expectations for Lockers not using core metadata 2.4 as proposed by this PEP, the locker SHOULD read the metadata of all files listed in ``[[package.files]]`` to make sure all potential metadata cases are covered -- If a locker chooses not to check every file, the tool MUST either provide the - user with the option to have all files checked (whether that is opt-in or out - is left up to the tool), or the user is somehow notified that such a - standards-violating shortcut is being taken (whether this is by documentation - or at runtime is left to the tool) +- If a locker chooses not to check every file for its metadata, the tool MUST + either provide the user with the option to have all files checked (whether + that is opt-in or out is left up to the tool), or the user is somehow notified + that such a standards-violating shortcut is being taken (whether this is by + documentation or at runtime is left to the tool) - Lockers MAY want to provide a way to let users provide the information necessary to install for multiple environments at once when doing per-file locking, e.g. supporting a JSON file format which specifies wheel tags and @@ -498,7 +519,7 @@ Expectations for Installers - Installers MAY support installation of non-binary files (i.e. source distributions and VCS) - Installers MUST provide a way to avoid non-binary file installation for - reproducibility and security + reproducibility and security purposes - Installers SHOULD make it opt-in for using non-binary file installation for a secure-by-default approach From 1214a78b1946306f6209b78cb802fcfd6ca92fae Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Mon, 25 Mar 2024 16:12:09 -0700 Subject: [PATCH 14/55] Add `package.direct`, `package.directory`, and `package.multiple-entries` --- peps/pep-9999.rst | 45 +++++++++++++++++++++++++++++++++++++-------- 1 file changed, 37 insertions(+), 8 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 2da68d22357..bd646d4fb60 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -286,13 +286,15 @@ other is disallowed. - Part of what's required to uniquely identify this entry -``package.project-details-url`` -''''''''''''''''''''''''''''''' +``package.multiple-entries`` +'''''''''''''''''''''''''''' -- Optional -- String -- The `project detail` URL -- Useful for generating Packaging URLs (aka *PURLs*) +- Required when there are multiple entries for the same package +- Boolean +- If package locking via ``[package-lock]``, then the multiple entries for the + same package MUST be mutually exclusive via ``package.marker`` (this is not + required for per-file locking as the ``package.*.lock`` entries imply mutual + exclusivity) ``package.description`` @@ -301,7 +303,17 @@ other is disallowed. - Optional - String - The package's ``Summary`` from its `core metadata`_ -- Useful to help understand why a package was included in the file +- Useful to help understand why a package was included in the file based on its + purpose + + +``package.project-details-url`` +''''''''''''''''''''''''''''''' + +- Optional +- String +- The `project detail` URL +- Useful for generating Packaging URLs (aka *PURLs*) ``package.marker`` @@ -337,7 +349,7 @@ other is disallowed. - Optional - Array of strings - A record of the packages that depend on this package and version -- Useful inn analyzing why a package happens to be listed in the file +- Useful for analyzing why a package happens to be listed in the file for auditing purposes - This does not provide information which influences installers @@ -355,6 +367,14 @@ other is disallowed. applicability is determined by ``package.marker`` +``package.direct`` +'''''''''''''''''' + +- Optional +- Boolean +- Represents whether the installation is via a `direct URL reference`_ + + ``[[package.files]]`` ''''''''''''''''''''' @@ -452,6 +472,14 @@ other is disallowed. version +``package.directory`` +''''''''''''''''''''' + +- Optional +- String +- A local directory where a source tree for the package and version exists + + ``[[package.build-requires]]`` '''''''''''''''''''''''''''''' @@ -623,6 +651,7 @@ CC0-1.0-Universal license, whichever is more permissive. _core metadata: https://packaging.python.org/en/latest/specifications/core-metadata/ _Dependabot: https://docs.github.com/en/code-security/dependabot _dependency specifiers: https://packaging.python.org/en/latest/specifications/dependency-specifiers/ +_direct URL reference: https://packaging.python.org/en/latest/specifications/direct-url/ _environment markers: https://packaging.python.org/en/latest/specifications/dependency-specifiers/#environment-markers _normalized name: https://packaging.python.org/en/latest/specifications/name-normalization/#name-normalization _PDM: https://pypi.org/project/pdm/ From 92de818ead51a837a1dcc0d0f8a31f64b62bbe70 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Tue, 26 Mar 2024 14:02:36 -0700 Subject: [PATCH 15/55] Rename `package.simple-repo-package-url` and add `package.files.simple-repo-package-url` Also tweak Core Metadata 2.4 to be simpler by outright banning varying metadata --- peps/pep-9999.rst | 34 ++++++++++++++++++++++------------ 1 file changed, 22 insertions(+), 12 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index bd646d4fb60..3d3f3e41f57 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -142,13 +142,12 @@ Specification Core Metadata 2.4 ----------------- -A new core metadata version of 2.4 is to be introduced. This new version will -introduce a new field named `Differing-Metadata`. The absence of the field -indicates that the metadata is consistent across all distribution files for a -certain version. A value of ``all`` signifies **any** distribution file for a -specific version *may* have differing metadata. A value of ``sdist`` signifies -that only source distribution file for a specific version *may* have differing -metadata while all other files have the same metadata. +A new core metadata version of 2.4 is to be introduced. When metadata uses this +version the contained metadata MUST be consistent across files for the same +package and version unless it is marked as ``Dynamic``. This effectively means +**all** wheel files will have the same metadata and source distributions _may_ +also have the same metadata **if** all the fields static (i.e. `Dynamic` is not +set). This should allow for the reading of the metadata from any distribution file to know whether examining the other distribution files have the same or differing @@ -262,8 +261,7 @@ other is disallowed. ''''''''''''''' - Array of tables -- Contains all data on the locked package versions -- Lockers MUST NOT list the same package and version more than once +- The array contains all data on the locked package versions - Lockers SHOULD record packages in order by ``package.name`` lexicographically and ``package.version`` by the sort order for `version specifiers`_ - Lockers SHOULD record keys in the same order as written in this PEP to @@ -307,12 +305,12 @@ other is disallowed. purpose -``package.project-details-url`` -''''''''''''''''''''''''''''''' +``package.simple-repo-package-url`` +''''''''''''''''''''''''''''''''''' - Optional - String -- The `project detail` URL +- Storess the `project detail`_ URL from the `Simple Repository API`_ - Useful for generating Packaging URLs (aka *PURLs*) @@ -405,6 +403,18 @@ other is disallowed. with +``package.files.simple-repo-package-url`` +''''''''''''''''''''''''''''''''''''''''' + +- Optional +- String +- The value has the same meaning as ``package.simple-repo-package-url`` +- If set, it overrides the value from ``package.simple-repo-package-url`` for + this file **only** +- This is to support :pep:`708` when some files override what's provided by + another `Simple Repository API`_ index + + ``package.files.origin`` '''''''''''''''''''''''' From 619bd6e0cc1f2f92f5494d9e891c041f3fa357e4 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Thu, 28 Mar 2024 12:12:25 -0700 Subject: [PATCH 16/55] Updates based on some review comments - Drop core metadata 2.4 - Suggest SHA-256 or better - Suggest lockers set the simple Repo package URL - Rearrange the order of some keys --- peps/pep-9999.rst | 72 ++++++++++++++++++----------------------------- 1 file changed, 28 insertions(+), 44 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 3d3f3e41f57..b62b9baaa69 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -124,14 +124,6 @@ open source projects that want to lock what people should use to build the documentation as open source projects do not necessarily know upfront what environments their contributors are working from. -One issue with this approach has with current packaging standards is that to be -wholly accurate with what dependencies could be installed, every distribution -file would need to be downloaded and analyzed. This is is not always practical -if a `project has a lot of distribution files `__, -especially when varying metadata between distribution files is considered rare. -As such, this PEP includes a proposal of a new `core metadata`_ field to specify -if metadata varies in any way between distribution files for a package version. - As already mentioned, this approach is supported by PDM_. Poetry_ has `shown some interest `__. @@ -139,21 +131,6 @@ As already mentioned, this approach is supported by PDM_. Poetry_ has Specification ============= -Core Metadata 2.4 ------------------ - -A new core metadata version of 2.4 is to be introduced. When metadata uses this -version the contained metadata MUST be consistent across files for the same -package and version unless it is marked as ``Dynamic``. This effectively means -**all** wheel files will have the same metadata and source distributions _may_ -also have the same metadata **if** all the fields static (i.e. `Dynamic` is not -set). - -This should allow for the reading of the metadata from any distribution file to -know whether examining the other distribution files have the same or differing -metadata. - - File Name --------- @@ -312,6 +289,8 @@ other is disallowed. - String - Storess the `project detail`_ URL from the `Simple Repository API`_ - Useful for generating Packaging URLs (aka *PURLs*) +- When possible, lockers SHOULD include this key to assist with generating + `software bill of materials`_ (aka SBOMs) ``package.marker`` @@ -389,18 +368,16 @@ other is disallowed. - Necessary for installers to decide what to install when using package locking -``package.files.hash`` +``package.files.lock`` '''''''''''''''''''''' -- String -- The hash of the file contents -- The format is ``f"{hashname}={hashvalue}"`` which is the same as the used by - the `Simple Repository API`_ and its HTML form -- Only a single hash value is used to allow the table to be written inline -- Using a single string to store both the hash algorithm and value instead of - separate keys for the two values is to make the inline table shorter -- Used by installers to verify the file contents match what the locker worked - with +- Required when ``[[file-lock]]`` is used +- Array of strings +- An array of ``file-lock.name`` values which signify that the file is to be + installed when the corresponding ``[[file-lock]]`` table applies to the + environment +- There MUST only be a single file with any one ``file-lock.name`` entry per + package, regardless of version ``package.files.simple-repo-package-url`` @@ -413,6 +390,8 @@ other is disallowed. this file **only** - This is to support :pep:`708` when some files override what's provided by another `Simple Repository API`_ index +- Lockers SHOULD include this key when appropriate to assist with generating + `software bill of materials`_ (aka SBOMs) ``package.files.origin`` @@ -425,16 +404,21 @@ other is disallowed. for the file if not already downloaded/available -``package.files.lock`` +``package.files.hash`` '''''''''''''''''''''' -- Required when ``[[file-lock]]`` is used -- Array of strings -- An array of ``file-lock.name`` values which signify that the file is to be - installed when the corresponding ``[[file-lock]]`` table applies to the - environment -- There MUST only be a single file with any one ``file-lock.name`` entry per - package, regardless of version +- String +- The hash of the file contents +- The format is ``f"{hashname}={hashvalue}"`` which is the same as the used by + the `Simple Repository API`_ and its HTML form +- Only a single hash value is used to allow the table to be written inline + for readability and compactness purposes +- Using a single string to store both the hash algorithm and value instead of + separate keys for the two values is to make the inline table shorter +- Used by installers to verify the file contents match what the locker worked + with +- Lockers SHOULD use a hash algorithm that is as least as strong as + `SHA-256 `__ ``[package.vcs]`` @@ -527,9 +511,8 @@ other is disallowed. Expectations for Lockers ------------------------ -- When creating a lock file for ``[package-lock]`` and a package and version are - not using core metadata 2.4 as proposed by this PEP, the locker SHOULD read - the metadata of all files listed in ``[[package.files]]`` to make sure all +- When creating a lock file for ``[package-lock]``, the locker SHOULD read + the metadata of **all** files listed in ``[[package.files]]`` to make sure all potential metadata cases are covered - If a locker chooses not to check every file for its metadata, the tool MUST either provide the user with the option to have all files checked (whether @@ -670,5 +653,6 @@ _Poetry: https://python-poetry.org/ _project detail: https://packaging.python.org/en/latest/specifications/simple-repository-api/#project-detail _pyproject.toml specification: https://packaging.python.org/en/latest/specifications/pyproject-toml/#pyproject-toml-specification _Simple Repository API: https://packaging.python.org/en/latest/specifications/simple-repository-api/ +_ software bill of materials: https://www.cisa.gov/sbom _version specifiers: https://packaging.python.org/en/latest/specifications/version-specifiers/ _wheel tags: https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/ From 244957c41c501f374d91d5b27759f543fda7ed6a Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Thu, 28 Mar 2024 16:42:54 -0700 Subject: [PATCH 17/55] Use a fake PEP number to make CI happy --- peps/pep-9999.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index b62b9baaa69..7323a33ef78 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -1,4 +1,4 @@ -PEP: +PEP: 9999 Title: A file format to list Python dependencies for installation reproducibility Author: Brett Cannon PEP-Delegate: From 6a92b8f1363a5bc94153b8a6759157de0f8b5bc8 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Thu, 28 Mar 2024 16:47:47 -0700 Subject: [PATCH 18/55] Add an explicit note that the directory to write a `pylock.toml` file is not specified on purpose. --- peps/pep-9999.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 7323a33ef78..f48ce42733e 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -142,6 +142,9 @@ and their usage in :pep:`735`). The use of the ``.toml`` file extension is to make syntax highlighting in editors easier and to reinforce the fact that the file format is meant to be human-readable. +This PEP has no opinion as to the location of lock files (i.e. in the root or +some subdirectory of a project). + File Format ----------- From 2eb7166ea4097177535b194aeb5bbf003ddeaa98 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Thu, 28 Mar 2024 19:09:26 -0700 Subject: [PATCH 19/55] Make the hash algorithm its own setting --- peps/pep-9999.rst | 138 +++++++++++++++++++++++++++++++++++----------- 1 file changed, 105 insertions(+), 33 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index f48ce42733e..04de61f9a1d 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -13,8 +13,6 @@ Replaces: 665 Abstract ======== -[A short (~200 word) description of the technical issue being addressed.] - This PEP proposes a new file format for specifying what dependencies to (possibly) install into a Python environment for consitent installation repoducibility. The format is designed to be human-readable but @@ -165,6 +163,24 @@ other is disallowed. updates to the standard -- as ``"1.0"`` +``hash-algorithm`` +'''''''''''''''''' + +- String +- The name of the hash algorithm used for calculating all hash values +- Only a single hash algorithm is used for the entire file to allow the + ``[[package.files]]`` table to be written inline for readability and + compactness purposes by only listing a single hash value instead of multiple + values based on multiple hash algorithms +- Specifying a single hash algorithm guarantees that an algorithm that the user + prefers is used consistently throughout the file without having to audit + each file hash value separately +- Allows for updating the entire file to a new hash algorithm without running + the risk of accidentally leaving an old hash value in the file +- Lockers SHOULD specify a hash algorithm that is as least as strong as + `SHA-256 `__ + + ``dependencies`` ''''''''''''''' @@ -234,7 +250,11 @@ other is disallowed. '''''''''''''''''''''''''''''''' - String -- Holds the `version specifiers`_ for Python version compatibility +- Holds the `version specifiers`_ for Python version compatibility for the + overall package locking +- Provides at-a-glance information to know if the lock file *may* apply to a + version of Python instead of having to scan the entire file to compile the + same information ``[[package]]`` @@ -288,11 +308,13 @@ other is disallowed. ``package.simple-repo-package-url`` ''''''''''''''''''''''''''''''''''' -- Optional +- Optional (although mutually exclusive with + ``package.files.simple-repo-package-url``) - String -- Storess the `project detail`_ URL from the `Simple Repository API`_ +- Stores the `project detail`_ URL from the `Simple Repository API`_ - Useful for generating Packaging URLs (aka *PURLs*) -- When possible, lockers SHOULD include this key to assist with generating +- When possible, lockers SHOULD include this or + ``package.files.simple-repo-package-url`` to assist with generating `software bill of materials`_ (aka SBOMs) @@ -386,15 +408,12 @@ other is disallowed. ``package.files.simple-repo-package-url`` ''''''''''''''''''''''''''''''''''''''''' -- Optional +- Optional (although mutually exclusive with + ``package.simple-repo-package-url``) - String - The value has the same meaning as ``package.simple-repo-package-url`` -- If set, it overrides the value from ``package.simple-repo-package-url`` for - this file **only** -- This is to support :pep:`708` when some files override what's provided by - another `Simple Repository API`_ index -- Lockers SHOULD include this key when appropriate to assist with generating - `software bill of materials`_ (aka SBOMs) +- This key is avaible per-file to support :pep:`708` when some files override + what's provided by another `Simple Repository API`_ index ``package.files.origin`` @@ -411,17 +430,10 @@ other is disallowed. '''''''''''''''''''''' - String -- The hash of the file contents -- The format is ``f"{hashname}={hashvalue}"`` which is the same as the used by - the `Simple Repository API`_ and its HTML form -- Only a single hash value is used to allow the table to be written inline - for readability and compactness purposes -- Using a single string to store both the hash algorithm and value instead of - separate keys for the two values is to make the inline table shorter +- The hash value of the file contents using the hash algorithm specified by + ``hash-algorithm`` - Used by installers to verify the file contents match what the locker worked with -- Lockers SHOULD use a hash algorithm that is as least as strong as - `SHA-256 `__ ``[package.vcs]`` @@ -527,7 +539,8 @@ Expectations for Lockers locking, e.g. supporting a JSON file format which specifies wheel tags and marker values much like in ``[[file-lock]]`` for which multiple files can be specified, which could then be directly recorded in the corresponding - ``[[file-lock]]`` table + ``[[file-lock]]`` table (if it allowed for unambiguous per-file locking + environment selection) .. code-block:: JSON @@ -541,11 +554,11 @@ Expectations for Installers --------------------------- - Installers MAY support installation of non-binary files - (i.e. source distributions and VCS) + (i.e. source distributions, source trees, and VCS) - Installers MUST provide a way to avoid non-binary file installation for reproducibility and security purposes -- Installers SHOULD make it opt-in for using non-binary file installation for a - secure-by-default approach +- Installers SHOULD make it opt-in for using non-binary file installation to + facilitate a secure-by-default approach Installing for per-file locking @@ -598,19 +611,59 @@ An example workflow is: Backwards Compatibility ======================= -[Describe potential impact and severity on pre-existing code.] +Because there is no preexisting lock file format, there are no explicit +backwards-compatibility concerns in terms of Python packaging standards. + +As for packaging tools themselves, that will be a per-tool decision. For tools +that don't document their lock file format, they could choose to simply start +using the format internally and then transition to saving their lock files with +a name supported by this PEP. For tools with a preexisting, documented format, +they could provide an option to choose which format to emit. Security Implications ===================== -[How could a malicious user take advantage of this new feature?] +The hope is that by standardizing on a lock file format that starts from a +security-first posture it will help make overall packaging installation safer. +However, this PEP does not solve all potential security concerns. + +One potential concern is tampering with a lock file. If a lock file is not kept +in source control and properly audited, a bad actor could change the file in +nefarious ways (e.g. point to a malware version of a package). Tampering could +also occur in transit to e.g. a cloud provider who will perform an installation +on the user's behalf. Both could be mitigated by signing the lock file either +within the file in a ``[tool]`` entry or via a side channel external to the lock +file itself. + +This PEP does not do anything to prevent a user from installing an incorrect +package. While including many details to help in audting a package's inclusion, +there isn't any mechanism to stop e.g. name confusion attacks via typosquatting. +Lockers may be able to provide some UX to help with this (e.g. by providing +download counts for a package). How to Teach This ================= -[How to teach users, new and experienced, how to apply the PEP to their work.] +Users should be informed that when they ask to install some package, that +package may have its own dependencies, those dependencies may have dependencies, +and so on. Without writing down what gets installed as part of installing the +package they requested, things could change from underneatch them (e.g. package +versions). Changes to the underlying dependencies can lead to accidental +breakage of their code. Lock files help deal with that by providing a way to +write down what was installed. + +Having what to install written down also helps in collabortaing with others. By +agreeing to a lock file's contents, everyone to end up with the same packages +installed. This helps make sure no one relies on e.g. an API that's only +available in a certain version that not everyone working on the project has +installed. + +Lock files also help with security by making sure you always get the same files +installed and not a malicious one that someone may have slipped in. It also +lets one be more deliberate in upgrading their dependencies and thus making sure +the change is on purpose and not one slipped in by a bad actor. Reference Implementation @@ -624,17 +677,36 @@ Rejected Ideas [Why certain ideas that were brought while discussing this PEP were not ultimately pursued.] +XXX multiple hashes +XXX hashing the contents of the lock file +XXX only supporting package locking +XXX recording the indexes used +XXX Using ``*.pylock.toml`` as the file name +XXX Using ``*.pylock`` as the file name +XXX Not having a naming convention for the file +XXX Use JSON over TOML +XXX Specifying a new core metadata version that requires consistent metadata + across wheel files for the same package and version +XXX Have the installer do resolution +XXX Recording the creation date of the lock file + Open Issues =========== -[Any points that are still being decided/discussed.] +N/A -Footnotes -========= +Acknowledgements +================ + +https://discuss.python.org/t/lock-files-again-but-this-time-w-sdists/46593/ +Alyssa Coghlan -[A collection of footnotes cited in the PEP, and a place to list non-inline hyperlink targets.] +Paul Moore +Seth Michael Larson +Randy Döring +Ofek Lev Copyright From 3623ccf90891d38d992031f86aaeba204ae98477 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Thu, 28 Mar 2024 19:14:43 -0700 Subject: [PATCH 20/55] Fill out the ACKS section --- peps/pep-9999.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 04de61f9a1d..cae5fe05acc 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -700,13 +700,13 @@ N/A Acknowledgements ================ -https://discuss.python.org/t/lock-files-again-but-this-time-w-sdists/46593/ -Alyssa Coghlan +Thanks to everyone who participated in the discussions in +https://discuss.python.org/t/lock-files-again-but-this-time-w-sdists/46593/, +especially Alyssa Coghlan who probably caused the biggest structural shifts from +the initial proposal. -Paul Moore -Seth Michael Larson -Randy Döring -Ofek Lev +Also thanks to Randy Döring, Seth Michael Larson, Paul Moore, and Ofek Lev for +providing feedback on a draft version of this PEP. Copyright From 22fc531e26f702ba5054cbd9485c1f504c97d570 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Thu, 28 Mar 2024 19:20:17 -0700 Subject: [PATCH 21/55] Outline rejected ideas --- peps/pep-9999.rst | 33 ++++++++++++++++++++++++--------- 1 file changed, 24 insertions(+), 9 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index cae5fe05acc..ba1863e9a37 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -669,25 +669,40 @@ the change is on purpose and not one slipped in by a bad actor. Reference Implementation ======================== -[Link to any existing implementation and details about its state, e.g. proof-of-concept.] +XXX per-file lockinng via mousebender +XXX package lockinng indirectly via PDM Rejected Ideas ============== -[Why certain ideas that were brought while discussing this PEP were not ultimately pursued.] - -XXX multiple hashes -XXX hashing the contents of the lock file XXX only supporting package locking -XXX recording the indexes used +XXX Specifying a new core metadata version that requires consistent metadata + across wheel files for the same package and version +XXX Have the installer do resolution + + +File naming +----------- + XXX Using ``*.pylock.toml`` as the file name XXX Using ``*.pylock`` as the file name XXX Not having a naming convention for the file + + +File format +----------- + XXX Use JSON over TOML -XXX Specifying a new core metadata version that requires consistent metadata - across wheel files for the same package and version -XXX Have the installer do resolution +XXX Use YAML over TOML + + +Other keys +---------- + +XXX multiple hashes +XXX hashing the contents of the lock file +XXX recording the indexes used XXX Recording the creation date of the lock file From 3182912ca8e4897403df868d93cb1a00d826ccfc Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Fri, 29 Mar 2024 22:04:53 -0700 Subject: [PATCH 22/55] Add a rejected idea --- peps/pep-9999.rst | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index ba1863e9a37..9dbd3a5a16d 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -669,14 +669,22 @@ the change is on purpose and not one slipped in by a bad actor. Reference Implementation ======================== -XXX per-file lockinng via mousebender -XXX package lockinng indirectly via PDM +XXX per-file locking via mousebender +XXX package locking indirectly via PDM Rejected Ideas ============== -XXX only supporting package locking +Only support package locking +---------------------------- + +At one point it was suggested to skip per-file locking and only support package +locking as the former was not explicitly supported in the larger Python +ecosystem while the latter was. But because this PEP has taken the position +that security is important and per-file locking is the more secure of the two +options, leaving out per-file locking was never considered. + XXX Specifying a new core metadata version that requires consistent metadata across wheel files for the same package and version XXX Have the installer do resolution From 0e1df0ce135f40bfdf84c80420d3f51cf958c174 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Fri, 24 May 2024 08:44:37 -0700 Subject: [PATCH 23/55] Record the rejected idea of new core metadata version for metadata consistency --- peps/pep-9999.rst | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 9dbd3a5a16d..bab0c504c3b 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -685,8 +685,20 @@ ecosystem while the latter was. But because this PEP has taken the position that security is important and per-file locking is the more secure of the two options, leaving out per-file locking was never considered. -XXX Specifying a new core metadata version that requires consistent metadata - across wheel files for the same package and version + +Specifying a new core metadata version that requires consistent metadata across files +------------------------------------------------------------------------------------- + +At one point, to handle the issue of metadata varying between files and thus +require examining every released file for a package and version for accurate +locking results, the idea was floated to introduce a new core metadata version +which would require all metadata for all wheel files be the same for a single +version of a package. Ultimately, though, it was deemed unnecessary as this PEP +will put pressure on people to make files consistent for performance reasons or +to make indexes provide all the metadata separate from the wheel files +themselves. As well, there's no easy enforcement mechanism, and so community +expectation would work as well as a new metadata version. + XXX Have the installer do resolution From 48811ee188a30a7a4e07a477fe2ac5b5346905e0 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Sat, 25 May 2024 08:42:07 -0700 Subject: [PATCH 24/55] Answer the rejected idea of having the installer do resolution --- peps/pep-9999.rst | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index bab0c504c3b..1f8fa072f9e 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -699,7 +699,22 @@ to make indexes provide all the metadata separate from the wheel files themselves. As well, there's no easy enforcement mechanism, and so community expectation would work as well as a new metadata version. -XXX Have the installer do resolution + +Have the installer do dependency resolution +------------------------------------------- + +In order to support a format more akin to how Poetry worked when this PEP was +drafted, it was suggested that lockers effectively record the packages and their +versions which may be necessary to make an install work in any possible +scenario, and then the installer resolves what to install. But since that +complicates auditing a lock file by requiring much more mental effort to know +what packages may be installed in any given scenario. Also, one of the Poetry +developers +`suggested `__ +that markers as represented in the package locking approach of this PEP may be +sufficient to cover the needs of Poetry. Not having the installer do a +resolution also simplifies their implementation, centralizing complexity in +lockers. File naming From b1c2fdcf3bd2ed3693335f2ae6b5f6e1de6bd529 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Sat, 25 May 2024 13:43:26 -0700 Subject: [PATCH 25/55] Fill in rejected ideas around file names --- peps/pep-9999.rst | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 1f8fa072f9e..036f12c74a5 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -720,9 +720,32 @@ lockers. File naming ----------- -XXX Using ``*.pylock.toml`` as the file name -XXX Using ``*.pylock`` as the file name -XXX Not having a naming convention for the file +Using ``*.pylock.toml`` as the file name +'''''''''''''''''''''''''''''''''''''''' + +It was proposed to put the ``pylock`` constant part of the file name after the +identifier for the purpose of the lock file. It was decided not to do this so +that lock files would sort together when looking at directory contents instead +of purely based on their purpose which could spread them out in a directory. + + +Using ``*.pylock`` as the file name +''''''''''''''''''''''''''''''''''' + +Not using ``.toml`` as the file extension and instead making it ``.pylock`` +itself was proposed. This was decided against so that code editors would know +how to provide syntax highlighting to a lock file without having special +knowledge about the file extension. + + +Not having a naming convention for the file +''''''''''''''''''''''''''''''''''''''''''' + +Having not requirements or guidance for a lock file's name was considered, but +ultimately rejected. By having a standardized naming convention it makes it easy +to identify a lock file for both a human and a code editor. This helps +facilitate discovery when e.g. a tool wants to know all of the lock files that +are available. File format From 135a13ef0353e7b201b2e84561f111331520ef6e Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Mon, 27 May 2024 08:09:49 -0700 Subject: [PATCH 26/55] Add rejected ideas around the file format --- peps/pep-9999.rst | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 036f12c74a5..52b6a7fefda 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -751,8 +751,21 @@ are available. File format ----------- -XXX Use JSON over TOML -XXX Use YAML over TOML +Use JSON over TOML +'''''''''''''''''' + +Since having a format that is machine-writable was a goal of this PEP, it was +suggested to use JSON. But it was deemed less human-readable than TOML while +not improving on the machine-writable aspect enough to warrant the change. + + +Use YAML over TOML +'''''''''''''''''' + +Some argued that YAML met the machine-writable/human-readable requirement in a +better way than TOML. But as that's subjective and ``pyproject.toml`` already +existed as the human-writable file used by Python packaging standards it was +deemed more important to keep using TOML. Other keys From 450a50f1c7e6b2397cfd7f2883069bd9662f6925 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Mon, 27 May 2024 12:37:44 -0700 Subject: [PATCH 27/55] Finish filling in the rejected ideas --- peps/pep-9999.rst | 57 +++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 53 insertions(+), 4 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 52b6a7fefda..c0fc1f234f9 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -717,6 +717,18 @@ resolution also simplifies their implementation, centralizing complexity in lockers. +Requiring specific hash algorithm support +''''''''''''''''''''''''''''''''''''''''' + +It was proposed to require a baseline hash algorithm for the files. This was +rejected as no other Python packaging specification requires specific hash +algorithm support. As well, the minimum hash algorithm suggested may eventually +become an outdated/unsafe suggetion, requiring further updates. In order to +promote using the best algorithm at all times, no baseline is provided to avoid +simply defaulting to the baseline in tools without considering the security +ramifications of that hash algorithm. + + File naming ----------- @@ -771,10 +783,47 @@ deemed more important to keep using TOML. Other keys ---------- -XXX multiple hashes -XXX hashing the contents of the lock file -XXX recording the indexes used -XXX Recording the creation date of the lock file +Multiple hashes per file +'''''''''''''''''''''''' + +An initial version of this PEP proposed supporting multiple hashes per file. The +idea was to allow one to choose which hashing algorithm they wanted to go with +when installing. But upon reflection it seemed like an unnecessary complication +as there was no guarantee the hashes provided would satisfy the user's needs. +As well, if the single hash algorithm used in the lock file wasn't sufficient, +rehashing the files involved as a way to migrate to a different algorithm didn't +seem insurmountable. + + +Hashing the contents of the lock file itself +'''''''''''''''''''''''''''''''''''''''''''' + +Hashing the contents of the bytes of the file and storing hash value within the +file itself was proposed at some point. This was removed to make it easier +when merging changes to the lock file as each merge would have to recalculate +the hash value to avoid a merge conflict. + +Hashing the semantic contents of the file was also proposed, but it would lead +to the same merge conflict issue. + +Regardless of which contents were hashed, either approach could have the hash +value stored outside of the file if such a hash was desired. + + +Recording the creation date of the lock file +'''''''''''''''''''''''''''''''''''''''''''' + +To know how potentially stale the lock file was, an earlier proposal suggested +recording the creation date of the lock file. But for some same merge conflict +reasons as storing the hash of the file contents, this idea was dropped. + + +Recording the packaged indexes used +''''''''''''''''''''''''''''''''''' + +Recording what package indexes were used by the locker to decide what to lock +for was considered. In the end, though, it was rejected as it was deemed +unnecessary bookkeeping. Open Issues From aa7840ff98fb94871b93cf3849b61eda355d9c1c Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Fri, 31 May 2024 14:39:09 -0700 Subject: [PATCH 28/55] Minor tweaks --- peps/pep-9999.rst | 140 +++++++++++++++++++++++++--------------------- 1 file changed, 77 insertions(+), 63 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index c0fc1f234f9..a9a1d59b600 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -14,11 +14,11 @@ Abstract ======== This PEP proposes a new file format for specifying what dependencies to -(possibly) install into a Python environment for consitent installation +(possibly) install into a Python environment for consistent installation repoducibility. The format is designed to be human-readable but -machine-generated. Installers consuming the file should also not need to resolve -which listed dependencies to install, but instead evaluate each dependency in -question in isolation. +machine-generated. Installers consuming the file should not need to resolve +which listed dependencies to install based on other packages listed in the lock +file, but instead evaluate each dependency in question in isolation. Motivation @@ -26,23 +26,23 @@ Motivation Currently, there is no standard way to specify what top-level dependencies one would like to see installed into a Python environment and then subsequently -record what war eventually installed in a file (called a *lock file*). +record what was eventually installed in a file (called a *lock file*). Considering there are four well-known solutions to this problem in the community (``pip freeze``, pip-tools_, Poetry_, and PDM_), there seems to be an appetite for lock files in general. -Those tools also vary in what locking scenarios they approach. For instance, +Those tools also vary in what locking scenarios they support. For instance, ``pip freeze`` and pip-tools only generate lock files for the current environment while PDM and Poetry try to lock for *any* environment to some degree. And none of them directly support locking to specific files to install which can be important for some workflows. There's also concerns around secure defaults in the face of supply chain attacks. Finally, not all the formats are -easy to audit to determine what what would be installed into an environment -ahead of time. +easy to audit to determine what would be installed into an environment ahead of +time. The lack of a standard also has some drawbacks. For instance, any tooling that wants to work with lock files must choose which format to support, potentially -leaving users unsupported (e.g. if Dependabot_ chose not to support PDM. +leaving users unsupported (e.g. if Dependabot_ chose not to support PDM, support by cloud providers who can do dependency installations on your behalf, etc.). @@ -50,45 +50,49 @@ etc.). Rationale ========= -The file format is designed to be human-readable. This is -so that the contents of the file can be audited by a human to make sure no -undesired dependencies end up being included in the lock file. It is also to -facilitate understanding what would be installed if the lock file were to be -used without necessitating running a tool, once again to help with auditing. - -The file format is also designed to not require a resolver at install time. Being -able to analyze dependencies in isolation from one another when listed in a lock -file provides a few benefits. First, it supports auditing by making it easy to -figure out if a certain dependency would be installed for a certain environment. -It should also lead to faster installs which are much more frequent than -creating a lock file. Finally, the four tools mentioned in the Motivation -section either already implement this approach of evaluating dependencies in -isolation or have suggested they could in -`Poetry's case `__. - -The format is also designed so that a *locker* which produces the lock file +The format is designed so that a *locker* which produces the lock file and an *installer* which consumes the lock file can be separate tools. This allows for situations such as cloud hosting providers to use their own installer that's optimized for their system which is independent of what locker the user used to create their lock file. +The file format is designed to be human-readable. This is +so that the contents of the file can be audited by a human to make sure no +undesired dependencies end up being included in the lock file. It is also to +facilitate easy understanding of what would be installed if the lock file +without necessitating running a tool, once again to help with auditing. + +The file format is also designed to not require a resolver at install time. +Being able to analyze dependencies in isolation from one another when listed in +a lock file provides a few benefits. First, it supports auditing by making it +easy to +figure out if a certain dependency would be installed for a certain environment +by looking at each dependency in isolation instead of having to reference other +parts of the file contextually. It should also lead to faster installs which are +much more frequent than creating a lock file. Finally, the four tools mentioned +in the Motivation section either already implement this approach of evaluating +dependencies in isolation or have suggested they could in +`Poetry's case `__. + Locking Scenarios ----------------- -The lock file format is designed to support two locking scenarios. +The lock file format is designed to support two locking scenarios. The format +should also be flexible enough that adding support for other locking scenarios +is possible via a separate PEP. Per-file Locking '''''''''''''''' *Per-file locking* operates under the premise that one wants to install exact -same files in matching environments. As such, the lock file specifies the -requirements for an environment and then matches that environment requirements +same files in any matching environment. As such, the lock file specifies the +requirements for an environment and then matches those environment requirements with the files to install. There can be multiple environments specified in a single file, each with their own set of files to install. By specifying the -exact files to install avoids needs performing a resolution to decide what to -install. +exact files to install installers avoid performing a resolution to decide what +to install. The motivation for this approach to locking is for those who have controlled environments that they work with. For instance, if you have specific, controlled @@ -97,30 +101,29 @@ make sure the **same** files are installed in either environment for everyone. This is similar to what ``pip freeze`` and pip-tools_ support, but with more strictness of the exact files as well as incorporating -into the file format the ability to specify the locked files for multiple -environments. +support to specify the locked files for multiple environments in the same file. Package Locking ''''''''''''''' *Package locking* lists the packages and their versions that *may* apply to any -environment being installed for. The of packages and their version are evaluated -individually and independently from any other packages and versions listed in -the file. This allows installation to be linear -- read each package and version -and make an isolated decision as to whether it should be installed -- and thus -not require the installer to perform a resolution to decide what to install. +environment being installed for. The list of packages and their versions are +evaluated individually and independently from any other packages and versions +listed in the file. This allows installation to be linear -- read each package +and version and make an isolated decision as to whether it should be installed. +This avoids requiring the installer to perform a *resolution* (i.e. +determine what to install based on what else is to be installed). The motivation of this approach comes from `PDM lock files `__. By listing the -potential packages and versions that may be installed, what packages may be -installed is controlled for in a way that's easy to reason about. This also -allows for not specifying the exact environments that would be supported by the -lock file so there's more flexibility and potentially easier --- and thus faster -- lock generation. This approach supports scenarios like -open source projects that want to lock what people should use to build the -documentation as open source projects do not necessarily know upfront what -environments their contributors are working from. +potential packages and versions that may be installed, what's installed is +controlled in a way that's easy to reason about. This also allows for not +specifying the exact environments that would be supported by the lock file so +there's more flexibility for what environments are compatible with the lock +file. This approach supports scenarios like open source projects that want to +lock what people should use to build the documentation without knowing upfront +what environments their contributors are working from. As already mentioned, this approach is supported by PDM_. Poetry_ has `shown some interest `__. @@ -133,15 +136,17 @@ File Name --------- A lock file MUST be named ``pylock.toml`` or match the regular expression -``f"pylock\.([A-Z0-9]|[A-Z0-9][A-Z0-9._-]*[A-Z0-9])\.toml"`` (the part between -``pylock.`` and ``.toml`` comes from -`valid non-normalized names `__ -and their usage in :pep:`735`). The use of the ``.toml`` file extension is to -make syntax highlighting in editors easier and to reinforce the fact that the -file format is meant to be human-readable. +``f"pylock\.(.+)\.toml"`` if a name is desired or if multiple lock files exist. +The use of the ``.toml`` file extension is to make syntax highlighting in +editors easier and to reinforce the fact that the file format is meant to be +human-readable. The prefix and suffix of a named file MUST be lowercase for easy +detection and stripping off to find the name, e.g.:: + + if filename.startswith("pylock.") and filename.endswith(".toml"): + name = filename.removeprefix("pylock.").removesuffix(".toml") This PEP has no opinion as to the location of lock files (i.e. in the root or -some subdirectory of a project). +the subdirectory of a project). File Format @@ -159,8 +164,8 @@ other is disallowed. - String - The version of the lock file format -- This PEP specifies the initial version -- and only valid vailue until future - updates to the standard -- as ``"1.0"`` +- This PEP specifies the initial version -- and only valid value until future + updates to the standard change it -- as ``"1.0"`` ``hash-algorithm`` @@ -194,7 +199,7 @@ other is disallowed. - Mutually exclusive with ``[package-lock]`` - Array of tables -- The table's existence infers the use of the per-file lockinng approach +- The array's existence infers the use of the per-file locking approach - An environment that meets all of the specified criteria in the table will be considered compatible with the environment that was locked for - Lockers MUST NOT generate multiple ``[file-lock]`` tables which would be @@ -205,7 +210,7 @@ other is disallowed. '''''''''''''''''' - String -- A unique name for the environment this table represents +- A unique name within the array for the environment this table represents ``[file-lock.marker-values]`` @@ -229,7 +234,8 @@ other is disallowed. - An unordered array of `wheel tags`_ which must be supported by the environment - The array MAY not be exhaustive to allow for a smaller array as well as to help prevent multiple ``[[file-lock]]`` tables being compatible with the - same environment + same environment by having the array be mutually exclusive with other + ``[[file-lock]]`` tables - Lockers SHOULD sort the keys lexicographically to minimize changes when updating the file - Lockers MUST NOT include @@ -243,7 +249,7 @@ other is disallowed. - Table - Mutually exclusive with ``[[file-lock]]`` -- Signifies +- Signifies the use of the package locking approach ``package-lock.requires-python`` @@ -287,12 +293,13 @@ other is disallowed. ``package.multiple-entries`` '''''''''''''''''''''''''''' -- Required when there are multiple entries for the same package - Boolean - If package locking via ``[package-lock]``, then the multiple entries for the same package MUST be mutually exclusive via ``package.marker`` (this is not required for per-file locking as the ``package.*.lock`` entries imply mutual exclusivity) +- Aids in auditing by knowing that there are multiple entries for the same + package that may need to be considered when auditing this specific package ``package.description`` @@ -341,7 +348,7 @@ other is disallowed. - Also helps document why the version restriction in ``package-lock.requires-python`` was chosen - It should not provide useful information for installers as it would be - captured by `package-lock.requires-python` or isn't relevant when + captured by `package-lock.requires-python` and isn't relevant when ``[[file-lock]]`` is used @@ -383,6 +390,7 @@ other is disallowed. - Must be specified if ``[package.vcs]`` is not - Array of tables - Tables can be written inline +- Represents the files to potentially install for the package and version ``package.files.name`` @@ -439,7 +447,8 @@ other is disallowed. ``[package.vcs]`` ''''''''''''''''' -- Must be specified if ``[[package.files]]`` is not +- Must be specified if ``[[package.files]]`` is not (altough may be specified + simultaneously with ``[[package.files]]``) - Table representing the version control system containing the package and version @@ -466,6 +475,8 @@ other is disallowed. - String - The commit ID for the repository which represents the package and version +- The value MUST be immutable for the VCS for security purposes + (e.g. no Git tags) ``package.vcs.lock`` @@ -484,9 +495,12 @@ other is disallowed. ``package.directory`` ''''''''''''''''''''' -- Optional +- Optional and only valid when ``[package-lock]`` is specified - String - A local directory where a source tree for the package and version exists +- Not valid under ``[[file-lock]]`` as this PEP does not make an attempt to + specify a mechanism for verifying file contents have not changed since locking + was performed ``[[package.build-requires]]`` From f6b02ef11f9f9522afe73a031da7e5b4a4329ff6 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Fri, 7 Jun 2024 13:58:05 -0700 Subject: [PATCH 29/55] Finish proofreading --- peps/pep-9999.rst | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index a9a1d59b600..cd8b71b135b 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -204,6 +204,8 @@ other is disallowed. considered compatible with the environment that was locked for - Lockers MUST NOT generate multiple ``[file-lock]`` tables which would be considered compatible for the same environment +- In instances where there would be a conflict but the lock is still desired, + either separate lock files can be written or per-package locking can be used ``file-lock.name`` @@ -234,7 +236,8 @@ other is disallowed. - An unordered array of `wheel tags`_ which must be supported by the environment - The array MAY not be exhaustive to allow for a smaller array as well as to help prevent multiple ``[[file-lock]]`` tables being compatible with the - same environment by having the array be mutually exclusive with other + same environment by having one array being a strict subset of another + ``file-lock.wheel-tags`` entry in the same file ``[[file-lock]]`` tables - Lockers SHOULD sort the keys lexicographically to minimize changes when updating the file @@ -541,8 +544,8 @@ Expectations for Lockers ------------------------ - When creating a lock file for ``[package-lock]``, the locker SHOULD read - the metadata of **all** files listed in ``[[package.files]]`` to make sure all - potential metadata cases are covered + the metadata of **all** files that end up being listed in + ``[[package.files]]`` to make sure all potential metadata cases are covered - If a locker chooses not to check every file for its metadata, the tool MUST either provide the user with the option to have all files checked (whether that is opt-in or out is left up to the tool), or the user is somehow notified @@ -568,7 +571,7 @@ Expectations for Installers --------------------------- - Installers MAY support installation of non-binary files - (i.e. source distributions, source trees, and VCS) + (i.e. source distributions, source trees, and VCS), but are not required to - Installers MUST provide a way to avoid non-binary file installation for reproducibility and security purposes - Installers SHOULD make it opt-in for using non-binary file installation to @@ -586,18 +589,20 @@ An example workflow is: - For the compatible environment, iterate through each entry in ``[[package]]`` - For each ``[[package]]`` entry, iterate through ``[[package.files]]`` to look - for any files with ``file-lock.name`` listed in ``package.files.lock`` -- If a file is found, install it and move on to the next ``[[package]]`` entry +- If a file is found with a matching lock name, add it to the list of candidate + files to insntall and move on to the next ``[[package]]`` entry - If no file is found then check if ``package.vcs.lock`` contains a match (no - match is acceptable) + match is also acceptable) - If a ``[[package.files]]`` contains multiple matching entries an error MUST be raised due to ambiguity for what is to be installed - If multiple ``[[package]]`` entries for the same package have matching files an error MUST be raised due to ambiguity for what is to be installed -- Find and verify the selected files and/or CVS entries based on their hash or +- Find and verify the candidate files and/or CVS entries based on their hash or commit ID as appropriate - If a source distribution or VCS was selected and ``[[package.build-requires]]`` exists, then repeat the above process as appropriate to install the build dependencies necessary to build the package +- Install the candidate files Installing for package locking @@ -610,7 +615,7 @@ An example workflow is: - Iterate through each entry in ``[package]]`` - For each entry, if there's a ``package.marker`` key, evaluate the expression - If the expression is false, then move on - - Otherwise the package entry must be installed + - Otherwise the package entry must be installed somehow - Iterate through the files listed in ``[[package.files]]``, looking for the "best" file to install - If no file is found, check for ``[package.vcs]`` @@ -620,6 +625,7 @@ An example workflow is: - If the match is a source distribution or VCS and ``[[package.build-requires]]`` is provided, repeat the above as appropriate to build the package +- Install the selected files Backwards Compatibility From b35057a985823e97af5e0154c1801f038874b1b4 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 10 Jul 2024 17:50:13 -0700 Subject: [PATCH 30/55] Clarify that installers should never guess which `[[file-lock]]` to install --- peps/pep-9999.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index cd8b71b135b..12488f1d052 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -576,6 +576,8 @@ Expectations for Installers reproducibility and security purposes - Installers SHOULD make it opt-in for using non-binary file installation to facilitate a secure-by-default approach +- Under per-file locking, if what to install is ambiguous then the installer + MUST raise an error Installing for per-file locking @@ -586,6 +588,7 @@ An example workflow is: - Iterate through each ``[[file-lock]]`` table to find the one that applies to the environment being installed for - If no compatible environment is found an error MUST be raised +- If multiple environments are found to be compatible then an error MUST be raised - For the compatible environment, iterate through each entry in ``[[package]]`` - For each ``[[package]]`` entry, iterate through ``[[package.files]]`` to look - for any files with ``file-lock.name`` listed in ``package.files.lock`` From c16848b8640a602bb661ba344c17151ca625316a Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Thu, 11 Jul 2024 10:51:44 -0700 Subject: [PATCH 31/55] reST fix --- peps/pep-9999.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 12488f1d052..b2aa6bfae46 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -187,7 +187,7 @@ other is disallowed. ``dependencies`` -''''''''''''''' +'''''''''''''''' - Array of strings - A listing the `dependency specifiers`_ that act as the input to the lock file, From 15b2c05130be9a7a9622908b80a1d43323300e3f Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Thu, 11 Jul 2024 11:40:13 -0700 Subject: [PATCH 32/55] Fix reST links --- peps/pep-9999.rst | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index b2aa6bfae46..eb7d509dd05 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -874,18 +874,18 @@ This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. -_core metadata: https://packaging.python.org/en/latest/specifications/core-metadata/ -_Dependabot: https://docs.github.com/en/code-security/dependabot -_dependency specifiers: https://packaging.python.org/en/latest/specifications/dependency-specifiers/ -_direct URL reference: https://packaging.python.org/en/latest/specifications/direct-url/ -_environment markers: https://packaging.python.org/en/latest/specifications/dependency-specifiers/#environment-markers -_normalized name: https://packaging.python.org/en/latest/specifications/name-normalization/#name-normalization -_PDM: https://pypi.org/project/pdm/ -_pip-tools: https://pypi.org/project/pip-tools/ -_Poetry: https://python-poetry.org/ -_project detail: https://packaging.python.org/en/latest/specifications/simple-repository-api/#project-detail -_pyproject.toml specification: https://packaging.python.org/en/latest/specifications/pyproject-toml/#pyproject-toml-specification -_Simple Repository API: https://packaging.python.org/en/latest/specifications/simple-repository-api/ -_ software bill of materials: https://www.cisa.gov/sbom -_version specifiers: https://packaging.python.org/en/latest/specifications/version-specifiers/ -_wheel tags: https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/ +.. _core metadata: https://packaging.python.org/en/latest/specifications/core-metadata/ +.. _Dependabot: https://docs.github.com/en/code-security/dependabot +.. _dependency specifiers: https://packaging.python.org/en/latest/specifications/dependency-specifiers/ +.. _direct URL reference: https://packaging.python.org/en/latest/specifications/direct-url/ +.. _environment markers: https://packaging.python.org/en/latest/specifications/dependency-specifiers/#environment-markers +.. _normalized name: https://packaging.python.org/en/latest/specifications/name-normalization/#name-normalization +.. _PDM: https://pypi.org/project/pdm/ +.. _pip-tools: https://pypi.org/project/pip-tools/ +.. _Poetry: https://python-poetry.org/ +.. _project detail: https://packaging.python.org/en/latest/specifications/simple-repository-api/#project-detail +.. _pyproject.toml specification: https://packaging.python.org/en/latest/specifications/pyproject-toml/#pyproject-toml-specification +.. _Simple Repository API: https://packaging.python.org/en/latest/specifications/simple-repository-api/ +.. _ software bill of materials: https://www.cisa.gov/sbom +.. _version specifiers: https://packaging.python.org/en/latest/specifications/version-specifiers/ +.. _wheel tags: https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/ From 040fa852bc8085ad2b915f6875aca0c7b47d7b58 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Thu, 11 Jul 2024 13:37:25 -0700 Subject: [PATCH 33/55] Add a TOML link target --- peps/pep-9999.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index eb7d509dd05..154cce92f32 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -886,6 +886,7 @@ CC0-1.0-Universal license, whichever is more permissive. .. _project detail: https://packaging.python.org/en/latest/specifications/simple-repository-api/#project-detail .. _pyproject.toml specification: https://packaging.python.org/en/latest/specifications/pyproject-toml/#pyproject-toml-specification .. _Simple Repository API: https://packaging.python.org/en/latest/specifications/simple-repository-api/ -.. _ software bill of materials: https://www.cisa.gov/sbom +.. _software bill of materials: https://www.cisa.gov/sbom +.. _TOML: https://toml.io/ .. _version specifiers: https://packaging.python.org/en/latest/specifications/version-specifiers/ .. _wheel tags: https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/ From 902ab69e6ac8a32052db0c53569080b3234c948d Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Tue, 23 Jul 2024 11:35:30 -0700 Subject: [PATCH 34/55] Clarify that the file format is designed to facilitate diff reading --- peps/pep-9999.rst | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 154cce92f32..5a0e4bafbef 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -60,7 +60,9 @@ The file format is designed to be human-readable. This is so that the contents of the file can be audited by a human to make sure no undesired dependencies end up being included in the lock file. It is also to facilitate easy understanding of what would be installed if the lock file -without necessitating running a tool, once again to help with auditing. +without necessitating running a tool, once again to help with auditing. Finally, +the format is designed so that viewing a diff of the file is easy by centralizing +relevant details to minimizing needing to look at other parts of the file. The file format is also designed to not require a resolver at install time. Being able to analyze dependencies in isolation from one another when listed in @@ -275,6 +277,8 @@ other is disallowed. and ``package.version`` by the sort order for `version specifiers`_ - Lockers SHOULD record keys in the same order as written in this PEP to minimmize changes when updating +- Designed so that relevant details as to why a package is included is + in one place to make diff reading easier ``package.name`` @@ -302,7 +306,7 @@ other is disallowed. required for per-file locking as the ``package.*.lock`` entries imply mutual exclusivity) - Aids in auditing by knowing that there are multiple entries for the same - package that may need to be considered when auditing this specific package + package that may need to be considered ``package.description`` From e1ec016d8a3ff8e3cdfebfacaf897d7d6a5dbfb1 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Tue, 23 Jul 2024 18:58:32 -0700 Subject: [PATCH 35/55] Refer to PoCs --- peps/pep-9999.rst | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index 5a0e4bafbef..ae545ddcd1c 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -1,7 +1,6 @@ PEP: 9999 Title: A file format to list Python dependencies for installation reproducibility Author: Brett Cannon -PEP-Delegate: Discussions-To: Status: Draft Type: Standards Track @@ -696,8 +695,16 @@ the change is on purpose and not one slipped in by a bad actor. Reference Implementation ======================== -XXX per-file locking via mousebender -XXX package locking indirectly via PDM +A rough proof-of-concept for per-file locking can be found at +https://github.com/brettcannon/mousebender/tree/pep . An example lock file can +be seen at +https://github.com/brettcannon/mousebender/blob/pep/pylock.example.toml . + +For per-package locking, PDM_ indirectly proves the approach works as this PEP +maintains equivalent data as PDM does for its lock files (which was inspired by +Poetry_). Some of the details of PDM's approach are covered in +https://frostming.com/en/2024/pdm-lockfile/ and +https://frostming.com/en/2024/pdm-lock-strategy/ . Rejected Ideas From 08e6fdbedf41c72a45547b9e2bbf73529ee26d85 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Tue, 23 Jul 2024 18:59:23 -0700 Subject: [PATCH 36/55] Tweak some wording --- peps/pep-9999.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-9999.rst b/peps/pep-9999.rst index ae545ddcd1c..b464d099f1e 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-9999.rst @@ -701,8 +701,8 @@ be seen at https://github.com/brettcannon/mousebender/blob/pep/pylock.example.toml . For per-package locking, PDM_ indirectly proves the approach works as this PEP -maintains equivalent data as PDM does for its lock files (which was inspired by -Poetry_). Some of the details of PDM's approach are covered in +maintains equivalent data as PDM does for its lock files (whose format was +inspired by Poetry_). Some of the details of PDM's approach are covered in https://frostming.com/en/2024/pdm-lockfile/ and https://frostming.com/en/2024/pdm-lock-strategy/ . From a3a14854d71aa3d2b97c130425dac5b23a6cd903 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 24 Jul 2024 12:08:51 -0700 Subject: [PATCH 37/55] Pick a PEP number --- peps/{pep-9999.rst => pep-0751.rst} | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) rename peps/{pep-9999.rst => pep-0751.rst} (99%) diff --git a/peps/pep-9999.rst b/peps/pep-0751.rst similarity index 99% rename from peps/pep-9999.rst rename to peps/pep-0751.rst index b464d099f1e..fdd57c7d75b 100644 --- a/peps/pep-9999.rst +++ b/peps/pep-0751.rst @@ -1,12 +1,10 @@ -PEP: 9999 +PEP: 751 Title: A file format to list Python dependencies for installation reproducibility Author: Brett Cannon -Discussions-To: Status: Draft Type: Standards Track Topic: Packaging -Created: -Post-History: +Created: 2024-07-24 Replaces: 665 Abstract @@ -565,8 +563,8 @@ Expectations for Lockers .. code-block:: JSON { - "marker-values": {...} - "wheel-tags": [...] + "marker-values": {"": ""}, + "wheel-tags": [""] } From 462b7e060c3b0bc71df035f369d42dc4556597ac Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 24 Jul 2024 12:10:59 -0700 Subject: [PATCH 38/55] Update CODEOWNERS --- .github/CODEOWNERS | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 8eb380bb2a7..362c88d5bcc 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -629,6 +629,8 @@ peps/pep-0749.rst @JelleZijlstra # ... peps/pep-0747.rst @JelleZijlstra # ... +peps/pep-0751.rst @brettcannon +# ... # peps/pep-0754.rst # ... peps/pep-0789.rst @njsmith From 75d661550d3295b6b5982f28a39ab32c66dcd593 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 24 Jul 2024 12:14:13 -0700 Subject: [PATCH 39/55] Clean up CODEOWNERS --- .github/CODEOWNERS | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 362c88d5bcc..1db3def14e1 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -625,10 +625,10 @@ peps/pep-0743.rst @vstinner peps/pep-0744.rst @brandtbucher peps/pep-0745.rst @hugovk peps/pep-0746.rst @JelleZijlstra -peps/pep-0749.rst @JelleZijlstra -# ... peps/pep-0747.rst @JelleZijlstra # ... +peps/pep-0749.rst @JelleZijlstra +# ... peps/pep-0751.rst @brettcannon # ... # peps/pep-0754.rst From dec8120a468872135cc94ea37ab20ae61519b16c Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 24 Jul 2024 12:16:02 -0700 Subject: [PATCH 40/55] Fix lint failures --- peps/pep-0751.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index fdd57c7d75b..b54b079bdff 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -352,7 +352,7 @@ other is disallowed. - Also helps document why the version restriction in ``package-lock.requires-python`` was chosen - It should not provide useful information for installers as it would be - captured by `package-lock.requires-python` and isn't relevant when + captured by ``package-lock.requires-python`` and isn't relevant when ``[[file-lock]]`` is used @@ -614,8 +614,8 @@ Installing for package locking An example workflow is: -- Verify that the environment is compatible with `package-lock.requires-python`; - if it isn't an error MUST be raised +- Verify that the environment is compatible with + ``package-lock.requires-python``; if it isn't an error MUST be raised - Iterate through each entry in ``[package]]`` - For each entry, if there's a ``package.marker`` key, evaluate the expression - If the expression is false, then move on From a4d1840fd3a400ba053e55b8f86970b93f069e1a Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 24 Jul 2024 12:17:12 -0700 Subject: [PATCH 41/55] Fix another lint failure --- peps/pep-0751.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index b54b079bdff..5a6e2e20f58 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -4,7 +4,7 @@ Author: Brett Cannon Status: Draft Type: Standards Track Topic: Packaging -Created: 2024-07-24 +Created: 24-Jul-2024 Replaces: 665 Abstract From b0d48777e34be292ef7c8633ce86b2e4d4208bff Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 24 Jul 2024 12:55:34 -0700 Subject: [PATCH 42/55] Nest sections more --- peps/pep-0751.rst | 106 ++++++++++++++++++++++++++++------------------ 1 file changed, 65 insertions(+), 41 deletions(-) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index 5a6e2e20f58..160a463a9e1 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -7,6 +7,7 @@ Topic: Packaging Created: 24-Jul-2024 Replaces: 665 +======== Abstract ======== @@ -18,6 +19,7 @@ which listed dependencies to install based on other packages listed in the lock file, but instead evaluate each dependency in question in isolation. +========== Motivation ========== @@ -44,6 +46,7 @@ support by cloud providers who can do dependency installations on your behalf, etc.). +========= Rationale ========= @@ -74,6 +77,7 @@ dependencies in isolation or have suggested they could in `Poetry's case `__. +----------------- Locking Scenarios ----------------- @@ -83,7 +87,7 @@ is possible via a separate PEP. Per-file Locking -'''''''''''''''' +================ *Per-file locking* operates under the premise that one wants to install exact same files in any matching environment. As such, the lock file specifies the @@ -104,7 +108,7 @@ support to specify the locked files for multiple environments in the same file. Package Locking -''''''''''''''' +=============== *Package locking* lists the packages and their versions that *may* apply to any environment being installed for. The list of packages and their versions are @@ -128,9 +132,11 @@ As already mentioned, this approach is supported by PDM_. Poetry_ has `shown some interest `__. +============= Specification ============= +--------- File Name --------- @@ -148,6 +154,7 @@ This PEP has no opinion as to the location of lock files (i.e. in the root or the subdirectory of a project). +----------- File Format ----------- @@ -159,7 +166,7 @@ other is disallowed. ``version`` -''''''''''' +=========== - String - The version of the lock file format @@ -168,7 +175,7 @@ other is disallowed. ``hash-algorithm`` -'''''''''''''''''' +================== - String - The name of the hash algorithm used for calculating all hash values @@ -186,7 +193,7 @@ other is disallowed. ``dependencies`` -'''''''''''''''' +================ - Array of strings - A listing the `dependency specifiers`_ that act as the input to the lock file, @@ -194,7 +201,7 @@ other is disallowed. ``[[file-lock]]`` -''''''''''''''''' +================= - Mutually exclusive with ``[package-lock]`` - Array of tables @@ -208,14 +215,14 @@ other is disallowed. ``file-lock.name`` -'''''''''''''''''' +------------------ - String - A unique name within the array for the environment this table represents ``[file-lock.marker-values]`` -''''''''''''''''''''''''''''' +----------------------------- - Optional - Table of strings @@ -228,7 +235,7 @@ other is disallowed. ``file-lock.wheel-tags`` -'''''''''''''''''''''''' +------------------------ - Optional - Array of strings @@ -247,7 +254,7 @@ other is disallowed. ``[package-lock]`` -'''''''''''''''''' +================== - Table - Mutually exclusive with ``[[file-lock]]`` @@ -255,7 +262,7 @@ other is disallowed. ``package-lock.requires-python`` -'''''''''''''''''''''''''''''''' +-------------------------------- - String - Holds the `version specifiers`_ for Python version compatibility for the @@ -266,7 +273,7 @@ other is disallowed. ``[[package]]`` -''''''''''''''' +=============== - Array of tables - The array contains all data on the locked package versions @@ -279,7 +286,7 @@ other is disallowed. ``package.name`` -'''''''''''''''' +---------------- - String - The `normalized name`_ of the package @@ -287,7 +294,7 @@ other is disallowed. ``package.version`` -''''''''''''''''''' +------------------- - String - The version of the package @@ -295,7 +302,7 @@ other is disallowed. ``package.multiple-entries`` -'''''''''''''''''''''''''''' +---------------------------- - Boolean - If package locking via ``[package-lock]``, then the multiple entries for the @@ -307,7 +314,7 @@ other is disallowed. ``package.description`` -''''''''''''''''''''''' +----------------------- - Optional - String @@ -317,7 +324,7 @@ other is disallowed. ``package.simple-repo-package-url`` -''''''''''''''''''''''''''''''''''' +----------------------------------- - Optional (although mutually exclusive with ``package.files.simple-repo-package-url``) @@ -330,7 +337,7 @@ other is disallowed. ``package.marker`` -'''''''''''''''''' +------------------ - Optional - String @@ -342,7 +349,7 @@ other is disallowed. ``package.requires-python`` -''''''''''''''''''''''''''' +--------------------------- - Optional - String @@ -357,7 +364,7 @@ other is disallowed. ``package.dependents`` -'''''''''''''''''''''' +---------------------- - Optional - Array of strings @@ -368,7 +375,7 @@ other is disallowed. ``package.dependencies`` -'''''''''''''''''''''''' +------------------------ - Optional - Array of strings @@ -381,7 +388,7 @@ other is disallowed. ``package.direct`` -'''''''''''''''''' +------------------ - Optional - Boolean @@ -389,7 +396,7 @@ other is disallowed. ``[[package.files]]`` -''''''''''''''''''''' +--------------------- - Must be specified if ``[package.vcs]`` is not - Array of tables @@ -449,7 +456,7 @@ other is disallowed. ``[package.vcs]`` -''''''''''''''''' +----------------- - Must be specified if ``[[package.files]]`` is not (altough may be specified simultaneously with ``[[package.files]]``) @@ -484,7 +491,7 @@ other is disallowed. ``package.vcs.lock`` -''''''''''''''''''''''''' +'''''''''''''''''''' - Required when ``[[file-lock]]`` is used - An array of strings @@ -497,7 +504,7 @@ other is disallowed. ``package.directory`` -''''''''''''''''''''' +--------------------- - Optional and only valid when ``[package-lock]`` is specified - String @@ -508,7 +515,7 @@ other is disallowed. ``[[package.build-requires]]`` -'''''''''''''''''''''''''''''' +------------------------------ - Optional - An array of tables whose structure matches that of ``[[package]]`` @@ -526,7 +533,7 @@ other is disallowed. ``[package.tool]`` -'''''''''''''''''' +------------------ - Optional - Same usage as that of the equivalent table from the @@ -534,13 +541,14 @@ other is disallowed. ``[tool]`` -'''''''''' +========== - Optional - Same usage as that of the equivalent table from the `pyproject.toml specification`_ +------------------------ Expectations for Lockers ------------------------ @@ -568,6 +576,7 @@ Expectations for Lockers } +--------------------------- Expectations for Installers --------------------------- @@ -582,7 +591,7 @@ Expectations for Installers Installing for per-file locking -''''''''''''''''''''''''''''''' +=============================== An example workflow is: @@ -610,7 +619,7 @@ An example workflow is: Installing for package locking -'''''''''''''''''''''''''''''' +============================== An example workflow is: @@ -632,6 +641,7 @@ An example workflow is: - Install the selected files +======================= Backwards Compatibility ======================= @@ -645,6 +655,7 @@ a name supported by this PEP. For tools with a preexisting, documented format, they could provide an option to choose which format to emit. +===================== Security Implications ===================== @@ -667,6 +678,7 @@ Lockers may be able to provide some UX to help with this (e.g. by providing download counts for a package). +================= How to Teach This ================= @@ -690,6 +702,7 @@ lets one be more deliberate in upgrading their dependencies and thus making sure the change is on purpose and not one slipped in by a bad actor. +======================== Reference Implementation ======================== @@ -705,9 +718,11 @@ https://frostming.com/en/2024/pdm-lockfile/ and https://frostming.com/en/2024/pdm-lock-strategy/ . +============== Rejected Ideas ============== +---------------------------- Only support package locking ---------------------------- @@ -718,6 +733,7 @@ that security is important and per-file locking is the more secure of the two options, leaving out per-file locking was never considered. +------------------------------------------------------------------------------------- Specifying a new core metadata version that requires consistent metadata across files ------------------------------------------------------------------------------------- @@ -732,6 +748,7 @@ themselves. As well, there's no easy enforcement mechanism, and so community expectation would work as well as a new metadata version. +------------------------------------------- Have the installer do dependency resolution ------------------------------------------- @@ -749,8 +766,9 @@ resolution also simplifies their implementation, centralizing complexity in lockers. +----------------------------------------- Requiring specific hash algorithm support -''''''''''''''''''''''''''''''''''''''''' +----------------------------------------- It was proposed to require a baseline hash algorithm for the files. This was rejected as no other Python packaging specification requires specific hash @@ -761,11 +779,12 @@ simply defaulting to the baseline in tools without considering the security ramifications of that hash algorithm. +----------- File naming ----------- Using ``*.pylock.toml`` as the file name -'''''''''''''''''''''''''''''''''''''''' +======================================== It was proposed to put the ``pylock`` constant part of the file name after the identifier for the purpose of the lock file. It was decided not to do this so @@ -774,7 +793,7 @@ of purely based on their purpose which could spread them out in a directory. Using ``*.pylock`` as the file name -''''''''''''''''''''''''''''''''''' +=================================== Not using ``.toml`` as the file extension and instead making it ``.pylock`` itself was proposed. This was decided against so that code editors would know @@ -783,7 +802,7 @@ knowledge about the file extension. Not having a naming convention for the file -''''''''''''''''''''''''''''''''''''''''''' +=========================================== Having not requirements or guidance for a lock file's name was considered, but ultimately rejected. By having a standardized naming convention it makes it easy @@ -792,11 +811,12 @@ facilitate discovery when e.g. a tool wants to know all of the lock files that are available. +----------- File format ----------- Use JSON over TOML -'''''''''''''''''' +================== Since having a format that is machine-writable was a goal of this PEP, it was suggested to use JSON. But it was deemed less human-readable than TOML while @@ -804,7 +824,7 @@ not improving on the machine-writable aspect enough to warrant the change. Use YAML over TOML -'''''''''''''''''' +================== Some argued that YAML met the machine-writable/human-readable requirement in a better way than TOML. But as that's subjective and ``pyproject.toml`` already @@ -812,11 +832,12 @@ existed as the human-writable file used by Python packaging standards it was deemed more important to keep using TOML. +---------- Other keys ---------- Multiple hashes per file -'''''''''''''''''''''''' +======================== An initial version of this PEP proposed supporting multiple hashes per file. The idea was to allow one to choose which hashing algorithm they wanted to go with @@ -828,7 +849,7 @@ seem insurmountable. Hashing the contents of the lock file itself -'''''''''''''''''''''''''''''''''''''''''''' +============================================ Hashing the contents of the bytes of the file and storing hash value within the file itself was proposed at some point. This was removed to make it easier @@ -843,7 +864,7 @@ value stored outside of the file if such a hash was desired. Recording the creation date of the lock file -'''''''''''''''''''''''''''''''''''''''''''' +============================================ To know how potentially stale the lock file was, an earlier proposal suggested recording the creation date of the lock file. But for some same merge conflict @@ -851,19 +872,21 @@ reasons as storing the hash of the file contents, this idea was dropped. Recording the packaged indexes used -''''''''''''''''''''''''''''''''''' +=================================== Recording what package indexes were used by the locker to decide what to lock for was considered. In the end, though, it was rejected as it was deemed unnecessary bookkeeping. +=========== Open Issues =========== N/A +================ Acknowledgements ================ @@ -876,6 +899,7 @@ Also thanks to Randy Döring, Seth Michael Larson, Paul Moore, and Ofek Lev for providing feedback on a draft version of this PEP. +========= Copyright ========= From 1b39e4937810027f090ecbdb3c3570470b475333 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 24 Jul 2024 13:29:17 -0700 Subject: [PATCH 43/55] Accept/address some comments --- peps/pep-0751.rst | 45 +++++++++++++++++++++------------------------ 1 file changed, 21 insertions(+), 24 deletions(-) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index 160a463a9e1..6a33a0b22ec 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -13,10 +13,10 @@ Abstract This PEP proposes a new file format for specifying what dependencies to (possibly) install into a Python environment for consistent installation -repoducibility. The format is designed to be human-readable but -machine-generated. Installers consuming the file should not need to resolve -which listed dependencies to install based on other packages listed in the lock -file, but instead evaluate each dependency in question in isolation. +reproducibility. The format is designed to be human-readable but +machine-generated. Installers consuming the file should be able to +evaluate each package in question in isolation, with no need for +dependency resolution at install-time. ========== @@ -26,7 +26,7 @@ Motivation Currently, there is no standard way to specify what top-level dependencies one would like to see installed into a Python environment and then subsequently record what was eventually installed in a file (called a *lock file*). -Considering there are four well-known solutions to this problem in the +Considering there are at least four well-known solutions to this problem in the community (``pip freeze``, pip-tools_, Poetry_, and PDM_), there seems to be an appetite for lock files in general. @@ -34,10 +34,10 @@ Those tools also vary in what locking scenarios they support. For instance, ``pip freeze`` and pip-tools only generate lock files for the current environment while PDM and Poetry try to lock for *any* environment to some degree. And none of them directly support locking to specific files to install -which can be important for some workflows. There's also concerns around secure -defaults in the face of supply chain attacks. Finally, not all the formats are -easy to audit to determine what would be installed into an environment ahead of -time. +which can be important for some workflows. There's also concerns around the lack +of secure defaults in the face of supply chain attacks (e.g., always including +hashes for files). Finally, not all the formats are easy to audit to determine +what would be installed into an environment ahead of time. The lack of a standard also has some drawbacks. For instance, any tooling that wants to work with lock files must choose which format to support, potentially @@ -62,19 +62,18 @@ undesired dependencies end up being included in the lock file. It is also to facilitate easy understanding of what would be installed if the lock file without necessitating running a tool, once again to help with auditing. Finally, the format is designed so that viewing a diff of the file is easy by centralizing -relevant details to minimizing needing to look at other parts of the file. +relevant details. The file format is also designed to not require a resolver at install time. Being able to analyze dependencies in isolation from one another when listed in a lock file provides a few benefits. First, it supports auditing by making it -easy to -figure out if a certain dependency would be installed for a certain environment -by looking at each dependency in isolation instead of having to reference other -parts of the file contextually. It should also lead to faster installs which are -much more frequent than creating a lock file. Finally, the four tools mentioned -in the Motivation section either already implement this approach of evaluating -dependencies in isolation or have suggested they could in -`Poetry's case `__. +easy to figure out if a certain dependency would be installed for a certain +environment without needing to reference other parts of the file contextually. +It should also lead to faster installs which are much more frequent than +creating a lock file. Finally, the four tools mentioned in the Motivation_ +section either already implement this approach of evaluating dependencies in +isolation or have suggested they could (in +`Poetry's case `__). ----------------- @@ -89,19 +88,17 @@ is possible via a separate PEP. Per-file Locking ================ -*Per-file locking* operates under the premise that one wants to install exact -same files in any matching environment. As such, the lock file specifies the -requirements for an environment and then matches those environment requirements +*Per-file locking* operates under the premise that one wants to install exactly +the same files in any matching environment. As such, the lock file specifies the with the files to install. There can be multiple environments specified in a single file, each with their own set of files to install. By specifying the -exact files to install installers avoid performing a resolution to decide what +exact files to install, installers avoid performing any resolution to decide what to install. The motivation for this approach to locking is for those who have controlled environments that they work with. For instance, if you have specific, controlled development and production environments then you can use per-file locking to -make sure the **same** files are installed in either environment for everyone. - +make sure the **same** files are installed in both environments for everyone. This is similar to what ``pip freeze`` and pip-tools_ support, but with more strictness of the exact files as well as incorporating support to specify the locked files for multiple environments in the same file. From 9031e5c82e5b5d0b1bbf74d0231ea3d7be7c6fba Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 24 Jul 2024 14:06:58 -0700 Subject: [PATCH 44/55] Apply suggestions from code review Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Jelle Zijlstra --- peps/pep-0751.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index 6a33a0b22ec..c8a4fb6b826 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -121,12 +121,12 @@ potential packages and versions that may be installed, what's installed is controlled in a way that's easy to reason about. This also allows for not specifying the exact environments that would be supported by the lock file so there's more flexibility for what environments are compatible with the lock -file. This approach supports scenarios like open source projects that want to +file. This approach supports scenarios like open-source projects that want to lock what people should use to build the documentation without knowing upfront what environments their contributors are working from. As already mentioned, this approach is supported by PDM_. Poetry_ has -`shown some interest `__. +`shown some interest `__. ============= @@ -137,8 +137,8 @@ Specification File Name --------- -A lock file MUST be named ``pylock.toml`` or match the regular expression -``f"pylock\.(.+)\.toml"`` if a name is desired or if multiple lock files exist. +A lock file MUST be named :file:`pylock.toml` or match the regular expression +``r"pylock\.(.+)\.toml"`` if a name for the lock file is desired or if multiple lock files exist. The use of the ``.toml`` file extension is to make syntax highlighting in editors easier and to reinforce the fact that the file format is meant to be human-readable. The prefix and suffix of a named file MUST be lowercase for easy @@ -193,7 +193,7 @@ other is disallowed. ================ - Array of strings -- A listing the `dependency specifiers`_ that act as the input to the lock file, +- A listing of the `dependency specifiers`_ that act as the input to the lock file, representing the direct, top-level dependencies to be installed From fd139c339e8e59e50502bd1b8ea8c9f6548b54b0 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 24 Jul 2024 14:08:13 -0700 Subject: [PATCH 45/55] Apply suggestions from code review Co-authored-by: Jelle Zijlstra Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --- peps/pep-0751.rst | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index c8a4fb6b826..4d5e311d034 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -202,7 +202,7 @@ other is disallowed. - Mutually exclusive with ``[package-lock]`` - Array of tables -- The array's existence infers the use of the per-file locking approach +- The array's existence implies the use of the per-file locking approach - An environment that meets all of the specified criteria in the table will be considered compatible with the environment that was locked for - Lockers MUST NOT generate multiple ``[file-lock]`` tables which would be @@ -278,7 +278,7 @@ other is disallowed. and ``package.version`` by the sort order for `version specifiers`_ - Lockers SHOULD record keys in the same order as written in this PEP to minimmize changes when updating -- Designed so that relevant details as to why a package is included is +- Designed so that relevant details as to why a package is included are in one place to make diff reading easier @@ -327,7 +327,7 @@ other is disallowed. ``package.files.simple-repo-package-url``) - String - Stores the `project detail`_ URL from the `Simple Repository API`_ -- Useful for generating Packaging URLs (aka *PURLs*) +- Useful for generating Packaging URLs (aka PURLs) - When possible, lockers SHOULD include this or ``package.files.simple-repo-package-url`` to assist with generating `software bill of materials`_ (aka SBOMs) @@ -376,7 +376,7 @@ other is disallowed. - Optional - Array of strings -- A record the dependencies of the package and version +- A record of the dependencies of the package and version - Useful in analyzing why a package happens to be listed in the file for auditing purposes - This does not provide information which influences the installer as @@ -428,7 +428,7 @@ other is disallowed. ``package.simple-repo-package-url``) - String - The value has the same meaning as ``package.simple-repo-package-url`` -- This key is avaible per-file to support :pep:`708` when some files override +- This key is available per-file to support :pep:`708` when some files override what's provided by another `Simple Repository API`_ index @@ -455,7 +455,7 @@ other is disallowed. ``[package.vcs]`` ----------------- -- Must be specified if ``[[package.files]]`` is not (altough may be specified +- Must be specified if ``[[package.files]]`` is not (although may be specified simultaneously with ``[[package.files]]``) - Table representing the version control system containing the package and version @@ -581,7 +581,7 @@ Expectations for Installers (i.e. source distributions, source trees, and VCS), but are not required to - Installers MUST provide a way to avoid non-binary file installation for reproducibility and security purposes -- Installers SHOULD make it opt-in for using non-binary file installation to +- Installers SHOULD make it opt-in to use non-binary file installation to facilitate a secure-by-default approach - Under per-file locking, if what to install is ambiguous then the installer MUST raise an error @@ -598,7 +598,7 @@ An example workflow is: - If multiple environments are found to be compatible then an error MUST be raised - For the compatible environment, iterate through each entry in ``[[package]]`` - For each ``[[package]]`` entry, iterate through ``[[package.files]]`` to look -- for any files with ``file-lock.name`` listed in ``package.files.lock`` + for any files with ``file-lock.name`` listed in ``package.files.lock`` - If a file is found with a matching lock name, add it to the list of candidate files to insntall and move on to the next ``[[package]]`` entry - If no file is found then check if ``package.vcs.lock`` contains a match (no From 89ff321c1c4757f9810c126d998c4ab663a5698b Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 24 Jul 2024 14:09:46 -0700 Subject: [PATCH 46/55] Apply suggestions from code review Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Jelle Zijlstra Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> --- peps/pep-0751.rst | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index 4d5e311d034..ad9ca0a8571 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -600,14 +600,14 @@ An example workflow is: - For each ``[[package]]`` entry, iterate through ``[[package.files]]`` to look for any files with ``file-lock.name`` listed in ``package.files.lock`` - If a file is found with a matching lock name, add it to the list of candidate - files to insntall and move on to the next ``[[package]]`` entry + files to install and move on to the next ``[[package]]`` entry - If no file is found then check if ``package.vcs.lock`` contains a match (no match is also acceptable) - If a ``[[package.files]]`` contains multiple matching entries an error MUST be raised due to ambiguity for what is to be installed - If multiple ``[[package]]`` entries for the same package have matching files an error MUST be raised due to ambiguity for what is to be installed -- Find and verify the candidate files and/or CVS entries based on their hash or +- Find and verify the candidate files and/or VCS entries based on their hash or commit ID as appropriate - If a source distribution or VCS was selected and ``[[package.build-requires]]`` exists, then repeat the above process as @@ -624,12 +624,13 @@ An example workflow is: ``package-lock.requires-python``; if it isn't an error MUST be raised - Iterate through each entry in ``[package]]`` - For each entry, if there's a ``package.marker`` key, evaluate the expression + - If the expression is false, then move on - Otherwise the package entry must be installed somehow - Iterate through the files listed in ``[[package.files]]``, looking for the "best" file to install - If no file is found, check for ``[package.vcs]`` -- If not match is found, an error MUST be raised +- If no match is found, an error MUST be raised - Find and verify the selected files and/or CVS entries based on their hash or commit ID as appropriate - If the match is a source distribution or VCS and From a914c12ed81473316a3124745d3e66ea74223203 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 24 Jul 2024 14:10:44 -0700 Subject: [PATCH 47/55] Apply suggestions from code review Co-authored-by: Jelle Zijlstra Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --- peps/pep-0751.rst | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index ad9ca0a8571..074b57f99bb 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -631,7 +631,7 @@ An example workflow is: "best" file to install - If no file is found, check for ``[package.vcs]`` - If no match is found, an error MUST be raised -- Find and verify the selected files and/or CVS entries based on their hash or +- Find and verify the selected files and/or VCS entries based on their hash or commit ID as appropriate - If the match is a source distribution or VCS and ``[[package.build-requires]]`` is provided, repeat the above as appropriate to @@ -670,7 +670,7 @@ within the file in a ``[tool]`` entry or via a side channel external to the lock file itself. This PEP does not do anything to prevent a user from installing an incorrect -package. While including many details to help in audting a package's inclusion, +package. While including many details to help in auditing a package's inclusion, there isn't any mechanism to stop e.g. name confusion attacks via typosquatting. Lockers may be able to provide some UX to help with this (e.g. by providing download counts for a package). @@ -683,13 +683,13 @@ How to Teach This Users should be informed that when they ask to install some package, that package may have its own dependencies, those dependencies may have dependencies, and so on. Without writing down what gets installed as part of installing the -package they requested, things could change from underneatch them (e.g. package +package they requested, things could change from underneath them (e.g. package versions). Changes to the underlying dependencies can lead to accidental breakage of their code. Lock files help deal with that by providing a way to write down what was installed. -Having what to install written down also helps in collabortaing with others. By -agreeing to a lock file's contents, everyone to end up with the same packages +Having what to install written down also helps in collaborating with others. By +agreeing to a lock file's contents, everyone ends up with the same packages installed. This helps make sure no one relies on e.g. an API that's only available in a certain version that not everyone working on the project has installed. @@ -705,15 +705,15 @@ Reference Implementation ======================== A rough proof-of-concept for per-file locking can be found at -https://github.com/brettcannon/mousebender/tree/pep . An example lock file can +https://github.com/brettcannon/mousebender/tree/pep. An example lock file can be seen at -https://github.com/brettcannon/mousebender/blob/pep/pylock.example.toml . +https://github.com/brettcannon/mousebender/blob/pep/pylock.example.toml. For per-package locking, PDM_ indirectly proves the approach works as this PEP maintains equivalent data as PDM does for its lock files (whose format was inspired by Poetry_). Some of the details of PDM's approach are covered in https://frostming.com/en/2024/pdm-lockfile/ and -https://frostming.com/en/2024/pdm-lock-strategy/ . +https://frostming.com/en/2024/pdm-lock-strategy/. ============== @@ -771,7 +771,7 @@ Requiring specific hash algorithm support It was proposed to require a baseline hash algorithm for the files. This was rejected as no other Python packaging specification requires specific hash algorithm support. As well, the minimum hash algorithm suggested may eventually -become an outdated/unsafe suggetion, requiring further updates. In order to +become an outdated/unsafe suggestion, requiring further updates. In order to promote using the best algorithm at all times, no baseline is provided to avoid simply defaulting to the baseline in tools without considering the security ramifications of that hash algorithm. @@ -802,7 +802,7 @@ knowledge about the file extension. Not having a naming convention for the file =========================================== -Having not requirements or guidance for a lock file's name was considered, but +Having no requirements or guidance for a lock file's name was considered, but ultimately rejected. By having a standardized naming convention it makes it easy to identify a lock file for both a human and a code editor. This helps facilitate discovery when e.g. a tool wants to know all of the lock files that From 8c2190967be0341f4bfe1e151825a3a511a337bc Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 24 Jul 2024 14:36:17 -0700 Subject: [PATCH 48/55] Adam likes periods --- peps/pep-0751.rst | 170 +++++++++++++++++++++++----------------------- 1 file changed, 85 insertions(+), 85 deletions(-) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index 074b57f99bb..92de581d2da 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -166,27 +166,27 @@ other is disallowed. =========== - String -- The version of the lock file format +- The version of the lock file format. - This PEP specifies the initial version -- and only valid value until future - updates to the standard change it -- as ``"1.0"`` + updates to the standard change it -- as ``"1.0"``. ``hash-algorithm`` ================== - String -- The name of the hash algorithm used for calculating all hash values +- The name of the hash algorithm used for calculating all hash values. - Only a single hash algorithm is used for the entire file to allow the ``[[package.files]]`` table to be written inline for readability and compactness purposes by only listing a single hash value instead of multiple - values based on multiple hash algorithms + values based on multiple hash algorithms. - Specifying a single hash algorithm guarantees that an algorithm that the user prefers is used consistently throughout the file without having to audit - each file hash value separately + each file hash value separately. - Allows for updating the entire file to a new hash algorithm without running - the risk of accidentally leaving an old hash value in the file + the risk of accidentally leaving an old hash value in the file. - Lockers SHOULD specify a hash algorithm that is as least as strong as - `SHA-256 `__ + `SHA-256 `__. ``dependencies`` @@ -194,28 +194,28 @@ other is disallowed. - Array of strings - A listing of the `dependency specifiers`_ that act as the input to the lock file, - representing the direct, top-level dependencies to be installed + representing the direct, top-level dependencies to be installed. ``[[file-lock]]`` ================= -- Mutually exclusive with ``[package-lock]`` - Array of tables -- The array's existence implies the use of the per-file locking approach +- Mutually exclusive with ``[package-lock]``. +- The array's existence implies the use of the per-file locking approach. - An environment that meets all of the specified criteria in the table will be - considered compatible with the environment that was locked for + considered compatible with the environment that was locked for. - Lockers MUST NOT generate multiple ``[file-lock]`` tables which would be - considered compatible for the same environment + considered compatible for the same environment. - In instances where there would be a conflict but the lock is still desired, - either separate lock files can be written or per-package locking can be used + either separate lock files can be written or per-package locking can be used. ``file-lock.name`` ------------------ - String -- A unique name within the array for the environment this table represents +- A unique name within the array for the environment this table represents. ``[file-lock.marker-values]`` @@ -224,11 +224,11 @@ other is disallowed. - Optional - Table of strings - The keys represent the names of `environment markers`_ and the values are the - values for those markers + values for those markers. - Compatibility is defined by the environment's values matching what is in the - table + table. - Lockers SHOULD sort the keys lexicographically to minimize changes when - updating the file + updating the file. ``file-lock.wheel-tags`` @@ -236,26 +236,26 @@ other is disallowed. - Optional - Array of strings -- An unordered array of `wheel tags`_ which must be supported by the environment +- An unordered array of `wheel tags`_ which must be supported by the environment. - The array MAY not be exhaustive to allow for a smaller array as well as to help prevent multiple ``[[file-lock]]`` tables being compatible with the same environment by having one array being a strict subset of another - ``file-lock.wheel-tags`` entry in the same file - ``[[file-lock]]`` tables + ``file-lock.wheel-tags`` entry in the same file's + ``[[file-lock]]`` tables. - Lockers SHOULD sort the keys lexicographically to minimize changes when - updating the file + updating the file. - Lockers MUST NOT include `compressed tag sets `__ or duplicate tags for consistency across lockers and to simplify checking for - compatibility + compatibility. ``[package-lock]`` ================== - Table -- Mutually exclusive with ``[[file-lock]]`` -- Signifies the use of the package locking approach +- Mutually exclusive with ``[[file-lock]]``. +- Signifies the use of the package locking approach. ``package-lock.requires-python`` @@ -263,39 +263,39 @@ other is disallowed. - String - Holds the `version specifiers`_ for Python version compatibility for the - overall package locking + overall package locking. - Provides at-a-glance information to know if the lock file *may* apply to a version of Python instead of having to scan the entire file to compile the - same information + same information. ``[[package]]`` =============== - Array of tables -- The array contains all data on the locked package versions +- The array contains all data on the locked package versions. - Lockers SHOULD record packages in order by ``package.name`` lexicographically - and ``package.version`` by the sort order for `version specifiers`_ + and ``package.version`` by the sort order for `version specifiers`_. - Lockers SHOULD record keys in the same order as written in this PEP to - minimmize changes when updating + minimmize changes when updating. - Designed so that relevant details as to why a package is included are - in one place to make diff reading easier + in one place to make diff reading easier. ``package.name`` ---------------- - String -- The `normalized name`_ of the package -- Part of what's required to uniquely identify this entry +- The `normalized name`_ of the package. +- Part of what's required to uniquely identify this entry. ``package.version`` ------------------- - String -- The version of the package -- Part of what's required to uniquely identify this entry +- The version of the package. +- Part of what's required to uniquely identify this entry. ``package.multiple-entries`` @@ -305,9 +305,9 @@ other is disallowed. - If package locking via ``[package-lock]``, then the multiple entries for the same package MUST be mutually exclusive via ``package.marker`` (this is not required for per-file locking as the ``package.*.lock`` entries imply mutual - exclusivity) + exclusivity). - Aids in auditing by knowing that there are multiple entries for the same - package that may need to be considered + package that may need to be considered. ``package.description`` @@ -315,9 +315,9 @@ other is disallowed. - Optional - String -- The package's ``Summary`` from its `core metadata`_ +- The package's ``Summary`` from its `core metadata`_. - Useful to help understand why a package was included in the file based on its - purpose + purpose. ``package.simple-repo-package-url`` @@ -326,11 +326,11 @@ other is disallowed. - Optional (although mutually exclusive with ``package.files.simple-repo-package-url``) - String -- Stores the `project detail`_ URL from the `Simple Repository API`_ -- Useful for generating Packaging URLs (aka PURLs) +- Stores the `project detail`_ URL from the `Simple Repository API`_. +- Useful for generating Packaging URLs (aka PURLs). - When possible, lockers SHOULD include this or ``package.files.simple-repo-package-url`` to assist with generating - `software bill of materials`_ (aka SBOMs) + `software bill of materials`_ (aka SBOMs). ``package.marker`` @@ -339,10 +339,10 @@ other is disallowed. - Optional - String - The `environment markers`_ expression which specifies whether this package and - version applies to the environment -- Only applicable via ``[package-lock]`` and the package locking scenario + version applies to the environment. +- Only applicable via ``[package-lock]`` and the package locking scenario. - The lack of this key means this package and version is required to be - installed + installed. ``package.requires-python`` @@ -351,13 +351,13 @@ other is disallowed. - Optional - String - Holds the `version specifiers`_ for Python version compatibility for the - package and version -- Useful for documenting why this package and version was included in the file + package and version. +- Useful for documenting why this package and version was included in the file. - Also helps document why the version restriction in - ``package-lock.requires-python`` was chosen + ``package-lock.requires-python`` was chosen. - It should not provide useful information for installers as it would be captured by ``package-lock.requires-python`` and isn't relevant when - ``[[file-lock]]`` is used + ``[[file-lock]]`` is used. ``package.dependents`` @@ -365,10 +365,10 @@ other is disallowed. - Optional - Array of strings -- A record of the packages that depend on this package and version +- A record of the packages that depend on this package and version. - Useful for analyzing why a package happens to be listed in the file - for auditing purposes -- This does not provide information which influences installers + for auditing purposes. +- This does not provide information which influences installers. ``package.dependencies`` @@ -376,12 +376,12 @@ other is disallowed. - Optional - Array of strings -- A record of the dependencies of the package and version +- A record of the dependencies of the package and version. - Useful in analyzing why a package happens to be listed in the file - for auditing purposes + for auditing purposes. - This does not provide information which influences the installer as ``[[file-lock]]`` specifies the exact files to use and ``[package-lock]`` - applicability is determined by ``package.marker`` + applicability is determined by ``package.marker``. ``package.direct`` @@ -389,7 +389,7 @@ other is disallowed. - Optional - Boolean -- Represents whether the installation is via a `direct URL reference`_ +- Represents whether the installation is via a `direct URL reference`_. ``[[package.files]]`` @@ -397,16 +397,16 @@ other is disallowed. - Must be specified if ``[package.vcs]`` is not - Array of tables -- Tables can be written inline -- Represents the files to potentially install for the package and version +- Tables can be written inline. +- Represents the files to potentially install for the package and version. ``package.files.name`` '''''''''''''''''''''' - String -- The file name -- Necessary for installers to decide what to install when using package locking +- The file name. +- Necessary for installers to decide what to install when using package locking. ``package.files.lock`` @@ -416,9 +416,9 @@ other is disallowed. - Array of strings - An array of ``file-lock.name`` values which signify that the file is to be installed when the corresponding ``[[file-lock]]`` table applies to the - environment + environment. - There MUST only be a single file with any one ``file-lock.name`` entry per - package, regardless of version + package, regardless of version. ``package.files.simple-repo-package-url`` @@ -427,9 +427,9 @@ other is disallowed. - Optional (although mutually exclusive with ``package.simple-repo-package-url``) - String -- The value has the same meaning as ``package.simple-repo-package-url`` +- The value has the same meaning as ``package.simple-repo-package-url``. - This key is available per-file to support :pep:`708` when some files override - what's provided by another `Simple Repository API`_ index + what's provided by another `Simple Repository API`_ index. ``package.files.origin`` @@ -437,9 +437,9 @@ other is disallowed. - Optional - String -- URI where the file was found when the lock file was generated +- URI where the file was found when the lock file was generated. - Useful for documenting where the file came from and potentially where to look - for the file if not already downloaded/available + for the file if not already downloaded/available. ``package.files.hash`` @@ -447,44 +447,44 @@ other is disallowed. - String - The hash value of the file contents using the hash algorithm specified by - ``hash-algorithm`` + ``hash-algorithm``. - Used by installers to verify the file contents match what the locker worked - with + with. ``[package.vcs]`` ----------------- - Must be specified if ``[[package.files]]`` is not (although may be specified - simultaneously with ``[[package.files]]``) + simultaneously with ``[[package.files]]``). - Table representing the version control system containing the package and - version + version. ``package.vcs.type`` '''''''''''''''''''' - String -- The type of version control system used +- The type of version control system used. - The valid values are specified by the `registered VCSs `__ - of the direct URL data structure + of the direct URL data structure. ``package.vcs.origin`` '''''''''''''''''''''' - String -- The URI of where the repository was located when the lock file was generated +- The URI of where the repository was located when the lock file was generated. ``package.vcs.commit`` '''''''''''''''''''''' - String -- The commit ID for the repository which represents the package and version +- The commit ID for the repository which represents the package and version. - The value MUST be immutable for the VCS for security purposes - (e.g. no Git tags) + (e.g. no Git tags). ``package.vcs.lock`` @@ -494,10 +494,10 @@ other is disallowed. - An array of strings - An array of ``file-lock.name`` values which signify that the repository at the specified commit is to be installed when the corresponding ``[[file-lock]]`` - table applies to the environment + table applies to the environment. - A name in the array may only appear if no file listed in ``package.files.lock`` contains the name for the same package, regardless of - version + version. ``package.directory`` @@ -505,28 +505,28 @@ other is disallowed. - Optional and only valid when ``[package-lock]`` is specified - String -- A local directory where a source tree for the package and version exists +- A local directory where a source tree for the package and version exists. - Not valid under ``[[file-lock]]`` as this PEP does not make an attempt to specify a mechanism for verifying file contents have not changed since locking - was performed + was performed. ``[[package.build-requires]]`` ------------------------------ - Optional -- An array of tables whose structure matches that of ``[[package]]`` +- An array of tables whose structure matches that of ``[[package]]``. - Each entry represents a package and version to use when building the - enclosing package and version + enclosing package and version. - Selection of which entries to use for an environment as the same as ``[[package]]`` itself, albeit only applying when installing the build - back-end and its dependencies + back-end and its dependencies. - This helps with reproducibility of the building of a package by recording either what was or would have been used if the locker needed to build the - package + package. - If the installer and user choose to install from source and this array is missing then the installer MAY choose to resolve what to install for building - at install time, otherwise the installer MUST raise an error + at install time, otherwise the installer MUST raise an error. ``[package.tool]`` @@ -534,7 +534,7 @@ other is disallowed. - Optional - Same usage as that of the equivalent table from the - `pyproject.toml specification`_ + `pyproject.toml specification`_. ``[tool]`` @@ -542,7 +542,7 @@ other is disallowed. - Optional - Same usage as that of the equivalent table from the - `pyproject.toml specification`_ + `pyproject.toml specification`_. ------------------------ From 98b4e82ddcf339009ff1da6368fbd346de91e179 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 24 Jul 2024 15:43:11 -0700 Subject: [PATCH 49/55] More details around hashing --- peps/pep-0751.rst | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index 92de581d2da..ada8b73273c 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -23,9 +23,13 @@ dependency resolution at install-time. Motivation ========== -Currently, there is no standard way to specify what top-level dependencies one -would like to see installed into a Python environment and then subsequently -record what was eventually installed in a file (called a *lock file*). +Currently, no standard exists to: + +- Specify what top-level dependencies should be installed into a Python + environment. +- Create an immutable record, such as a lock file, of which dependencies were + installed. + Considering there are at least four well-known solutions to this problem in the community (``pip freeze``, pip-tools_, Poetry_, and PDM_), there seems to be an appetite for lock files in general. @@ -41,7 +45,7 @@ what would be installed into an environment ahead of time. The lack of a standard also has some drawbacks. For instance, any tooling that wants to work with lock files must choose which format to support, potentially -leaving users unsupported (e.g. if Dependabot_ chose not to support PDM, +leaving users unsupported (e.g., if Dependabot_ chose not to support PDM, support by cloud providers who can do dependency installations on your behalf, etc.). @@ -185,8 +189,11 @@ other is disallowed. each file hash value separately. - Allows for updating the entire file to a new hash algorithm without running the risk of accidentally leaving an old hash value in the file. -- Lockers SHOULD specify a hash algorithm that is as least as strong as - `SHA-256 `__. +- The allowed hash algorithms and how to specify them come from + :ref:`packaging:simple-repository-api-json` and the ``hashes`` dictionary of + of the ``files`` dictionary of the Project Details dictionary. +- Failure to validate any hash values for any file that is to be installed MUST + raise an error. ``dependencies`` From e27b73da15bbc15d86de4413e772452b4e5378ff Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 24 Jul 2024 15:44:12 -0700 Subject: [PATCH 50/55] Clarify `package.direct` defaults to `false` There were some questions around whether this being optional meant there was an actual tri-state, but I view "unset" as false-y. --- peps/pep-0751.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index ada8b73273c..7a286a22791 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -189,9 +189,9 @@ other is disallowed. each file hash value separately. - Allows for updating the entire file to a new hash algorithm without running the risk of accidentally leaving an old hash value in the file. -- The allowed hash algorithms and how to specify them come from - :ref:`packaging:simple-repository-api-json` and the ``hashes`` dictionary of - of the ``files`` dictionary of the Project Details dictionary. +- :ref:`packaging:simple-repository-api-json` and the ``hashes`` dictionary of + of the ``files`` dictionary of the Project Details dictionary specifies what + values are valid and guidelines on what hash algorithms to use. - Failure to validate any hash values for any file that is to be installed MUST raise an error. @@ -394,7 +394,7 @@ other is disallowed. ``package.direct`` ------------------ -- Optional +- Optional (defaults to ``false``) - Boolean - Represents whether the installation is via a `direct URL reference`_. From c0d46a1256572c9a2364aac49c2872372e67f1a5 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 24 Jul 2024 15:46:21 -0700 Subject: [PATCH 51/55] Specify ``[[package.files]]` should be sorted by name --- peps/pep-0751.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index 7a286a22791..2f79a79a744 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -414,6 +414,8 @@ other is disallowed. - String - The file name. - Necessary for installers to decide what to install when using package locking. +- Entries in ``[[package.files]]`` SHOULD be lexicographically sorted by this + key to minimze changes in diffs. ``package.files.lock`` From 0c45e8771301f4fe8da90cdeda6c5db9b105fa1d Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 24 Jul 2024 16:00:22 -0700 Subject: [PATCH 52/55] Clarify that `[[package.build-requires]]` is locked --- peps/pep-0751.rst | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index 2f79a79a744..b8b990e4e80 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -216,6 +216,7 @@ other is disallowed. considered compatible for the same environment. - In instances where there would be a conflict but the lock is still desired, either separate lock files can be written or per-package locking can be used. +- Entries in array SHOULD be sorted by ``file-lock.name`` lexicographically. ``file-lock.name`` @@ -406,6 +407,8 @@ other is disallowed. - Array of tables - Tables can be written inline. - Represents the files to potentially install for the package and version. +- Entries in ``[[package.files]]`` SHOULD be lexicographically sorted by + ``package.files.name`` key to minimze changes in diffs. ``package.files.name`` @@ -414,8 +417,6 @@ other is disallowed. - String - The file name. - Necessary for installers to decide what to install when using package locking. -- Entries in ``[[package.files]]`` SHOULD be lexicographically sorted by this - key to minimze changes in diffs. ``package.files.lock`` @@ -527,6 +528,9 @@ other is disallowed. - An array of tables whose structure matches that of ``[[package]]``. - Each entry represents a package and version to use when building the enclosing package and version. +- The array is complete/locked like ``[[package]]`` itself (i.e. installers + follow the same installation procedure for ``[[package.build-requires]]`` as + ``[[package]]``) - Selection of which entries to use for an environment as the same as ``[[package]]`` itself, albeit only applying when installing the build back-end and its dependencies. From ce18c5ce2d8f2d335be24de9403de35b5052b452 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 24 Jul 2024 16:02:19 -0700 Subject: [PATCH 53/55] Clean up a sentence --- peps/pep-0751.rst | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index b8b990e4e80..d009d86f0d2 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -766,10 +766,9 @@ Have the installer do dependency resolution In order to support a format more akin to how Poetry worked when this PEP was drafted, it was suggested that lockers effectively record the packages and their versions which may be necessary to make an install work in any possible -scenario, and then the installer resolves what to install. But since that -complicates auditing a lock file by requiring much more mental effort to know -what packages may be installed in any given scenario. Also, one of the Poetry -developers +scenario, and then the installer resolves what to install. But that complicates +auditing a lock file by requiring much more mental effort to know what packages +may be installed in any given scenario. Also, one of the Poetry developers `suggested `__ that markers as represented in the package locking approach of this PEP may be sufficient to cover the needs of Poetry. Not having the installer do a From f5b574e460a88895ecb562c391755a356a20416b Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 24 Jul 2024 16:03:18 -0700 Subject: [PATCH 54/55] Fix a section title --- peps/pep-0751.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index d009d86f0d2..e0177bbde51 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -881,8 +881,8 @@ recording the creation date of the lock file. But for some same merge conflict reasons as storing the hash of the file contents, this idea was dropped. -Recording the packaged indexes used -=================================== +Recording the package indexes used +================================== Recording what package indexes were used by the locker to decide what to lock for was considered. In the end, though, it was rejected as it was deemed From 17ce7cdb2bfa030e25e4403cbf3e125fb6bd7f54 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 24 Jul 2024 16:05:48 -0700 Subject: [PATCH 55/55] Tweak abstract --- peps/pep-0751.rst | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index e0177bbde51..aebafdece95 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -11,12 +11,11 @@ Replaces: 665 Abstract ======== -This PEP proposes a new file format for specifying what dependencies to -(possibly) install into a Python environment for consistent installation -reproducibility. The format is designed to be human-readable but -machine-generated. Installers consuming the file should be able to -evaluate each package in question in isolation, with no need for -dependency resolution at install-time. +This PEP proposes a new file format for dependency specification +to enable reproducible installation in a Python environment. The format is +designed to be human-readable and machine-generated. Installers consuming the +file should be able to evaluate each package in question in isolation, with no +need for dependency resolution at install-time. ==========