[24.1] self signed certificates are no longer loaded

[rasmus91]

Description

I am using a gitlab python repo, that proxies pypi.

After upgrading to pip 24.1 the certificate I have installed in /usr/local/share/ca-certificates is no longer loaded it seems, to validate that source.

After going back to pip 24.0 It works just fine once again.

Expected behavior

I would expect no problem installing packages through a private repo for which I have appropriate certificates installed systemwide.

pip version

24.1

Python version

3.10

OS

Ubuntu 22.04

How to Reproduce

1. Install certificate in /usr/local/share/ca-certificates
2. run sudo update-ca-certificates
3. setup a source repo to your private repository
4. install a package from it

Output

(this is output from ansible, but its still pip)

Looking in indexes: https://gitlab%2Bdeploy-token-3:****@gitlab.ourinstance.com/api/v4/groups/25/-/packages/pypi/simple
Obtaining file:///var/local/sbat-man-server-dev
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
:stderr:   error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [9 lines of output]
      Looking in indexes: https://gitlab%2Bdeploy-token-3:****@gitlab.ourinstance.com/api/v4/groups/25/-/packages/pypi/simple
      WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)'))': /api/v4/groups/25/-/packages/pypi/simple/poetry-core/
      WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)'))': /api/v4/groups/25/-/packages/pypi/simple/poetry-core/
      WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)'))': /api/v4/groups/25/-/packages/pypi/simple/poetry-core/
      WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)'))': /api/v4/groups/25/-/packages/pypi/simple/poetry-core/
      WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)'))': /api/v4/groups/25/-/packages/pypi/simple/poetry-core/
      Could not fetch URL https://gitlab%2Bdeploy-token-3:****@gitlab.ourinstance.com/api/v4/groups/25/-/packages/pypi/simple/poetry-core/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='gitlab.ourinstance.com', port=443): Max retries exceeded with url: /api/v4/groups/25/-/packages/pypi/simple/poetry-core/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)'))) - skipping
      ERROR: Could not find a version that satisfies the requirement poetry-core (from versions: none)
      ERROR: No matching distribution found for poetry-core
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[sweetlilmre]

I can confirm similar behaviour on windows.
For 24.1 pip fails with certificate errors for a self-signed cert and only works by pointing REQUESTS_CA_BUNDLE to a bundle containing the self-signed CA.
24.0 requires --use-feature truststore but works as expected. I tested all non-beta versions from 23 onwards.

[ichard26]

Hey all, apologies for the disruption. Pip by default does not use the system certificate store. It is an optional feature enabled by the integration of truststore which did get bumped in 24.1. cc @sethmlarson

[sethmlarson]

@rasmus91 @sweetlilmre Just to confirm, you're running into this issue without using the optional "Truststore" feature or while you're using it (ie --with-feature=truststore)? Have to clarify because it's not clear from your posted reproduction steps.

[rasmus91]

Not using that option, no. fre. 21. jun. 2024 16.06 skrev Seth Michael Larson ***@***.***

…

: @sweetlilmre <https://github.com/sweetlilmre> Just to confirm, you're running into this issue *without* using the optional "Truststore" feature or while you're using it (ie --with-feature=truststore)? Have to clarify because it's not clear from your posted reproduction steps. — Reply to this email directly, view it on GitHub <#12779 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARQ7C5RHVTYOACO42J24XDZIQXNXAVCNFSM6AAAAABJVQC7RSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBSHAZDGNRWHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[sweetlilmre]

@sethmlarson, while using it:

• versions 23.0 to 24.0 using --with-feature=truststore work (these are the versions I tested).
• version 24.1 using --with-feature=truststore does not work and I need to have REQUESTS_CA_BUNDLE defined and pointing to a bundle containing the required CA.

Sorry for the ambiguity.
How it ever worked for @rasmus91 without the feature enabled surprises me.

[rasmus91]

Yes, it did until 24.1. i dont understand why it wouldn't when the CA cert is installed system wide?

[sethmlarson]

So I'm actually suspecting that a defect in Requests is the source of the issues here, pip uses Requests 2.32.0 but there have been 3 bugfix releases since that release to Requests with some relating to certificates.

[notatallshaw]

I've always wanted to know how pip's vendoring process worked, so I've created two branches to test if requests is the issue:

Branch with latest version of requests: pip install git+https://github.com/notatallshaw/pip@requests-2.32.3
Branch with version of requests pip 24.0 had: pip install git+https://github.com/notatallshaw/pip@requests-2.31.0

Can someone please try their workflow that works on 24.0 but breaks on 24.1 on both of these branches and report the results for both please (may need to use --no-cache-dir and --force to make sure there's a network connetion). It would also be apprecative if you could give exact steps of your workflow.

[ichard26]

Looking at the requests changelog, I strongly suspect that psf/requests#6655 is implicated. psf/requests#6716 which landed in requests v2.32.3 restores the ability to specify custom SSLContexts in sub-classes of HTTPAdapter which pip does have.

pip/src/pip/_internal/network/session.py

Line 290 in 2753c77

class HTTPAdapter(_SSLContextAdapterMixin, _BaseHTTPAdapter):

Please try @notatallshaw's branches to help us determine whether requests is the root problem and upgrading requests fixes the bug. Thanks!

[sweetlilmre]

Let me preface this with: I am no python / pip expert, in fact I probably don't know what I'm doing.
That said I ran the following tests:

1. Create 2 directories, 2.32.3 and 2.31.0
2. Create a venv in both
3. Install the appropriate branch of pip in each
4. Test

Results:

2.32.3:

pip --version
pip 24.0 from C:\source\piptest\2.32.3\.venv\Lib\site-packages\pip (python 3.12)

Set REQUESTS_CA_BUNDLE environment variable
Install:
python.exe -m pip install git+https://github.com/notatallshaw/pip@requests-2.32.3

pip --version
pip 24.2.dev0 from C:\source\piptest\2.32.3\.venv\Lib\site-packages\pip (python 3.12)

Clear REQUESTS_CA_BUNDLE environment variable

Check:
- pip install pip-system-certs FAILS (SSL cert errors)
- pip --use-feature truststore install pip-system-certs SUCCEEDS

2.31.0

pip --version
pip 24.0 from C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip (python 3.12)

Set REQUESTS_CA_BUNDLE environment variable
Install:
python.exe -m pip install git+https://github.com/notatallshaw/pip@requests-2.31.0

pip --version
pip 24.2.dev0 from C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip (python 3.12)

Clear REQUESTS_CA_BUNDLE environment variable

Check:
pip install pip-system-certs FAILS (traceback)

  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\source\piptest\2.31.0\.venv\Scripts\pip.exe\__main__.py", line 7, in <module>
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_internal\cli\main.py", line 78, in main
    command = create_command(cmd_name, isolated=("--isolated" in cmd_args))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_internal\commands\__init__.py", line 114, in create_command
    module = importlib.import_module(module_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python312\Lib\importlib\__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_internal\commands\install.py", line 15, in <module>
    from pip._internal.cli.req_command import (
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_internal\cli\req_command.py", line 18, in <module>
    from pip._internal.index.collector import LinkCollector
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_internal\index\collector.py", line 31, in <module>
    from pip._vendor import requests
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_vendor\requests\__init__.py", line 45, in <module>
    from .exceptions import RequestsDependencyWarning
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_vendor\requests\exceptions.py", line 9, in <module>
    from .compat import JSONDecodeError as CompatJSONDecodeError
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_vendor\requests\compat.py", line 10, in <module>
    import chardet
ModuleNotFoundError: No module named 'chardet'