Importing ctypes results in extra open libffi file descriptor since 3.14

[DaanDeMeyer]

Bug report

Bug description:

On python 3.13 (Fedora), importing ctypes results in the following output from lsfd:

python3 893377 daandemeyer    mem r-x---  REG   0:37     0  304862113 /usr/lib64/libffi.so.8.1.4
python3 893377 daandemeyer    mem r-----  REG   0:37     0  304862113 /usr/lib64/libffi.so.8.1.4
python3 893377 daandemeyer    mem r-----  REG   0:37     0  304862113 /usr/lib64/libffi.so.8.1.4
python3 893377 daandemeyer    mem rw----  REG   0:37     0  304862113 /usr/lib64/libffi.so.8.1.4

On python 3.14 (Fedora), importing ctypes results in the following output from lsfd:

python3 892407 daandemeyer      5 r-----  REG  0:238  1342       7553 /usr/lib64/libffi.so.8.2.0

This ended up breaking https://github.com/systemd/mkosi because it relied on the fact that no new regular file descriptors were opened by importing ctypes (on python 3.14 the file descriptor shows up in /proc/self/fd whereas on python 3.13 the file descriptors are memory maps which don't show up in /proc/self/fd). I searched through the commit list and couldn't find the commit that introduced this change. It also didn't seem to be documented in the release notes. Can anyone point me to the commit introducing this change and the reasoning behind it?

CPython versions tested on:

3.14

Operating systems tested on:

Linux

[behrmann]

This can be reproduced outside of Fedora as well with uv run -p python3.14 python and then running lsfd --pid on the interpreter process before and after importing ctypes.

With python 3.12 and 3.13 the diff of the lsfd output before and after importing ctypes is empty, with 3.14 the diff is

--- py314_witout	2025-06-24 18:12:47.689673532 +0200
+++ /dev/fd/63	2025-06-24 18:12:59.767811969 +0200
@@ -50,6 +50,7 @@
 python3.14 30153 user    mem r-----  REG btrfs:37     0   22340483 /usr/lib/libpthread.so.0
 python3.14 30153 user    mem r-----  REG btrfs:37     0   22340483 /usr/lib/libpthread.so.0
 python3.14 30153 user    mem rw----  REG btrfs:37     0   22340483 /usr/lib/libpthread.so.0
+python3.14 30153 user    mem r-x---  REG btrfs:58     0   29265231 /home/user/.local/share/uv/python/cpython-3.14.0b2-linux-x86_64-gnu/bin/python3.14
 python3.14 30153 user    shm r-----  REG btrfs:37     0   22341176 /usr/lib/gconv/gconv-modules.cache
 python3.14 30153 user    mem r-----  REG btrfs:37     0   22340447 /usr/lib/ld-linux-x86-64.so.2
 python3.14 30153 user    mem r-x---  REG btrfs:37     0   22340447 /usr/lib/ld-linux-x86-64.so.2
@@ -61,3 +62,4 @@
 python3.14 30153 user      2 rw----  CHR    pts:1    38          4 /dev/pts/1
 python3.14 30153 user      3 rw----  CHR    pts:1    38          4 /dev/pts/1
 python3.14 30153 user      4 rw----  CHR    pts:1    38          4 /dev/pts/1
+python3.14 30153 user      5 r-----  REG btrfs:58   755   29265231 /home/user/.local/share/uv/python/cpython-3.14.0b2-linux-x86_64-gnu/bin/python3.14

Looking at python-build-standalone they do seem to use the same libffi version except for musl, which could rule out a change in libffi.

[vstinner]

This new behavior comes from the commit 04d4aac of the issue gh-128485:

commit 04d4aacaacbe26146e9b5ff6b0043f86f3c543d4
Author: Kumar Aditya <kumaraditya@python.org>
Date:   Mon Mar 24 18:14:25 2025 +0530

    gh-128485: ensure that dlmalloc initializes itself at import time in ctypes  (#131633)

cc @kumaraditya303

[vstinner]

This ended up breaking https://github.com/systemd/mkosi because it relied on the fact that no new regular file descriptors were opened by importing ctypes (on python 3.14 the file descriptor shows up in /proc/self/fd whereas on python 3.13 the file descriptors are memory maps which don't show up in /proc/self/fd)

Can't you update mkosi for this ctypes change?

[DaanDeMeyer]

This ended up breaking https://github.com/systemd/mkosi because it relied on the fact that no new regular file descriptors were opened by importing ctypes (on python 3.14 the file descriptor shows up in /proc/self/fd whereas on python 3.13 the file descriptors are memory maps which don't show up in /proc/self/fd)

Can't you update mkosi for this ctypes change?

I can but there's nothing preventing cpython from changing this again in the future then. I'm also still confused why there needs to be a regular file descriptor open to libffi, doesn't it just need to be mmapped?

[DaanDeMeyer]

To add some more details, the offending function in libffi which opens the file descriptor without closing it is ffi_tramp_get_libffi(). Since we don't make use of closures in mkosi, it should be OK for us to close the file descriptor behind libffi's back.

[vstinner]

Example to reproduce the issue:

$ ./python
>>> import os
>>> os.system(f"ls -l /proc/{os.getpid()}/fd/")
total 0
lrwx------ 1 vstinner vstinner 64 25 juin  11:43 0 -> /dev/pts/0
lrwx------ 1 vstinner vstinner 64 25 juin  11:43 1 -> /dev/pts/0
lrwx------ 1 vstinner vstinner 64 25 juin  11:43 2 -> /dev/pts/0
lrwx------ 1 vstinner vstinner 64 25 juin  11:43 3 -> /dev/pts/0
lrwx------ 1 vstinner vstinner 64 25 juin  11:43 4 -> /dev/pts/0
0

# Importing _ctypes creates a new file descriptor (fd 5)
>>> import _ctypes
>>> os.system(f"ls -l /proc/{os.getpid()}/fd/")
total 0
lrwx------ 1 vstinner vstinner 64 25 juin  11:43 0 -> /dev/pts/0
lrwx------ 1 vstinner vstinner 64 25 juin  11:43 1 -> /dev/pts/0
lrwx------ 1 vstinner vstinner 64 25 juin  11:43 2 -> /dev/pts/0
lrwx------ 1 vstinner vstinner 64 25 juin  11:43 3 -> /dev/pts/0
lrwx------ 1 vstinner vstinner 64 25 juin  11:43 4 -> /dev/pts/0
lr-x------ 1 vstinner vstinner 64 25 juin  11:43 5 -> /usr/lib64/libffi.so.8.1.4
0

[DaanDeMeyer]

To add some more details, the offending function in libffi which opens the file descriptor without closing it is ffi_tramp_get_libffi(). Since we don't make use of closures in mkosi, it should be OK for us to close the file descriptor behind libffi's back.

Actually given @behrmann's example above, doing this is very hard, since it's not trivial to detect which file descriptor we need to close.

[kumaraditya303]

Example to reproduce the issue:

I am not able to reproduce this on main:

❯ ./python
Python 3.15.0a0 (heads/main:15c6d63fe6f, Jun 25 2025, 15:33:31) [Clang 18.1.3 (1ubuntu1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.system(f"ls -l /proc/{os.getpid()}/fd/")
total 0
lrwx------ 1 realkumaraditya realkumaraditya 64 Jun 25 16:05 0 -> /dev/pts/0
lrwx------ 1 realkumaraditya realkumaraditya 64 Jun 25 16:05 1 -> /dev/pts/0
lrwx------ 1 realkumaraditya realkumaraditya 64 Jun 25 16:05 2 -> /dev/pts/0
lrwx------ 1 realkumaraditya realkumaraditya 64 Jun 25 16:05 3 -> /dev/pts/0
lrwx------ 1 realkumaraditya realkumaraditya 64 Jun 25 16:05 4 -> /dev/pts/0
0
>>> import _ctypes
>>> os.system(f"ls -l /proc/{os.getpid()}/fd/")
total 0
lrwx------ 1 realkumaraditya realkumaraditya 64 Jun 25 16:05 0 -> /dev/pts/0
lrwx------ 1 realkumaraditya realkumaraditya 64 Jun 25 16:05 1 -> /dev/pts/0
lrwx------ 1 realkumaraditya realkumaraditya 64 Jun 25 16:05 2 -> /dev/pts/0
lrwx------ 1 realkumaraditya realkumaraditya 64 Jun 25 16:05 3 -> /dev/pts/0
lrwx------ 1 realkumaraditya realkumaraditya 64 Jun 25 16:05 4 -> /dev/pts/0
0
>>>

[vstinner]

I am not able to reproduce this on main

Oh. It may depend on the libffi version. I'm using libffi-3.4.6-5.fc42.x86_64 on Fedora 42.

[kumaraditya303]

Oh. It may depend on the libffi version. I'm using libffi-3.4.6-5.fc42.x86_64 on Fedora 42.

Okay, I think the best fix is just ignore any file descriptors created by ctypes as the behavior is caused by the underlying libffi library not by the _ctypes extension.

[DaanDeMeyer]

Oh. It may depend on the libffi version. I'm using libffi-3.4.6-5.fc42.x86_64 on Fedora 42.

Okay, I think the best fix is just ignore any file descriptors created by ctypes as the behavior is caused by the underlying libffi library not by the _ctypes extension.

We cannot ignore any file descriptors because systemd's file descriptor packing protocol depends on being able to move file descriptors to 3, 4, 5, 6, ... . When importing ctypes, libffi's file descriptors squats on file descriptor number 3 so we have to close it to be able to implement the file descriptor packing protocol properly. We're slightly lucky for now that this doesn't break any functionality of ctypes but there is no guarantee that it will stay that way. I would prefer an officially supported way to prevent libffi's file descriptor from being opened like it was before python 3.14.