_Py_wfopen no longer exported

[PlanetCNC]

Bug report

Bug description:

_Py_wfopen in no longer exported since 3.13.
I'm using embed version and I can not use fopen or _wfopen.
Please reconsider decision to remove _Py_wfopen since it is only way to open file when used in embed mode and fopen/_wfopen is not available. Without it PyRun_FileExFlags is useless to me and my application can no longer call external scripts.

CPython versions tested on:

3.13

Operating systems tested on:

Windows

Linked PRs

• gh-127350: Add Py_fopen() function #127821
• gh-127350: Add more tests for Py_fopen() #128587

[skirpichev]

Hidden by #107213

CC @vstinner

[vstinner]

Hi,

I'm using embed version and I can not use fopen or _wfopen.

Why can't you use fopen() or _wfopen()?

[PlanetCNC]

In my case PyRun_FileExFlags crashes if I use FILE* from fopen() or _wfopen().

I suspect this is because different toolset versions are used to compile python313.dll and our software. That is why we don't mix them.

With _Py_wfopen everything works flawlessly.

We integrated Python into our CNC machine control software and it was working great for lots of users until now. Here is an example which is no longer possible to run:
https://www.youtube.com/watch?v=MruGmi1JoCc

Here is our module documentation:
https://cnc.zone/sdk/python/python

[vstinner]

In my case PyRun_FileExFlags crashes if I use FILE* from fopen() or _wfopen(). I suspect this is because different toolset versions are used to compile python313.dll and our software. That is why we don't mix them.

The FILE* is not NULL? That's strange.

You may give a try to _Py_fopen_obj() function which is the last "open" function which returns a FILE* in the Python C API. It's private, you're not supposed to use it.

[PlanetCNC]

No, it is not NULL.

I tried _Py_fopen_obj() but I was unable to make it work. It is "private" anyway so I expect to be removed in next version.
How about making new, nonprivate function with same implementation as _Py_wfopen ?

[PlanetCNC]

I managed to make _Py_fopen_obj() work. Please do not remove my last option to open file.

[ZeroIntensity]

In my case PyRun_FileExFlags crashes if I use FILE* from fopen() or _wfopen().

That sounds like a bug, could you file a new issue with a reproducer?

But overall, I think we just need a public API if this is the only way to do something. You shouldn't ever rely on anything prefixed with _Py as a user.

[vstinner]

Yeah, you should be able to call fopen() and pass the result to a Python API which expect a FILE* file.

[PlanetCNC]

This is not correct. FILE* should not be passed across a DLL boundary which is case when using embed Python.
Here is Microsoft article explaining this in detail.
https://learn.microsoft.com/en-us/cpp/c-runtime-library/potential-errors-passing-crt-objects-across-dll-boundaries?view=msvc-170

[asvetlov]

I read the article as:

1. 10 years ago you should be careful and use the same Visual Studio version and the same multithreaded (/MT vs /MD) mode as it was used for compiling Python itself. It was always a good advice. Now CRT is cross-compatible which helps people a lot; good to know.
2. Release the memory by the same dynamic library that was used for allocation. For example, call fclose() in your dll for files opened with fopen(). Also it sounds reasonable.

[ZeroIntensity]

As far as I can tell, _Py_wfopen is just a wrapper over fopen and _wfopen anyway. It shouldn't matter where the FILE * comes from.

[PlanetCNC]

As far as I can tell, _Py_wfopen is just a wrapper over fopen and _wfopen anyway. It shouldn't matter where the FILE * comes from.

This is not true. Microsoft clearly states that passing passing CRT objects across DLL boundaries causes potential errors.

I spend a lot of time trying to "fix" this issue and is simply not possible. And I'm not a novice - I have more than 30 years experience in programming very complex stuff that runs on all popular operating systems. It is not that I don't know how to correctly use 'fopen' and '_wfopen'.

If there is such a problem keeping '_Py_wfopen' then functions that accept FILE* should also expect filename as PyObject and then internally call _Py_fopen_obj(). This change should be fairly easy to make.

[ZeroIntensity]

I'm not doubting your expertise, but note that the PyRun APIs have existed for years; if they weren't usable without the private API on Windows, I would hope that we would have noticed by now. Could this be a matter of some incorrect linker flags?

cc @zooba