os.execvp on windows doesn't wait for the executed program to finish and doesn't report it's exit code

[snoopyjc]

Bug report

os.execvp on windows doesn't wait for the executed program to finish and doesn't report it's exit code. Code:

import os
import sys

if len(sys.argv) > 1 and sys.argv[1] == "my_argument":
    sys.stderr.write("my_argument detected. Exiting with code 1.\n")
    sys.exit(1)
else:
    sys.stderr.write("no argument detected. Calling os.execvp\n")
    os.execvp(sys.executable, [sys.executable] + sys.argv + ["my_argument"])

bat file:

@echo off
python exec.py
if %errorlevel% NEQ 0 (
    echo The script gave exit code %errorlevel%
) else (
    echo Error: The script exited with exit code 0
)

Output from command prompt:

C:\pythonizer\play>exec.bat
no argument detected. Calling os.execvp
Error: The script exited with exit code 0

C:\pythonizer\play>my_argument detected. Exiting with code 1.

Note the command prompt prompted me for a new command before the exec'd process wrote out it's message.

Your environment

• CPython versions tested on: 3.10.6
• Operating system and architecture: Windows 10

[eryksun]

The POSIX exec*() functions cannot be implemented on Windows since the OS doesn't support replacing the process image. At most, the exec*() functions can be crudely emulated. The C runtime library on Windows does the minimum that remains generally safe. It spawns the new process and exits. Python, for its part, simply calls C _wexecv() and _wexecve().

If we were to implement our own versions of execv() and execve() that spawn a child process and wait for it to exit, we'd have to take some defensive steps. We would have to either forcefully terminate or suspend all other threads in the current process. We'd have to walk a process snapshot to close unnecessary file handles in order to avoid problems with closing pipes or problems with the share mode of files (this includes having to change the working directory to the user's temp directory). That's not even considering the problems with other inherited kernel objects (e.g. a mutex). Also, the spawned process should be added to a kill-on-close, silent-breakaway job object.

This would still be very far from truly emulating an exec*() that replaces the image of the current process. We can't support any scenario with tight coupling to the parent process or some other ancestor process. For example, an ancestor of the current process might expect to be able to use a handle for the current process, or its PID, to get or set various information or otherwise modify the current process. This can include any of the following functions, depending on the context:

Base API

• QueryFullProcessImageNameW
• GetProcessAffinityMask / SetProcessAffinityMask
• GetProcessIoCounters
• GetApplicationRestartSettings
• DebugBreakProcess

Handle API

• DuplicateHandle

Process and Thread API

• OpenProcess
• OpenProcessToken
• CreateRemoteThread / CreateRemoteThreadEx
• QueueUserAPC / QueueUserAPC2
• GetExitCodeProcess
• GetProcessId
• GetProcessInformation
• GetProcessHandleCount
• GetProcessTimes
• GetPriorityClass / SetPriorityClass
• GetProcessPriorityBoost / SetProcessPriorityBoost
• GetProcessDefaultCpuSets / SetProcessDefaultCpuSets
• GetProcessDefaultCpuSetMasks / SetProcessDefaultCpuSetMasks
• QueryProcessAffinityUpdateMode / SetProcessAffinityUpdateMode
• GetProcessMitigationPolicy
• GetProcessVersion

Memory API

• VirtualAllocEx / VirtualAlloc2 / VirtualFreeEx
• VirtualProtectEx
• VirtualQueryEx
• MapViewOfFile2 / MapViewOfFile3 / UnmapViewOfFile2
• PrefetchVirtualMemory
• QueryVirtualMemoryInformation
• ReadProcessMemory / WriteProcessMemory
• GetProcessWorkingSetSize / GetProcessWorkingSetSizeEx
• SetProcessWorkingSetSize / SetProcessWorkingSetSizeEx
• AllocateUserPhysicalPages / FreeUserPhysicalPages

Process Snapshotting API

• PssCaptureSnapshot / PssFreeSnapshot
• PssDuplicateSnapshot
• PssQuerySnapshot
• PssWalkSnapshot

Process Status API

• GetProcessImageFileNameW
• GetModuleBaseNameW
• GetModuleFileNameExW
• GetModuleInformation
• GetMappedFileNameW
• EnumProcessModules / EnumProcessModulesEx
• EmptyWorkingSet
• GetProcessMemoryInfo
• GetWsChanges / GetWsChangesEx / InitializeProcessForWsWatch
• QueryWorkingSet / QueryWorkingSetEx

Tool Help API

• CreateToolhelp32Snapshot
• Heap32First
• Heap32ListFirst / Heap32ListNext
• Module32FirstW / Module32NextW
• Process32FirstW / Process32NextW
• Thread32First / Thread32Next

[snoopyjc]

Thank you for the detailed analysis. I looked into how perl handles exec on windows, and it calls _execvp - can Python just do the same thing as perl? https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/execvp-wexecvp?view=vs-2019

[eryksun]

All of the C runtime's _[w]exec*() functions are implemented to spawn the executable and then exit the current process. They do not wait on the spawned process.

Python uses _wexecv() to implement os.execv(), and it uses _wexecve() to implement os.execve(). It defines the other os.exec*() functions in terms of those two -- including os.execvp(), os.execvpe(), os.execl(), os.execle(), os.execlp(), and os.execlpe().

[venthur]

Would it make sense to put a big warning in the corresponding docs?

Currently it says:

These functions all execute a new program, replacing the current process; they do not return. On Unix, the new executable is loaded into the current process, and will have the same process id as the caller. Errors will be reported as OSError exceptions.

[...]

Availability: Unix, Windows, not Emscripten, not WASI.

Maybe one could clarify that it does not work as described on Windows (and point to Popen as the next best alternative)

[jiasli]

It seems this issue has been reported before in #63323.