From e146e8435acb0e7b3d081a6b64e49759f4ce5d05 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 20 Jun 2025 10:25:59 -0600 Subject: [PATCH 1/9] Expand interpreters module doc introduction. --- Doc/library/concurrent.interpreters.rst | 138 ++++++++++++++++++++++-- 1 file changed, 128 insertions(+), 10 deletions(-) diff --git a/Doc/library/concurrent.interpreters.rst b/Doc/library/concurrent.interpreters.rst index 8860418e87a585..f772ec84bf3ee3 100644 --- a/Doc/library/concurrent.interpreters.rst +++ b/Doc/library/concurrent.interpreters.rst @@ -13,17 +13,26 @@ -------------- - -Introduction ------------- - The :mod:`!concurrent.interpreters` module constructs higher-level interfaces on top of the lower level :mod:`!_interpreters` module. -.. XXX Add references to the upcoming HOWTO docs in the seealso block. +The module is primarily meant to provide a basic API for managing +interpreters (AKA "subinterpreters") and running things in them. +Running mostly involves switching to an interpreter (in the current +thread) and calling a function in that execution context. + +For concurrency, interpreters themselves (and this module) don't +provide much more than isolation, which on its own isn't useful. +Actual concurrency is available separately through +:mod:`threads ` See `below `_ .. seealso:: + :class:`InterpreterPoolExecutor` + combines threads with interpreters in a familiar interface. + + .. XXX Add references to the upcoming HOWTO docs in the seealso block. + :ref:`isolating-extensions-howto` how to update an extension module to support multiple interpreters @@ -41,18 +50,125 @@ interfaces on top of the lower level :mod:`!_interpreters` module. Key details ----------- -Before we dive into examples, there are a small number of details +Before we dive in further, there are a small number of details to keep in mind about using multiple interpreters: -* isolated, by default +* `isolated `_, by default * no implicit threads * not all PyPI packages support use in multiple interpreters yet .. XXX Are there other relevant details to list? -In the context of multiple interpreters, "isolated" means that -different interpreters do not share any state. In practice, there is some -process-global data they all share, but that is managed by the runtime. + +.. _interpreters-intro: + +Introduction +------------ + +An "interpreter" is effectively the execution context of the Python +runtime. It contains all of the state the runtime needs to execute +a program. This includes things like the import state and builtins. +(Each thread, even if there's only the main thread, has some extra +runtime state, in addition to the current interpreter, related to +the current exception and the bytecode eval loop.) + +The concept and functionality of the interpreter has been a part of +Python since version 2.2, but the feature was only available through +the C-API and not well known, and the `isolation `_ +was relatively incomplete until version 3.12. + +.. _interp-isolation: + +Multiple Interpreters and Isolation +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +A Python implementation may support using multiple interpreters in the +same process. CPython has this support. Each interpreter is +effectively isolated from the others (with a limited number of +carefully managed process-global exceptions to the rule). + +That isolation is primarily useful as a strong separation between +distinct logical components of a program, where you want to have +careful control of how those components interact. + +.. note:: + + Interpreters in the same process can technically never be strictly + isolated from one another since there are few restrictions on memory + access within the same process. The Python runtime makes a best + effort at isolation but extension modules may easily violate that. + Therefore, do not use multiple interpreters in security-senstive + situations, where they shouldn't have access to each other's data. + +Running in an Interpreter +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Running in a different interpreter involves switching to it in the +current thread and then calling some function. The runtime will +execute the function using the current interpreter's state. The +:mod:`!concurrent.interpreters` module provides a basic API for +creating and managing interpreters, as well as the switch-and-call +operation. + +No other threads are automatically started for the operation. +There is `a helper `_ for that though. +There is another dedicated helper for calling the builtin +:func:`exec` in an interpreter. + +When :func:`exec` (or :func:`eval`) are called in an interpreter, +they run using the interpreter's :mod:`!__main__` module as the +"globals" namespace. The same is true for functions that aren't +associated with any module. This is the same as how scripts invoked +from the command-line run in the :mod:`!__main__` module. + + +.. _interp-concurrency: + +Concurrency and Parallelism +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +As noted earlier, interpreters do not provide any concurrency +on their own. They strictly represent the isolated execution +context the runtime will use *in the current thread*. That isolation +makes them similar to processes, but they still enjoy in-process +efficiency, like threads. + +All that said, interpreters do naturally support certain flavors of +concurrency, as a powerful side effect of that isolation. +There's a powerful side effect of that isolation. It enables a +different approach to concurrency than you can take with async or +threads. It's a similar concurrency model to CSP or the actor model, +a model which is relatively easy to reason about. + +You can take advantage of that concurrency model in a single thread, +switching back and forth between interpreters, Stackless-style. +However, this model is more useful when you combine interpreters +with multiple threads. This mostly involves starting a new thread, +where you switch to another interpreter and run what you want there. + +Each actual thread in Python, even if you're only running in the main +thread, has its own *current* execution context. Multiple threads can +use the same interpreter or different ones. + +At a high level, you can think of the combination of threads and +interpreters as threads with opt-in sharing. + +As a significant bonus, interpreters are sufficiently isolated that +they do not share the :term:`GIL`, which means combining threads with +multiple interpreters enables full multi-core parallelism. +(This has been the case since Python 3.12.) + +Communication Between Interpreters +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +In practice, multiple interpreters are useful only if we have a way +to communicate between them. This usually involves some form of +message passing, but can even mean sharing data in some carefully +managed way. + +With this in mind, the :mod:`!concurrent.interpreters` module provides +a :class:`queue.Queue` implementation, available through +:func:`create_queue`. Reference @@ -125,6 +241,8 @@ Interpreter objects Return the result of calling running the given function in the interpreter (in the current thread). + .. _interp-call-in-thread: + .. method:: call_in_thread(callable, /, *args, **kwargs) Run the given function in the interpreter (in a new thread). From c1acf93c363d6497f70ea8703dde2e794d4e9c6c Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 20 Jun 2025 16:36:16 -0600 Subject: [PATCH 2/9] Add docs for Queue and create_queue(). --- Doc/library/concurrent.interpreters.rst | 36 ++++++++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-) diff --git a/Doc/library/concurrent.interpreters.rst b/Doc/library/concurrent.interpreters.rst index f772ec84bf3ee3..cbf5310d644366 100644 --- a/Doc/library/concurrent.interpreters.rst +++ b/Doc/library/concurrent.interpreters.rst @@ -195,6 +195,11 @@ This module defines the following functions: Initialize a new (idle) Python interpreter and return a :class:`Interpreter` object for it. +.. function:: create_queue() + + Initialize a new cross-interpreter queue and return a :class:`Queue` + object for it. + Interpreter objects ^^^^^^^^^^^^^^^^^^^ @@ -277,7 +282,36 @@ Exceptions an object cannot be sent to another interpreter. -.. XXX Add functions for communicating between interpreters. +Communicating Between Interpreters +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. class:: Queue(id) + + A wrapper around a low-level, cross-interpreter queue, which + implements the :class:`queue.Queue` interface. The underlying queue + can only be created through :func:`create_queue`. + + Cross-interpreter queues support all the same objects as + :meth:`Interpreter.prepare_main`. + + .. attribute:: id + + (read-only) + + The queue's ID. + + +.. exception:: QueueEmptyError + + This is exception, a subclass of :exc:`queue.Empty`, is raised from + :meth:`!Queue.get` and :meth:`!Queue.get_nowait` when the queue + is empty. + +.. exception:: QueueFullError + + This is exception, a subclass of :exc:`queue.Full`, is raised from + :meth:`!Queue.put` and :meth:`!Queue.put_nowait` when the queue + is full. Basic usage From d41d8c7f6dbf83ccf286e9c604860e7621387860 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 24 Jun 2025 13:12:43 -0600 Subject: [PATCH 3/9] Drop references to object sharing. --- Doc/library/concurrent.interpreters.rst | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/Doc/library/concurrent.interpreters.rst b/Doc/library/concurrent.interpreters.rst index cbf5310d644366..e563cda5788b59 100644 --- a/Doc/library/concurrent.interpreters.rst +++ b/Doc/library/concurrent.interpreters.rst @@ -234,8 +234,7 @@ Interpreter objects .. method:: prepare_main(ns=None, **kwargs) - Bind "shareable" objects in the interpreter's - :mod:`!__main__` module. + Bind objects in the interpreter's :mod:`!__main__` module. .. method:: exec(code, /, dedent=True) @@ -291,9 +290,6 @@ Communicating Between Interpreters implements the :class:`queue.Queue` interface. The underlying queue can only be created through :func:`create_queue`. - Cross-interpreter queues support all the same objects as - :meth:`Interpreter.prepare_main`. - .. attribute:: id (read-only) @@ -345,6 +341,3 @@ Creating an interpreter and running code in it:: t = interp.call_in_thread(run) t.join() - - -.. XXX Explain about object "sharing". From 9c6defc136965d47c11dc292ea15d21f9f14821e Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 24 Jun 2025 13:49:40 -0600 Subject: [PATCH 4/9] Add info about object sharing. --- Doc/library/concurrent.interpreters.rst | 36 +++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/Doc/library/concurrent.interpreters.rst b/Doc/library/concurrent.interpreters.rst index e563cda5788b59..d8617c304b58dd 100644 --- a/Doc/library/concurrent.interpreters.rst +++ b/Doc/library/concurrent.interpreters.rst @@ -170,6 +170,36 @@ With this in mind, the :mod:`!concurrent.interpreters` module provides a :class:`queue.Queue` implementation, available through :func:`create_queue`. +.. _interp-object-sharing: + +"Sharing" Objects +^^^^^^^^^^^^^^^^^ + +Any data actually shared between interpreters loses the thread-safety +provided by the :term:`GIL`. There are various options for dealing with +this in extension modules. However, from Python code the lack of +thread-safety means objects can't actually be shared, with a few +exceptions. Instead, a copy must be created, which means mutable +objects won't stay in sync. + +By default, most objects are copied with :mod:`pickle` when they are +passed to another interpreter. Nearly all of the immutable builtin +objects are either directly shared or copied efficiently. For example: + +* :const:`None` +* :class:`bool` (:const:`True` and :const:`False`) +* :class:`bytes` +* :class:`str` +* :class:`int` +* :class:`float` +* :class:`tuple` (of similarly supported objects) + +There is a small number of Python types that actually share mutable +data between interpreters: + +* :class:`memoryview` +* :class:`Queue` + Reference --------- @@ -236,6 +266,9 @@ Interpreter objects Bind objects in the interpreter's :mod:`!__main__` module. + Some objects are actually shared and some are copied efficiently, + but most are copied via :mod:`pickle`. See :ref:`interp-object-sharing`. + .. method:: exec(code, /, dedent=True) Run the given source code in the interpreter (in the current thread). @@ -290,6 +323,9 @@ Communicating Between Interpreters implements the :class:`queue.Queue` interface. The underlying queue can only be created through :func:`create_queue`. + Some objects are actually shared and some are copied efficiently, + but most are copied via :mod:`pickle`. See :ref:`interp-object-sharing`. + .. attribute:: id (read-only) From 8f5e90fba608e33569f8ef5a30ede626ac970ded Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 24 Jun 2025 13:58:22 -0600 Subject: [PATCH 5/9] Elaborate about the main interpreter. --- Doc/library/concurrent.interpreters.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Doc/library/concurrent.interpreters.rst b/Doc/library/concurrent.interpreters.rst index d8617c304b58dd..84c9b6d02ebb1f 100644 --- a/Doc/library/concurrent.interpreters.rst +++ b/Doc/library/concurrent.interpreters.rst @@ -219,6 +219,8 @@ This module defines the following functions: .. function:: get_main() Return an :class:`Interpreter` object for the main interpreter. + This is the interpreter the runtime created to run the :term:`REPL` + or the script given at the command-line. It is usually the only one. .. function:: create() From d96ed19a76f80a34d22891357bc35d8f943e2335 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 24 Jun 2025 14:01:57 -0600 Subject: [PATCH 6/9] Clarify about the ID. --- Doc/library/concurrent.interpreters.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/library/concurrent.interpreters.rst b/Doc/library/concurrent.interpreters.rst index 84c9b6d02ebb1f..fd61647f4622aa 100644 --- a/Doc/library/concurrent.interpreters.rst +++ b/Doc/library/concurrent.interpreters.rst @@ -247,7 +247,7 @@ Interpreter objects (read-only) - The interpreter's ID. + The underlying interpreter's ID. .. attribute:: whence From fe0ce841ac29d2e2cb9b6ef0b96d0cf726625746 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 24 Jun 2025 14:05:38 -0600 Subject: [PATCH 7/9] Add an extra example. --- Doc/library/concurrent.interpreters.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Doc/library/concurrent.interpreters.rst b/Doc/library/concurrent.interpreters.rst index fd61647f4622aa..1848a87bf578e5 100644 --- a/Doc/library/concurrent.interpreters.rst +++ b/Doc/library/concurrent.interpreters.rst @@ -370,6 +370,12 @@ Creating an interpreter and running code in it:: print('spam!') """)) + def run(arg): + return arg + + res = interp.call(run, 'spam!') + print(res) + def run(): print('spam!') From 543363bb79c31fda0f0d2184a54abd951b88fb42 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 24 Jun 2025 14:43:25 -0600 Subject: [PATCH 8/9] Fix a ref. --- Doc/library/concurrent.interpreters.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/library/concurrent.interpreters.rst b/Doc/library/concurrent.interpreters.rst index 1848a87bf578e5..709685349fa989 100644 --- a/Doc/library/concurrent.interpreters.rst +++ b/Doc/library/concurrent.interpreters.rst @@ -28,7 +28,7 @@ Actual concurrency is available separately through .. seealso:: - :class:`InterpreterPoolExecutor` + :class:`~concurrent.futures.InterpreterPoolExecutor` combines threads with interpreters in a familiar interface. .. XXX Add references to the upcoming HOWTO docs in the seealso block. From 7cbd93372aebc145da01b477b2100e5567651414 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Thu, 26 Jun 2025 15:10:13 -0600 Subject: [PATCH 9/9] Fix typos. Co-authored-by: neonene <53406459+neonene@users.noreply.github.com> --- Doc/library/concurrent.interpreters.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Doc/library/concurrent.interpreters.rst b/Doc/library/concurrent.interpreters.rst index 709685349fa989..524d505bcf144f 100644 --- a/Doc/library/concurrent.interpreters.rst +++ b/Doc/library/concurrent.interpreters.rst @@ -72,7 +72,7 @@ a program. This includes things like the import state and builtins. runtime state, in addition to the current interpreter, related to the current exception and the bytecode eval loop.) -The concept and functionality of the interpreter has been a part of +The concept and functionality of the interpreter have been a part of Python since version 2.2, but the feature was only available through the C-API and not well known, and the `isolation `_ was relatively incomplete until version 3.12. @@ -97,7 +97,7 @@ careful control of how those components interact. isolated from one another since there are few restrictions on memory access within the same process. The Python runtime makes a best effort at isolation but extension modules may easily violate that. - Therefore, do not use multiple interpreters in security-senstive + Therefore, do not use multiple interpreters in security-sensitive situations, where they shouldn't have access to each other's data. Running in an Interpreter @@ -337,13 +337,13 @@ Communicating Between Interpreters .. exception:: QueueEmptyError - This is exception, a subclass of :exc:`queue.Empty`, is raised from + This exception, a subclass of :exc:`queue.Empty`, is raised from :meth:`!Queue.get` and :meth:`!Queue.get_nowait` when the queue is empty. .. exception:: QueueFullError - This is exception, a subclass of :exc:`queue.Full`, is raised from + This exception, a subclass of :exc:`queue.Full`, is raised from :meth:`!Queue.put` and :meth:`!Queue.put_nowait` when the queue is full.