diff --git a/Doc/howto/concurrency.rst b/Doc/howto/concurrency.rst new file mode 100644 index 00000000000000..d7464cdc5c18ff --- /dev/null +++ b/Doc/howto/concurrency.rst @@ -0,0 +1,1846 @@ +:tocdepth: 3 + +.. XXX reference vs. tutorial? + +.. _concurrency-howto: + +***************** +Concurrency HOWTO +***************** + +There are many outstanding resources, both online and in print, +that would do an excellent job of introducing you to concurrency. +This howto document builds on those by walking you through how +to apply that knowledge using Python. + +Python supports the following concurrency models directly: + +* **free-threading** (stdlib, C-API) +* **isolated threads**, *AKA CSP/actor model* (stdlib\*, C-API) +* **coroutines**, *AKA async/await* (language, stdlib, C-API) +* **multi-processing** (stdlib) +* **distributed**, *e.g. SMP* (stdlib (limited)) + +In this document, we'll look at how to take advantage of Python's +concurrency support. The overall focus is on the following: + +* `understanding the supported concurrency models `_ +* `factors to consider when designing a concurrent solution `_ +* `key concurrency primitives `_ +* `high-level, app-oriented practical examples `_ + +.. note:: + + You should always make sure concurrency is the right tool for the job + before you reach for it when solving your problem. There are many + cases where concurrency simply isn't applicable or will only + complicate the solution. In-depth discussion of this point + is outside the scope of this document. + +.. note:: + + Free-threading is one of the oldest concurrency models, fundamental + to operating systems, and widely supported in programming languages. + However, it is generally considered perilous and not human-friendly. + Other concurrency models have demonstrated better usability and + newer programming languages typically avoid exposing threads directly. + Take that into consideration before reaching for threads and look at + the alternatives first. + +.. note:: + + Python supports other concurrency models indirectly through + community-maintained PyPI packages. One well-known example is + :pypi:`dask`, which supports "distributed" computing. + + +Quick reference +=============== + +**Terminology** + +We'll be using the following terms and ideas throughout: + +task (logical thread) + | a cohesive *linear* sequence of abstract steps in a program; + | effectively, a mini-program; + | the logical equivalent of executed instructions corresponding to code; + | also known as "logical process" + +physical thread (OS thread) + | where the actual code for a logical thread runs on the CPU (and operating system); + | we avoid using using plain "thread" for this, to avoid ambiguity + +Python thread + | the Python runtime running in a physical thread + | particularly the portion of the runtime state active in the physical thread + | (see :class:`threading.Thread`) + +concurrency (multitasking) + | a program with multiple logical threads running simultaneously + | (not necessarily in parallel) + +parallelism (multi-core) + running a program's multiple logical threads on multiple physical + threads (CPU cores) + +.. raw:: html + + + +For convenience, here is a summary of what we'll cover later. + +**Concurrency Primitives** + +.. list-table:: + :header-rows: 1 + :class: borderless vert-aligned + :align: left + + * - primitive + - used with + - purpose + * - ... + - ... + - ... + +**High-level App Examples** + +.. list-table:: + :header-rows: 1 + :class: borderless vert-aligned + :align: left + + * - workload (app) + - per-request inputs + - per-request outputs + - *N* core tasks + - core task + * - `grep `_ + - | *N* filenames (**stdin**) + | file bytes x *N* (**disk**) + - *M* matches (**stdout**) + - 1+ per file + - | **time**: ~ file size + | **mem**: small + * - `... `_ + - ... + - ... + - ... + - ... + * - `... `_ + - ... + - ... + - ... + - ... + +Each has side-by-side implementations for the different models: + +.. list-table:: + :header-rows: 1 + :class: borderless vert-aligned + :align: left + + * - workload (app) + - side-by-side examples + * - `grep `_ + - `by concurrency models `_ + * - `... `_ + - `by concurrency models `_ + * - `... `_ + - `by concurrency models `_ + + +.. raw:: html + +
+ +---- + +.. _concurrency-models: + +Python Concurrency Models +========================= + +As mentioned, there are essentially five concurrency models that +Python supports directly: + +.. list-table:: + :header-rows: 1 + :class: borderless vert-aligned + :align: left + + * - model + - Python stdlib + - description + * - free threading + - :mod:`threading` + - | using multiple physical threads in the same process, + | with no isolation between them + * - | isolated threads + | (multiple interpreters) + - `interpreters `_ + - | threads, often physical, with strict isolation between them + | (e.g. CSP and actor model) + * - coroutines (async/await) + - :mod:`asyncio` + - switching between logical threads is explicitly controlled by each + * - multi-processing + - :mod:`multiprocessing` + - using multiple isolated processes + * - distributed + - | `multiprocessing `_ + | (limited) + - multiprocessing across multiple computers + +There are tradeoffs to each, whether in performance or complexity. +We'll take a look at those tradeoffs in detail +`later `_. + +Before that, we'll review various comparisons of the concurrency models, +and we'll briefly talk about `critical caveats `_ +for specific models. + +Comparison tables +----------------- + +The following tables provide a detailed look with side-by-side comparisons. + +key characteristics +^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :header-rows: 1 + :class: borderless vert-aligned + :align: left + + * - + - scale + - multi-core + - `races `_ + - overhead + * - free-threading + - small-medium + - `yes* `_ + - **yes** + - very low + * - multiple interpreters + - small-medium + - yes + - limited + - `low+ `_ + * - coroutines + - small-medium + - **no** + - no + - low + * - multi-processing + - small + - yes + - limited + - **medium** + * - distributed + - large + - yes + - limited + - **medium** + +overhead details +^^^^^^^^^^^^^^^^ + +.. list-table:: + :header-rows: 1 + :class: borderless vert-aligned + :align: left + + * - + - memory + - startup + - cross-task + - management + - system + * - free threading + - very low + - very low + - none + - very low + - none + * - multiple interpreters + - `low* `_ + - `medium* `_ + - low + - very low + - none + * - coroutines + - low + - low + - none + - low + - none + * - multi-processing + - medium + - medium + - medium + - medium + - low + * - distributed + - medium+ + - medium+ + - medium-high + - medium + - low-medium + +complexity +^^^^^^^^^^ + +.. TODO "human-friendly" + +.. list-table:: + :header-rows: 1 + :class: borderless vert-aligned + :align: left + + * - + - parallel + - | shared + | mem + - | shared + | I/O + - | shared + | env + - | cross + | thread + - :abbr:`sync (synchronization between logical threads)` + - :abbr:`tracking (how easy it is to keep track of where one logical thread is running relative to another, especially when one terminates)` + - :abbr:`compat (compatibility with code not using this concurrency model)` + - | extra + | LOC + * - free-threading + - `yes* `_ + - **all** + - **all** + - **yes** + - **high** + - **explicit** + - + - yes + - low? + * - multiple interpreters + - yes + - limited + - **all** + - **yes** + - low + - implicit + - ??? + - yes + - low? + * - coroutines + - **no** + - all + - all + - yes + - low-med? + - implicit + - ??? + - **no** + - low-med + * - multi-processing + - yes + - limited + - no + - no? + - low + - | implicit + | +optional + - ??? + - yes + - low-med? + * - distributed + - yes + - limited + - no + - no? + - low + - | implicit + | +optional + - ??? + - yes + - medium? + +exposure +^^^^^^^^ + +.. list-table:: + :header-rows: 1 + :class: borderless vert-aligned + :align: left + + * - + - | academic + | research + - | academic + | curriculum + - industry + - examples + - | Python + | history + * - free-threading + - very high + - high + - high + - high + - 0.9? + * - | isolated threads + | (multiple interpreters) + - high + - low? + - low-medium? + - low-medium? + - `2.2 `_ + * - coroutines + - medium-high? + - medium? + - medium? + - medium-high? + - 3.3-3.5 (2.2) + * - multi-processing + - ??? + - low? + - low-medium? + - low? + - 2.6 + * - distributed + - medium-high? + - low? + - medium? + - medium? + - n/a + +Critical caveats +---------------- + +Here are some important details to consider, specific to individual +concurrency models in Python. + +.. contents:: + :local: + +.. _concurrency-races: + +Data races and non-deterministic scheduling (free-threading) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The principal caveat for physical threads is that each thread +shares the *full* memory of the process with all its other threads. +Combined with their non-deterministic scheduling (and parallel +execution), threads expose programs to a significant risk of races. + +The potential consequences of a race are data corruption and invalidated +expectations of data consistency. In each case, the non-deterministic +scheduling of threads means it is both hard to reproduce races and to +track down where a race happened. These qualities make these bugs +especially frustrating and worth diligently avoiding. + +Python threads are light wrappers around physical threads and thus have +the same caveats. The majority of data in a Python program is mutable +and *all* of the program's data is subject to potential modification +by any thread at any moment. This requires extra effort, to synchronize +around reads and writes. Furthermore, given the maximally-broad scope +of the data involved, it's difficult to be sure all possible races +have been dealt with, especially as a code base changes over time. + +The other concurrency models essentially don't have this problem. +In the case of coroutines, explicit cooperative scheduling eliminates +the risk of a simultaneous read-write or write-write. It also means +program logic can rely on memory consistency between synchronization +points (``await``). + +With the remaining concurrency models, data is never shared between +logical threads unless done explicitly (typically at the existing +inherent points of synchronization). By default that shared data is +either read-only or managed in a thread-safe way. Most notably, +the opt-in sharing means the set of shared data to manage is +explicitly defined (and often small) instead of covering +*all* memory in the process. + +.. _python-gil: + +The Global Interpreter Lock (GIL) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +While physical threads are the direct route to multi-core parallelism, +Python's threads have always had an extra wrinkle that gets in the way: +the :term:`global interpreter lock` (GIL). + +The :term:`!GIL` is very efficient tool for keeping the Python +implementation simple, which is an important constraint for the project. +In fact, it protects Python's maintainers *and* users from a large +category of concurrency problems that one must normally face when +threads are involved. + +The big tradeoff is that the bytecode interpreter, which executes your +Python code, only runs while holding the :term:`!GIL`. That means only +one thread can be running Python code at a time. Threads will take +short turns, so none have to wait too long, but it still prevents +any actual parallelism of CPU-bound code. + +That said, the Python runtime (and extension modules) can release the +:term:`!GIL` when the thread is doing slow or long-running work +unrelated to Python, like a blocking IO operation. + +There is also an ongoing effort to eliminate the :term:`!GIL`: +:pep:`703`. Any attempt to remove the :term:`!GIL` necessarily involves +some slowdown to single-threaded performance and extra maintenance +burden to the Python project and extension module maintainers. +However, there is sufficient interest in unlocking full multi-core +parallelism to justify the current experiment. + +You can also move from free-threading to isolated threads using multiple +interpreters. Each interpreter has has its own +:term:`GIL `. Thus, If you want multi-core +parallelism, run a different interpreter in each thread. Their +isolation means that each can run unblocked in that thread. + +Thread isolation and multiple interpreters +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +As just noted, races effectively stop being a problem if the memory +used by each physical thread is effectively isolated from the others. +That isolation can also help with the other caveats related to +physical threads. In Python you can get this isolation +by using multiple interpreters. + +In this context, an "interpreter" represents nearly all the capability +and state of the Python runtime, for its C-API and to execute Python +code. The full runtime supports multiple interpreters and includes +some state that all interpreters share. Most importantly, the state +of each interpreter is effectively isolated from the others. + +That isolation includes things like :data:`sys.modules`. By default, +interpreters mostly don't share any data (including objects) at all. +Anything that gets shared is done on a strictly opt-in basis. That +means programmers wouldn't need to worry about possible races with +*any* data in the program. They would only need to worry about data +that was explicitly shared. + +Interpreters themselves are not specific to any thread, but instead +each physical thread has (at most) one interpreter active at any given +moment. Each interpreter can be associated in this way with any number +of threads. Since each interpreter is isolated from the others, +any thread using one interpreter is thus isolated from threads +using any other interpreter. + +Using multiple interpreters is fairly straight-forward: + +1. create a new interpreter +2. switch the current thread to use that interpreter +3. call :func:`exec`, but targeting the new interpreter +4. switch back + +Note that no threads were involved; running in an +interpreter happens relative to the current thread. New threads +aren't implicitly involved. + +Multi-processing and distributed computing provide similar isolation, +though with some tradeoffs. + +.. _python-stdlib-interpreters: + +A stdlib module for using multiple interpreters +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +While use of multiple interpreters has been part of Python's C-API +for decades, the feature hasn't been exposed to Python code through +the stdlib. :pep:`734` proposes changing that by adding a new +:mod:`!interpreters` module. + +In the meantime, an implementation of that PEP is available for +Python 3.13+ on PyPI: :pypi:`interpreters-pep-734`. + +.. _python-interpreters-overhead: + +Improving performance for multiple interpreters +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The long-running effort to improve on Python's implementation of multiple +interpreters focused on isolation and stability; very little done +to improve performance. This has the most impact on: + +* how much memory each interpreter uses + (i.e. how many can run at the same time) +* how long it takes to create a new interpreter + +It also impacts how efficiently data/objects can be passed between +interpreters, and how effectively objects can be shared. + +As the work on isolation wraps up, improvements will shift to focus +on performance and memory usage. Thus, the overhead of +using multiple interpreters will drastically decrease over time. + +Shared resources +^^^^^^^^^^^^^^^^ + +Aside from memory, all physical threads in a process share the +following resources: + +* command line arguments ("argv") +* env vars +* current working directory +* signals, IPC, etc. +* open I/O resources (file descriptors, sockets, etc.) + +When relevant, these must be managed in a thread-safe way. + +Tracing execution +^^^^^^^^^^^^^^^^^ + +TBD + +.. TODO finish + + The other potential problem with using threads is that the conceptual + model has no inherent synchronization, so it can be hard to follow + what is going on in the program at any given moment. That + would especially impact your efforts at testing and debugging. + + * "callback hell" + * "where was this thread/coroutine started?" + * composing a reliable sequential representation of the program? + * "what happened (in order) leading up to this point?" + + Besides unlocking full multi-core parallelism, the isolation between + interpreters means that, from a conceptual level, concurrency can be + simpler. + + The second category of complexity is the problem of tracing the execution + of one logical thread relative to another. This is especially relevant + for error handling, when an error in the one thread is exposed in the + other. This applies equally to threads that start other threads as to + concurrency models that use callbacks. Knowing where the failing thread + was started is valuable when debugging, as is knowing where a callback + was registered. + +Coroutines are contagious +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Coroutines can be an effective mechanism for letting a program's +non-blocking code run while simultaneously waiting for blocking code +to finish. The tricky part is that the underlying machinery (the +:ref:`event loop `) relies on each coroutine +explicitly yielding control at the appropriate moments. + +Normal functions do not follow this pattern, so they cannot take +advantage of that cooperative scheduling to avoid blocking +the program. Thus, coroutines and non-coroutines don't mix well. +While there are tools for wrapping normal functions to act like +coroutines, they are often converted into coroutines instead. +At that point, if any non-async code relies on the function then +either you'll need to convert the other code a coroutine or you'll +need to keep the original non-async implementation around along +with the new, almost identical async one. + +You can see how that can proliferate, leading to possible extra +maintenance/development costs. + +Processes consume extra resources +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When using multi-processing for concurrency, keep in mind that the +operating system will assign a certain set of limited resources to each +process. For example, each process has its own PID and handle to the +executable. You can run only so many processes before you run out of +these resources. Concurrency in a single process doesn't have this +problem, and a distributed program can work around it. + +.. _multiprocessing-distributed: + +Using multiprocessing for distributed computing +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Not only does the :mod:`multiprocessing` module support concurrency +with multiple local processes, it can also support a distributed model +using remote computers. That said, consider first looking into tools +that have been designed specifically for distributed computing, +like :pypi:`dask`. + +Resilience to crashes +^^^^^^^^^^^^^^^^^^^^^ + +A process can crash if it does something it shouldn't, like try to +access memory outside what the OS has provided it. If your program +is running in multiple processes (incl. distributed) then you can +more easily recover from a crash in any one process. Recovering +from a crash when using free-threading, multiple interpreters, or +coroutines isn't nearly so easy. + +High-level APIs +--------------- + +Also note that Python's stdlib provides various higher-level APIs +that support these concurrency models in various contexts: + +.. list-table:: + :header-rows: 1 + :class: borderless vert-aligned + :align: left + + * - + - :mod:`concurrent.futures` + - :mod:`socketserver` + - :mod:`http.server` + * - free-threading + - :class:`yes ` + - :class:`yes ` + - :class:`yes ` + * - multiple interpreters + - (`pending `_) + - + - + * - coroutines + - ??? + - + - + * - multi-processing + - | :class:`yes ` + | (:class:`similar `) + - :class:`yes ` + - + * - distributed + - ??? + - + - + + +.. raw:: html + +
+ +---- + +.. _concurrency-design: + +Designing A Program For Concurrency +=================================== + +Whether you are starting a new project using concurrency or refactoring +an existing one to use it, it's important to design for concurrency +before taking one more step. Doing so will save you a lot of +headache later. + +1. decide if your program *might* benefit from concurrency +2. `break down your *logical* program into distinct tasks `_ +3. `determine which tasks could run at the same time `_ +4. `identify the other concurrency-related characteristics of your program `_ +5. `decide which concurrency model fits best `_ +6. go for it! + +At each step you should be continuously asking yourself if concurrency +is still a good fit for your program. + +Some problems are obviously not solvable with concurrency. Otherwise, +even if you *could* use concurrency, it might not provide much value. +Furthermore, even if it seems like it would provide meaningful value, +the additional costs in performance, complexity, or maintainability +might outweigh that benefit. + +Thus, when you're thinking of solving a problem using concurrency, +it's crucial that you understand the problem well. + +Getting started +--------------- + +How can concurrency help? +^^^^^^^^^^^^^^^^^^^^^^^^^ + +TBD + +.. TODO finish + + Here are the benefits concurrency can bring to the table: + + * ... + + + Primarily, concurrency can be helpful by making your program faster + and more responsive (less latency), when possible. In other words, + you get better computational throughput. That happens by enabling + the following: + + * run on multiple CPU cores (parallelism) + * keep blocking resources from blocking the whole program + * make sure critical tasks have priority + * make sure other tasks have a fair share of time + * process results as they come, instead of waiting for them all + + Other possible benefits: + + * asynchronous events can be handled more cleanly + * better efficiency using hardware resources + * improved scalability + +How can concurrency hurt? +^^^^^^^^^^^^^^^^^^^^^^^^^ + +TBD + +.. TODO finish + + The main challenge when using concurrency is the (potential) extra + complexity. This complexity comes from the effect of multiple logical + threads running at the same time and interacting with each other. + In practice, this falls into two categories: data races and tracing + relative execution. Both are a form of "spooky action at a distance" [#f1]_ + (meaning something changes unexpectedly in one place due to unknown + changes somewhere else). + +Analyze your problem +-------------------- + +.. _concurrency-identify-tasks: + +Identifying the logical tasks in your program +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +TBD + +.. TODO finish + + At its most fundamental, concurrency means doing multiple things at once, + from a strictly *logical* viewpoint. + + When a computer program runs, it executes a sequence of code + in a given order. If you were to trace the actual execution, you would + still end up with a *linear* series of executed instructions that matches + the code. We call this sequence of code (and instructions) a logical + "thread" of execution. + + Sometimes it makes sense to break up that sequence into smaller pieces, + where some of them can run independently of others. Thus, the program + then involves multiple logical threads. This is also called + "multitasking" and each logical thread a "task". + + One important observation is that most concurrent programs + can be represented instead as a single task, with the code of the + concurrent tasks merged into a single sequence. + +.. _concurrency-characteristics: + +The concurrency characteristics of your program +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +TBD + +.. TODO finish + + For a given workload, here are some characteristics that will help you + understand the problem and, potentially, which concurrency model would + be the best fit: + + * requests + + * frequency + * expected latency for (at least partial) response + + * inputs per request + + * how many + * size of each input + + * tasks (logical threads) per input + + * how many + * variety vs. uniformity + * compute per task: how much + * data per task: how much and what kinds + * I/O per task: how much and what kinds + * tasks not tied to outputs + + * task interaction + + * how much and in what ways + * what data is shared between tasks + * how much blocking while waiting + + * outputs per request + + * how many + * size pf each output + * correlation to inputs + + To some extent the most critical factors can be compressed down to: + + * many inputs vs. 1 large divisible input + * many outputs vs. combined output vs. matching large output + * many short computations vs. fewer medium/long computations + + We could also break it down into quadrants:: + + . stream of tasks queue of tasks + C | + P | + U | + - | + b | + o | + u | + n | + d | + -----------------------|----------------------- + I | + O | + - | + b | + o | + u | + n | + d | + + + Aside from the concurrency model, the answers to the above can impact + the following: + + * use of a worker pool + * use of background tasks/threads + + In the context of the above characteristics, let's revisit the ways that + concurrency can be helpful: + + * get work done faster + + * run more tasks at once (multi-core) + + * make the app feel more responsive + + * make sure critical tasks have priority + * process results as they come, instead of waiting for them all + * send payload to multiple targets before starting next task + + * use system resources more efficiently + + * keep slow parts from blocking fast parts + * keep blocking resources from blocking the whole program + * make sure other tasks have a fair share of time + * task scheduling & resource usage optimization + + * scaling + * handle asynchronous events + +Other considerations +^^^^^^^^^^^^^^^^^^^^ + +TBD + +.. TODO finish + + * are there libraries that can take care of the concurrency parts? + * ... + +.. _concurrency-pick-a-model: + +Pick a concurrency model +------------------------ + +TBD + +.. TODO finish + + As mentioned earlier, each concurrency model has its own set of tradeoffs. + Free-threading probably has the most notoriety and the most examples, + but it also has the most pitfalls (see `Critical caveats`_ above). + Isolated threads have few of those pitfalls but are less familiar + and at least a little less efficient. + Multiprocessing and distributed computing are likewise isolated, + but less efficient, which can have a larger negative impact + at smaller scales. + Async can be straightforward, but may cascade throughout a code base + and doesn't provide parallelism. + + free-threading: + + * main value: efficient multi-core + * main costs: races & conceptual overhead + + * minimal conceptual indirection: closely tied to low-level physical threads + * the most direct route to taking advantage of multi-core parallelism + + + A high-level look: + + .. list-table:: + :header-rows: 1 + :class: borderless vert-aligned + :align: left + + * - model + - pros + - cons + * - free threading + - * very light-weight and efficient + * wide-spread + * can enable multi-core parallelism (`caveat: GIL `_) + - * all memory is shared, subject to races + * some IO may have races (e.g. writing to stdout) + * can be hard for humans to follow what's happening in different + threads at any given point + * - multiple interpreters (isolated threads) + - * isolation eliminates nearly all races, by default + (sharing is strictly opt-in) + * synchronization is built in to cross-interpreter interaction + * enables full multi-core parallelism of all Python code + - * unfamiliar to many + * less efficient than threads + * (currently) limited in what data can be shared between + interpreters + * - coroutines (async/await) + - * not subject to races + * increasingly familiar to many; popular in newer languages + * has a long history in Python (e.g. ``twisted``) + - * async and non-async functions don't mix well, + potentially leading to duplication of code + * switching to async can require substantial cascading code churn + * callbacks can make it difficult to follow program logic, + making debugging harder + * does not enable multi-core parallelism + * - multiprocessing + - * isolated (no races) + * enables full multi-core parallelism of all Python code + - * substantially less efficient than using a single process + * can lead to exhaustion of system resources + (e.g. file handles, PIDs) + * API can be hard to use + * - distributed + - * isolated (no races) + * fully parallel + * facilitates massive scaling + - * not necessarily a good fit for small-scale applications + * often requires configuration + + +.. raw:: html + +
+ +---- + +.. _concurrency-primitives: + +Python Concurrency Primitives +============================= + +TBD + +.. TODO finish + + Dealing with data races is often managed using locks (AKA mutexes), + at a low level, and thread-safe types and APIs at a high level. + Depending on the programming language, the complexity is sometimes + mitigated by the compiler and runtime. There are even + libraries and frameworks that help abstract away the complexity + to an extent. On top of that, there are tools that can help identify + potential races via static analysis. Unfortunately, none of these aids + is foolproof and the risk of hitting a race is always looming. + + synchronization + --------------- + + Additionally, concurrency often involves some degree of synchronization + between the logical threads. At the most basic conceptual level: + one thread may wait for another to finish. + + shared resources + ---------------- + + Aside from code running at the same time, concurrency typically + also involves some amount of resources shared between the concurrent + tasks. That may include memory, files, and sockets. + +Group A +------- + +primitive 1 +^^^^^^^^^^^ + +TBD + +.. TODO finish + +Group B +------- + +primitive 1 +^^^^^^^^^^^ + +TBD + +.. TODO finish + + +.. raw:: html + +
+ +---- + +.. XXX Move this section to a separate doc? + +.. _concurrency-workload-examples: + +Python Concurrency Workload Examples +==================================== + +Below, we have a series of examples of how to implement the most +common Python workloads that take advantage of concurrency. +For each workload, you will find an implementation for each of the +concurrency models. + +The implementations are meant to accurately demonstrate how best +to solve the problem using the given concurrency model. The examples +for the workload are presented side-by-side, for easier comparison. +The examples for threads, multiprocessing, and multiple interpreters +will use :mod:`concurrent.futures` when that is the better approach. +Performance comparisons are not included here. + +Here's a summary of the examples, by workload: + +.. list-table:: + :header-rows: 1 + :class: borderless vert-aligned + :align: left + + * - workload + - req in + - req out + - *N* core tasks + - core task + * - `grep `_ + - | *N* filenames (**stdin**) + | file bytes x *N* (**disk**) + - *M* matches (**stdout**) + - 1+ per file + - | **time**: ~ file size + | **mem**: small + * - `... `_ + - ... + - ... + - ... + - ... + * - `... `_ + - ... + - ... + - ... + - ... + +.. other examples: + + * (scientific, finance, ML, matrices) + * conway's game of life + * raytracer + * mandelbrot + * find primes + * compute factorials + * + * + + * + * + * + +Also see: + +* https://github.com/faster-cpython/ideas/wiki/Tables:-Workloads +* https://github.com/ericsnowcurrently/concurrency-benchmarks + +.. note:: + + Each example is implemented as a basic command line tool, but can be + easily adapted to run as a web service. + +Workload: grep +-------------- + +This is a basic Python implementation of the linux ``grep`` tool. +We read from one or more files and report about lines that match +(or don't match) the given regular expression. + +This represents a workload involving a mix of moderate IO and CPU work. + +For full example code see the `side-by-side implementations +`_ below. + +Design and analysis +^^^^^^^^^^^^^^^^^^^ + +Design steps from `above `_: + +1. concurrency fits? + + Yes! There is potentially a bunch of work happening at the same time, + and we want results as fast as possible. + +2. identify logical tasks + + At a high level, the application works like this: + + 1. handle args (including compile regex) + 2. if recursive, walk tree to find filenames + 3. for each file, yield each match + 4. print each match + 5. exit with 0 if matched and 1 otherwise + + At step 3 we do the following for each file: + + a. open the file + b. iterate over the lines + c. apply the regex to each line + d. yield each match + e. close the file + +3. select concurrent tasks + + Concurrent work happens at step 3. Sub-steps a, b, and e are + IO-intensive. Sub-step c is CPU-intensive. The simplest approach + would be one concurrent worker per file. Relative to a strictly + sequential approach, there's extra complexity here in managing the + workers, fanning out the work to them, and merging the results back + into a single iterator. + + If we were worried about any particularly large file or sufficiently + large regular expression, we could take things further. That would + involve splitting up step 3 even further by breaking the file into + chunks that are divided up among multiple workers. However, doing + so would introduce extra complexity that might not pay for itself. + +4. concurrency-related characteristics + + TBD + + .. TODO finish + +5. pick best model + + TBD + + .. TODO finish + +Here are additional key constraints and considerations: + +* there's usually a limit to how many files can be open concurrently, + so we'll have to be careful not to process too many at once +* the order of the yielded/printed matches must match the order of the + requested files and the order of each files lines + +High-level code +^^^^^^^^^^^^^^^ + +With the initial design and analysis done, let's move on to code. +We'll start with the high-level code corresponding to the application's +five top-level tasks we identified earlier. + +Most of the high-level code has nothing to do with concurrency. +The part that does, ``search()``, is highlighted. + +.. raw:: html + +
+ (expand) + +.. literalinclude:: ../includes/concurrency/grep-parts.py + :start-after: [start-high-level] + :end-before: [end-high-level] + :dedent: + :linenos: + :emphasize-lines: 7 + +.. raw:: html + +
+ +The ``search()`` function that gets called returns an iterator +(or async iterator) that yields the matches, which get printed. +Here's the high-level code again, but with highlighting on each line +that uses the iterator. + +.. raw:: html + +
+ (expand) + +.. literalinclude:: ../includes/concurrency/grep-parts.py + :start-after: [start-high-level] + :end-before: [end-high-level] + :dedent: + :linenos: + :emphasize-lines: 13,16,22,35,38,42,44,47,53,66,69 + +.. raw:: html + +
+ +Here's the search function for a non-concurrent implementation: + +.. literalinclude:: ../includes/concurrency/grep-parts.py + :start-after: [start-impl-sequential] + :end-before: [end-impl-sequential] + :linenos: + +``iter_lines()`` is a straight-forward helper that opens the file +and yields each line. + +``search_lines()`` is a sequential-search helper used by all the +example implementations here: + +.. literalinclude:: ../includes/concurrency/grep-parts.py + :start-after: [start-search-lines] + :end-before: [end-search-lines] + :linenos: + +Concurrent Code +^^^^^^^^^^^^^^^ + +Now lets look at how concurrency actually fits in. We'll start +with an example using threads. However, the pattern is essentially +the same for all the concurrency models. + +.. literalinclude:: ../includes/concurrency/grep-parts.py + :start-after: [start-impl-threads] + :end-before: [end-impl-threads] + :linenos: + +We loop over the filenames and start a thread for each one. Each one +sends the matches it finds back using a queue. + +We want to start yielding matches as soon as possible, so we also use +a background thread to run the code that loops over the filenames. + +We use a queue of queues (``matches_by_file``) to make sure +we get results back in the right order, regardless of when the worker +threads provide them. + +The operating system will only let us have so many files open at once, +so we limit how many workers are running. (``MAX_FILES``) + +If the workers find matches substantially faster than we can use them +then we may end up using more memory than we need to. To avoid any +backlog, we limit how many matches can be queued up for any given file. +(``MAX_MATCHES``) + +One notable point is that the actual files are not opened until +we need to iterate over the lines. For the most part, this is so we +can avoid dealing with passing an open file to a concurrency worker. +Instead we pass the filename, which is much simpler. + +Finally, we have to manage the workers manually. If we used +`concurrent.futures`_, it would take care of that for us. + +.. TODO finish + +Here are some things we don't do but *might* be worth doing: + +* stop iteration when requested (or for ``ctrl-C``) +* split up each file between multiple workers +* ... + +Recall that the ``search()`` function returns an iterator that yields +all the matches. Concurrency may be happening as long as that iterator +hasn't been exhausted. That means it is happening more or less the +entire time we loop over the matches to print them in ``main()`` +(in the high-level code above). + +.. _concurrency-grep-side-by-side: + +Side-by-side +^^^^^^^^^^^^ + +Here are the implementations for the different concurrency models, +side-by-side for easy comparison (main differences highlighted): + +.. list-table:: + :header-rows: 1 + :class: borderless vert-aligned + :align: left + + * - sequential + - threads + - multiple interpreters + - coroutines + - multiple processes + - concurrent.futures + * - .. raw:: html + +
+ (expand) + + .. literalinclude:: ../includes/concurrency/grep-sequential.py + :linenos: + :emphasize-lines: 7-11 + + .. raw:: html + +
+ + - .. raw:: html + +
+ (expand) + + .. literalinclude:: ../includes/concurrency/grep-threads.py + :linenos: + :emphasize-lines: 6-52 + + .. raw:: html + +
+ + - .. raw:: html + +
+ (expand) + + .. literalinclude:: ../includes/concurrency/grep-interpreters.py + :linenos: + :emphasize-lines: 6-82 + + .. raw:: html + +
+ + - .. raw:: html + +
+ (expand) + + .. literalinclude:: ../includes/concurrency/grep-asyncio.py + :linenos: + :emphasize-lines: 6-53 + + .. raw:: html + +
+ + - .. raw:: html + +
+ (expand) + + .. literalinclude:: ../includes/concurrency/grep-multiprocessing.py + :linenos: + :emphasize-lines: 6-83 + + .. raw:: html + +
+ + - .. raw:: html + +
+ (expand) + + .. literalinclude:: ../includes/concurrency/grep-threads-cf.py + :linenos: + :emphasize-lines: 6-46 + + .. raw:: html + +
+ +Model-specific details +^^^^^^^^^^^^^^^^^^^^^^ + +Here are some implementation-specific details we had to deal with. + +threads: + +* ... + +interpreters: + +* ... + +multiprocessing: + +* ... + +asyncio: + +* ... + +concurrent.futures +^^^^^^^^^^^^^^^^^^ + +For threads, multiprocessing, and +`multiple interpreters * `_, +you can also use :mod:`concurrent.futures`: + +.. raw:: html + +
+ (expand) + +.. literalinclude:: ../includes/concurrency/grep-parts.py + :start-after: [start-impl-cf-threads] + :end-before: [end-impl-cf-threads] + :linenos: + +.. raw:: html + +
+ +For processes`, use :class:`concurrent.futures.ProcessPoolExecutor`. +For interpreters, use :class:`!InterpreterPoolExecutor`. +In both cases you must use the proper queue type and there +are a few other minor differences. + + +.. raw:: html + +
+ + +Workload 2: ... +--------------- + +.. TODO include full code + +TBD + +.. TODO finish + +Design and analysis +^^^^^^^^^^^^^^^^^^^ + +Design steps from `above `_: + +1. concurrency fits? + + TBD + + .. TODO finish + +2. identify logical tasks + + TBD + + .. TODO finish + +3. select concurrent tasks + + TBD + + .. TODO finish + +4. concurrency-related characteristics + + TBD + + .. TODO finish + +5. pick best model + + TBD + + .. TODO finish + +Here are additional key constraints and considerations: + +* ... + +High-level code +^^^^^^^^^^^^^^^ + +# ... + +.. _concurrency-example-2-side-by-side: + +Side-by-side +^^^^^^^^^^^^ + +Here's the implementations for the different concurrency models, +side-by-side for easy comparison: + +.. list-table:: + :header-rows: 1 + :class: borderless vert-aligned + :align: left + + * - sequential + - threads + - multiple interpreters + - coroutines + - multiple processes + - concurrent.futures + * - .. raw:: html + +
+ (expand) + + .. literalinclude:: ../includes/concurrency/run-examples.py + :start-after: [start-w2-sequential] + :end-before: [end-w2-sequential] + :dedent: + :linenos: + + .. raw:: html + +
+ + - .. raw:: html + +
+ (expand) + + .. literalinclude:: ../includes/concurrency/run-examples.py + :start-after: [start-w2-threads] + :end-before: [end-w2-threads] + :dedent: + :linenos: + + .. raw:: html + +
+ + - .. raw:: html + +
+ (expand) + + .. literalinclude:: ../includes/concurrency/run-examples.py + :start-after: [start-w2-subinterpreters] + :end-before: [end-w2-subinterpreters] + :dedent: + :linenos: + + .. raw:: html + +
+ + - .. raw:: html + +
+ (expand) + + .. literalinclude:: ../includes/concurrency/run-examples.py + :start-after: [start-w2-async] + :end-before: [end-w2-async] + :dedent: + :linenos: + + .. raw:: html + +
+ + - .. raw:: html + +
+ (expand) + + .. literalinclude:: ../includes/concurrency/run-examples.py + :start-after: [start-w2-multiprocessing] + :end-before: [end-w2-multiprocessing] + :dedent: + :linenos: + + .. raw:: html + +
+ + - .. raw:: html + +
+ (expand) + + .. literalinclude:: ../includes/concurrency/run-examples.py + :start-after: [start-w2-cf] + :end-before: [end-w2-cf] + :dedent: + :linenos: + + .. raw:: html + +
+ + +.. raw:: html + +
+ + +Workload 3: ... +--------------- + +.. TODO include full code + +TBD + +.. TODO finish + +Design and analysis +^^^^^^^^^^^^^^^^^^^ + +Design steps from `above `_: + +1. concurrency fits? + + TBD + + .. TODO finish + +2. identify logical tasks + + TBD + + .. TODO finish + +3. select concurrent tasks + + TBD + + .. TODO finish + +4. concurrency-related characteristics + + TBD + + .. TODO finish + +5. pick best model + + TBD + + .. TODO finish + +Here are additional key constraints and considerations: + +* ... + +High-level code +^^^^^^^^^^^^^^^ + +# ... + +.. _concurrency-example-3-side-by-side: + +Side-by-side +^^^^^^^^^^^^ + +Here's the implementations for the different concurrency models, +side-by-side for easy comparison: + +.. list-table:: + :header-rows: 1 + :class: borderless vert-aligned + :align: left + + * - sequential + - threads + - multiple interpreters + - coroutines + - multiple processes + - concurrent.futures + * - .. raw:: html + +
+ (expand) + + .. literalinclude:: ../includes/concurrency/run-examples.py + :start-after: [start-w3-sequential] + :end-before: [end-w3-sequential] + :dedent: + :linenos: + + .. raw:: html + +
+ + - .. raw:: html + +
+ (expand) + + .. literalinclude:: ../includes/concurrency/run-examples.py + :start-after: [start-w3-threads] + :end-before: [end-w3-threads] + :dedent: + :linenos: + + .. raw:: html + +
+ + - .. raw:: html + +
+ (expand) + + .. literalinclude:: ../includes/concurrency/run-examples.py + :start-after: [start-w3-subinterpreters] + :end-before: [end-w3-subinterpreters] + :dedent: + :linenos: + + .. raw:: html + +
+ + - .. raw:: html + +
+ (expand) + + .. literalinclude:: ../includes/concurrency/run-examples.py + :start-after: [start-w3-async] + :end-before: [end-w3-async] + :dedent: + :linenos: + + .. raw:: html + +
+ + - .. raw:: html + +
+ (expand) + + .. literalinclude:: ../includes/concurrency/run-examples.py + :start-after: [start-w3-multiprocessing] + :end-before: [end-w3-multiprocessing] + :dedent: + :linenos: + + .. raw:: html + +
+ + - .. raw:: html + +
+ (expand) + + .. literalinclude:: ../includes/concurrency/run-examples.py + :start-after: [start-w3-cf] + :end-before: [end-w3-cf] + :dedent: + :linenos: + + .. raw:: html + +
+ + +.. XXX + + .. rubric:: Footnotes + + .. [#f1] The phrase was originally said by Albert Einstein about + quantum entanglement. diff --git a/Doc/howto/index.rst b/Doc/howto/index.rst index f350141004c2db..430f0e6b43e814 100644 --- a/Doc/howto/index.rst +++ b/Doc/howto/index.rst @@ -11,6 +11,7 @@ Python Library Reference. :maxdepth: 1 :hidden: + concurrency.rst cporting.rst curses.rst descriptor.rst @@ -53,6 +54,7 @@ General: Advanced development: +* :ref:`concurrency-howto` * :ref:`curses-howto` * :ref:`freethreading-python-howto` * :ref:`freethreading-extensions-howto` diff --git a/Doc/includes/concurrency/grep-asyncio.py b/Doc/includes/concurrency/grep-asyncio.py new file mode 100644 index 00000000000000..59accf564773e9 --- /dev/null +++ b/Doc/includes/concurrency/grep-asyncio.py @@ -0,0 +1,202 @@ +import os +import os.path +import re +import sys + +import asyncio + + +async def search(filenames, regex, opts): + matches_by_file = asyncio.Queue() + + async def do_background(): + MAX_FILES = 10 + MAX_MATCHES = 100 + + # Make sure we don't have too many coros at once, + # i.e. too many files open at once. + counter = asyncio.Semaphore(MAX_FILES) + + async def search_file(filename, matches): + # aiter_lines() opens the file too. + lines = iter_lines(filename) + async for m in search_lines( + lines, regex, opts, filename): + await matches.put(match) + await matches.put(None) + # Let a new coroutine start. + counter.release() + + async with asyncio.TaskGroup() as tg: + for filename in filenames: + # Prepare for the file. + matches = asyncio.Queue(MAX_MATCHES) + await matches_by_file.put(matches) + + # Start a coroutine to process the file. + tg.create_task( + search_file(filename, matches), + ) + await counter.acquire() + await matches_by_file.put(None) + + background = asyncio.create_task(do_background()) + + # Yield the results as they are received, in order. + matches = await matches_by_file.get() # blocking + while matches is not None: + match = await matches.get() # blocking + while match is not None: + yield match + match = await matches.get() # blocking + matches = await matches_by_file.get() # blocking + + await asyncio.wait([background]) + + +async def iter_lines(filename): + if filename == '-': + infile = sys.stdin + line = await read_line_async(infile) + while line: + yield line + line = await read_line_async(infile) + else: + # XXX Open using async? + with open(filename) as infile: + line = await read_line_async(infile) + while line: + yield line + line = await read_line_async(infile) + + +async def read_line_async(infile): + # XXX Do this async! + # maybe make use of asyncio.to_thread() + # or loop.run_in_executor()? + return infile.readline() + + +async def search_lines(lines, regex, opts, filename): + try: + if opts.filesonly: + if opts.invert: + async for line in lines: + m = regex.search(line) + if m: + break + else: + yield (filename, None) + else: + async for line in lines: + m = regex.search(line) + if m: + yield (filename, None) + break + else: + assert not opts.invert, opts + async for line in lines: + m = regex.search(line) + if not m: + continue + if line.endswith(os.linesep): + line = line[:-len(os.linesep)] + yield (filename, line) + except UnicodeDecodeError: + # It must be a binary file. + return + + +def resolve_filenames(filenames, recursive=False): + for filename in filenames: + assert isinstance(filename, str), repr(filename) + if filename == '-': + yield '-' + elif not os.path.isdir(filename): + yield filename + elif recursive: + for d, _, files in os.walk(filename): + for base in files: + yield os.path.join(d, base) + + +if __name__ == '__main__': + # Parse the args. + import argparse + ap = argparse.ArgumentParser(prog='grep') + + ap.add_argument('-r', '--recursive', + action='store_true') + ap.add_argument('-L', '--files-without-match', + dest='filesonly', + action='store_const', const='invert') + ap.add_argument('-l', '--files-with-matches', + dest='filesonly', + action='store_const', const='match') + ap.add_argument('-q', '--quiet', action='store_true') + ap.set_defaults(invert=False) + + reopts = ap.add_mutually_exclusive_group(required=True) + reopts.add_argument('-e', '--regexp', dest='regex', + metavar='REGEX') + reopts.add_argument('regex', nargs='?', + metavar='REGEX') + + ap.add_argument('files', nargs='+', metavar='FILE') + + opts = ap.parse_args() + ns = vars(opts) + + regex = ns.pop('regex') + filenames = ns.pop('files') + recursive = ns.pop('recursive') + if opts.filesonly: + if opts.filesonly == 'invert': + opts.invert = True + else: + assert opts.filesonly == 'match', opts + opts.invert = False + opts.filesonly = bool(opts.filesonly) + + async def main(regex=regex, filenames=filenames): + # step 1 + regex = re.compile(regex) + # step 2 + filenames = resolve_filenames(filenames, recursive) + # step 3 + matches = search(filenames, regex, opts) + matches = type(matches).__aiter__(matches) + + # step 4 + + # Handle the first match. + async for filename, line in matches: + if opts.quiet: + return 0 + elif opts.filesonly: + print(filename) + else: + async for second in matches: + print(f'{filename}: {line}') + filename, line = second + print(f'{filename}: {line}') + break + else: + print(line) + break + else: + return 1 + + # Handle the remaining matches. + if opts.filesonly: + async for filename, _ in matches: + print(filename) + else: + async for filename, line in matches: + print(f'{filename}: {line}') + + return 0 + rc = asyncio.run(main()) + + # step 5 + sys.exit(rc) diff --git a/Doc/includes/concurrency/grep-interpreters.py b/Doc/includes/concurrency/grep-interpreters.py new file mode 100644 index 00000000000000..4bb0b7d865aed7 --- /dev/null +++ b/Doc/includes/concurrency/grep-interpreters.py @@ -0,0 +1,228 @@ +import os +import os.path +import re +import sys + +import test.support.interpreters as interpreters +import test.support.interpreters.queues as interp_queues +import types +import queue +import threading + + +def search(filenames, regex, opts): + matches_by_file = queue.Queue() + + def do_background(): + MAX_FILES = 10 + MAX_MATCHES = 100 + new_queue = interpreters.queues.create + + def new_interpreter(): + interp = interpreters.create() + interp.exec(f"""if True: + with open({__file__!r}) as infile: + text = infile.read() + ns = dict() + exec(text, ns, ns) + prep_interpreter = ns['prep_interpreter'] + del ns, text + + search_file = prep_interpreter( + {regex.pattern!r}, + {regex.flags}, + {tuple(vars(opts).items())}, + ) + """) + return interp + + ready_workers = queue.Queue(MAX_FILES) + workers = [] + + def next_worker(): + if len(workers) < MAX_FILES: + interp = new_interpreter() + workers.append(interp) + ready_workers.put(interp) + return ready_workers.get() # blocking + + def do_work(filename, matches, interp): + interp.prepare_main(matches=matches) + interp.exec( + f'search_file({filename!r}, matches)') + # Let a new thread start. + ready_workers.put(interp) + + for filename in filenames: + # Prepare for the file. + matches = interp_queues.create(MAX_MATCHES) + matches_by_file.put(matches) + interp = next_worker() + + # Start a thread to process the file. + t = threading.Thread( + target=do_work, + args=(filename, matches, interp), + ) + t.start() + matches_by_file.put(None) + + background = threading.Thread(target=do_background) + background.start() + + # Yield the results as they are received, in order. + matches = matches_by_file.get() # blocking + while matches is not None: + match = matches.get() # blocking + while match is not None: + yield match + match = matches.get() # blocking + matches = matches_by_file.get() # blocking + + background.join() + + +def prep_interpreter(regex_pat, regex_flags, opts): + regex = re.compile(regex_pat, regex_flags) + opts = types.SimpleNamespace(**dict(opts)) + + def search_file(filename, matches): + lines = iter_lines(filename) + for match in search_lines( + lines, regex, opts, filename): + matches.put(match) # blocking + matches.put(None) # blocking + return search_file + + +def iter_lines(filename): + if filename == '-': + yield from sys.stdin + else: + with open(filename) as infile: + yield from infile + + +def search_lines(lines, regex, opts, filename): + try: + if opts.filesonly: + if opts.invert: + for line in lines: + m = regex.search(line) + if m: + break + else: + yield (filename, None) + else: + for line in lines: + m = regex.search(line) + if m: + yield (filename, None) + break + else: + assert not opts.invert, opts + for line in lines: + m = regex.search(line) + if not m: + continue + if line.endswith(os.linesep): + line = line[:-len(os.linesep)] + yield (filename, line) + except UnicodeDecodeError: + # It must be a binary file. + return + + +def resolve_filenames(filenames, recursive=False): + for filename in filenames: + assert isinstance(filename, str), repr(filename) + if filename == '-': + yield '-' + elif not os.path.isdir(filename): + yield filename + elif recursive: + for d, _, files in os.walk(filename): + for base in files: + yield os.path.join(d, base) + + +if __name__ == '__main__': + # Parse the args. + import argparse + ap = argparse.ArgumentParser(prog='grep') + + ap.add_argument('-r', '--recursive', + action='store_true') + ap.add_argument('-L', '--files-without-match', + dest='filesonly', + action='store_const', const='invert') + ap.add_argument('-l', '--files-with-matches', + dest='filesonly', + action='store_const', const='match') + ap.add_argument('-q', '--quiet', action='store_true') + ap.set_defaults(invert=False) + + reopts = ap.add_mutually_exclusive_group(required=True) + reopts.add_argument('-e', '--regexp', dest='regex', + metavar='REGEX') + reopts.add_argument('regex', nargs='?', + metavar='REGEX') + + ap.add_argument('files', nargs='+', metavar='FILE') + + opts = ap.parse_args() + ns = vars(opts) + + regex = ns.pop('regex') + filenames = ns.pop('files') + recursive = ns.pop('recursive') + if opts.filesonly: + if opts.filesonly == 'invert': + opts.invert = True + else: + assert opts.filesonly == 'match', opts + opts.invert = False + opts.filesonly = bool(opts.filesonly) + + def main(regex=regex, filenames=filenames): + # step 1 + regex = re.compile(regex) + # step 2 + filenames = resolve_filenames(filenames, recursive) + # step 3 + matches = search(filenames, regex, opts) + matches = iter(matches) + + # step 4 + + # Handle the first match. + for filename, line in matches: + if opts.quiet: + return 0 + elif opts.filesonly: + print(filename) + else: + for second in matches: + print(f'{filename}: {line}') + filename, line = second + print(f'{filename}: {line}') + break + else: + print(line) + break + else: + return 1 + + # Handle the remaining matches. + if opts.filesonly: + for filename, _ in matches: + print(filename) + else: + for filename, line in matches: + print(f'{filename}: {line}') + + return 0 + rc = main() + + # step 5 + sys.exit(rc) diff --git a/Doc/includes/concurrency/grep-multiprocessing-cf.py b/Doc/includes/concurrency/grep-multiprocessing-cf.py new file mode 100644 index 00000000000000..3b784ed96d7195 --- /dev/null +++ b/Doc/includes/concurrency/grep-multiprocessing-cf.py @@ -0,0 +1,184 @@ +import os +import os.path +import re +import sys + +from concurrent.futures import ProcessPoolExecutor +import multiprocessing +import queue +import threading + + +def search(filenames, regex, opts): + matches_by_file = queue.Queue() + + def do_background(): + MAX_FILES = 10 + MAX_MATCHES = 100 + + with ProcessPoolExecutor(MAX_FILES) as workers: + for filename in filenames: + # Prepare for the file. + matches = multiprocessing.Queue(MAX_MATCHES) + matches_by_file.put(matches) + + # Start a thread to process the file. + workers.submit( + search_file, filename, matches) + matches_by_file.put(None) + + background = threading.Thread(target=do_background) + background.start() + + # Yield the results as they are received, in order. + matches = matches_by_file.get() # blocking + while matches is not None: + match = matches.get() # blocking + while match is not None: + yield match + match = matches.get() # blocking + matches = matches_by_file.get() # blocking + + background.join() + + +def search_file(filename, matches, regex, opts): + lines = iter_lines(filename) + for match in search_lines(lines, regex, opts, filename): + matches.put(match) # blocking + matches.put(None) # blocking + + +def iter_lines(filename): + if filename == '-': + yield from sys.stdin + else: + with open(filename) as infile: + yield from infile + + +def search_lines(lines, regex, opts, filename): + try: + if opts.filesonly: + if opts.invert: + for line in lines: + m = regex.search(line) + if m: + break + else: + yield (filename, None) + else: + for line in lines: + m = regex.search(line) + if m: + yield (filename, None) + break + else: + assert not opts.invert, opts + for line in lines: + m = regex.search(line) + if not m: + continue + if line.endswith(os.linesep): + line = line[:-len(os.linesep)] + yield (filename, line) + except UnicodeDecodeError: + # It must be a binary file. + return + + +def resolve_filenames(filenames, recursive=False): + for filename in filenames: + assert isinstance(filename, str), repr(filename) + if filename == '-': + yield '-' + elif not os.path.isdir(filename): + yield filename + elif recursive: + for d, _, files in os.walk(filename): + for base in files: + yield os.path.join(d, base) + + +if __name__ == '__main__': + multiprocessing.set_start_method('spawn') + + # Parse the args. + import argparse + ap = argparse.ArgumentParser(prog='grep') + + ap.add_argument('-r', '--recursive', + action='store_true') + ap.add_argument('-L', '--files-without-match', + dest='filesonly', + action='store_const', const='invert') + ap.add_argument('-l', '--files-with-matches', + dest='filesonly', + action='store_const', const='match') + ap.add_argument('-q', '--quiet', action='store_true') + ap.set_defaults(invert=False) + + reopts = ap.add_mutually_exclusive_group(required=True) + reopts.add_argument('-e', '--regexp', dest='regex', + metavar='REGEX') + reopts.add_argument('regex', nargs='?', + metavar='REGEX') + + ap.add_argument('files', nargs='+', metavar='FILE') + + opts = ap.parse_args() + ns = vars(opts) + + regex = ns.pop('regex') + filenames = ns.pop('files') + recursive = ns.pop('recursive') + if opts.filesonly: + if opts.filesonly == 'invert': + opts.invert = True + else: + assert opts.filesonly == 'match', opts + opts.invert = False + opts.filesonly = bool(opts.filesonly) + + def main(regex=regex, filenames=filenames): + # step 1 + regex = re.compile(regex) + # step 2 + filenames = resolve_filenames(filenames, recursive) + # step 3 + matches = search(filenames, regex, opts) + matches = iter(matches) + + # step 4 + + # Handle the first match. + for filename, line in matches: + if opts.quiet: + return 0 + elif opts.filesonly: + print(filename) + else: + for second in matches: + print(f'{filename}: {line}') + filename, line = second + print(f'{filename}: {line}') + break + else: + print(line) + break + else: + return 1 + + # Handle the remaining matches. + if opts.filesonly: + for filename, _ in matches: + print(filename) + else: + for filename, line in matches: + print(f'{filename}: {line}') + + return 0 + rc = main() + + # step 5 + sys.exit(rc) diff --git a/Doc/includes/concurrency/grep-multiprocessing.py b/Doc/includes/concurrency/grep-multiprocessing.py new file mode 100644 index 00000000000000..24f383b20547ce --- /dev/null +++ b/Doc/includes/concurrency/grep-multiprocessing.py @@ -0,0 +1,220 @@ +import os +import os.path +import re +import sys + +import multiprocessing +import queue +import threading + + +def search(filenames, regex, opts): + matches_by_file = queue.Queue() + + def do_background(): + MAX_FILES = 10 + MAX_MATCHES = 100 + + # Make sure we don't have too many procs at once, + # i.e. too many files open at once. + counter = threading.Semaphore(MAX_FILES) + finished = multiprocessing.Queue() + active = {} + done = False + + def monitor_tasks(): + while not done: + try: + index = finished.get(timeout=0.1) + except queue.Empty: + continue + proc = active.pop(index) + proc.join(0.1) + if proc.is_alive(): + # It's taking too long to terminate. + # We can wait for it at the end. + active[index] = proc + # Let a new process start. + counter.release() + monitor = threading.Thread(target=monitor_tasks) + monitor.start() + + for index, filename in enumerate(filenames): + # Prepare for the file. + matches = multiprocessing.Queue(MAX_MATCHES) + matches_by_file.put(matches) + + # Start a subprocess to process the file. + proc = multiprocessing.Process( + target=search_file, + args=(filename, matches, regex, opts, + index, finished), + ) + counter.acquire(blocking=True) + active[index] = proc + proc.start() + matches_by_file.put(None) + # Wait for all remaining tasks to finish. + done = True + monitor.join() + for proc in active.values(): + proc.join() + + background = threading.Thread(target=do_background) + background.start() + + # Yield the results as they are received, in order. + matches = matches_by_file.get() # blocking + while matches is not None: + match = matches.get() # blocking + while match is not None: + yield match + match = matches.get() # blocking + matches = matches_by_file.get() # blocking + + background.join() + + +def search_file(filename, matches, regex, opts, + index, finished): + lines = iter_lines(filename) + for match in search_lines(lines, regex, opts, filename): + matches.put(match) # blocking + matches.put(None) # blocking + # Let a new process start. + finished.put(index) + + +def iter_lines(filename): + if filename == '-': + yield from sys.stdin + else: + with open(filename) as infile: + yield from infile + + +def search_lines(lines, regex, opts, filename): + try: + if opts.filesonly: + if opts.invert: + for line in lines: + m = regex.search(line) + if m: + break + else: + yield (filename, None) + else: + for line in lines: + m = regex.search(line) + if m: + yield (filename, None) + break + else: + assert not opts.invert, opts + for line in lines: + m = regex.search(line) + if not m: + continue + if line.endswith(os.linesep): + line = line[:-len(os.linesep)] + yield (filename, line) + except UnicodeDecodeError: + # It must be a binary file. + return + + +def resolve_filenames(filenames, recursive=False): + for filename in filenames: + assert isinstance(filename, str), repr(filename) + if filename == '-': + yield '-' + elif not os.path.isdir(filename): + yield filename + elif recursive: + for d, _, files in os.walk(filename): + for base in files: + yield os.path.join(d, base) + + +if __name__ == '__main__': + multiprocessing.set_start_method('spawn') + + # Parse the args. + import argparse + ap = argparse.ArgumentParser(prog='grep') + + ap.add_argument('-r', '--recursive', + action='store_true') + ap.add_argument('-L', '--files-without-match', + dest='filesonly', + action='store_const', const='invert') + ap.add_argument('-l', '--files-with-matches', + dest='filesonly', + action='store_const', const='match') + ap.add_argument('-q', '--quiet', action='store_true') + ap.set_defaults(invert=False) + + reopts = ap.add_mutually_exclusive_group(required=True) + reopts.add_argument('-e', '--regexp', dest='regex', + metavar='REGEX') + reopts.add_argument('regex', nargs='?', + metavar='REGEX') + + ap.add_argument('files', nargs='+', metavar='FILE') + + opts = ap.parse_args() + ns = vars(opts) + + regex = ns.pop('regex') + filenames = ns.pop('files') + recursive = ns.pop('recursive') + if opts.filesonly: + if opts.filesonly == 'invert': + opts.invert = True + else: + assert opts.filesonly == 'match', opts + opts.invert = False + opts.filesonly = bool(opts.filesonly) + + def main(regex=regex, filenames=filenames): + # step 1 + regex = re.compile(regex) + # step 2 + filenames = resolve_filenames(filenames, recursive) + # step 3 + matches = search(filenames, regex, opts) + matches = iter(matches) + + # step 4 + + # Handle the first match. + for filename, line in matches: + if opts.quiet: + return 0 + elif opts.filesonly: + print(filename) + else: + for second in matches: + print(f'{filename}: {line}') + filename, line = second + print(f'{filename}: {line}') + break + else: + print(line) + break + else: + return 1 + + # Handle the remaining matches. + if opts.filesonly: + for filename, _ in matches: + print(filename) + else: + for filename, line in matches: + print(f'{filename}: {line}') + + return 0 + rc = main() + + # step 5 + sys.exit(rc) diff --git a/Doc/includes/concurrency/grep-parts.py b/Doc/includes/concurrency/grep-parts.py new file mode 100644 index 00000000000000..a3382609e84cbd --- /dev/null +++ b/Doc/includes/concurrency/grep-parts.py @@ -0,0 +1,553 @@ +import os +import os.path +import re +import sys + +import asyncio +from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor +import test.support.interpreters as interpreters +import test.support.interpreters.queues +import multiprocessing +import queue +import threading +import types + + +def iter_lines(filename): + if filename == '-': + yield from sys.stdin + else: + with open(filename) as infile: + yield from infile + + +async def aiter_lines(filename): + if filename == '-': + infile = sys.stdin + line = await read_line_async(infile) + while line: + yield line + line = await read_line_async(infile) + else: + # XXX Open using async? + with open(filename) as infile: + line = await read_line_async(infile) + while line: + yield line + line = await read_line_async(infile) + + +async def read_line_async(infile): + # XXX Do this async! + # maybe make use of asyncio.to_thread() or loop.run_in_executor()? + return infile.readline() + + +# [start-search-lines] +def search_lines(lines, regex, opts, filename): + try: + if opts.filesonly: + if opts.invert: + for line in lines: + m = regex.search(line) + if m: + break + else: + yield (filename, None) + else: + for line in lines: + m = regex.search(line) + if m: + yield (filename, None) + break + else: + assert not opts.invert, opts + for line in lines: + m = regex.search(line) + if not m: + continue + if line.endswith(os.linesep): + line = line[:-len(os.linesep)] + yield (filename, line) + except UnicodeDecodeError: + # It must be a binary file. + return +# [end-search-lines] + + +async def asearch_lines(lines, regex, opts, filename): + try: + if opts.filesonly: + if opts.invert: + async for line in lines: + m = regex.search(line) + if m: + break + else: + yield (filename, None) + else: + async for line in lines: + m = regex.search(line) + if m: + yield (filename, None) + break + else: + assert not opts.invert, opts + async for line in lines: + m = regex.search(line) + if not m: + continue + if line.endswith(os.linesep): + line = line[:-len(os.linesep)] + yield (filename, line) + except UnicodeDecodeError: + # It must be a binary file. + return + + +def resolve_filenames(filenames, recursive=False): + for filename in filenames: + assert isinstance(filename, str), repr(filename) + if filename == '-': + yield '-' + elif not os.path.isdir(filename): + yield filename + elif recursive: + for d, _, files in os.walk(filename): + for base in files: + yield os.path.join(d, base) + + +####################################### +# implemetations + +# [start-impl-sequential] +def search_sequential(filenames, regex, opts): + for filename in filenames: + lines = iter_lines(filename) + yield from search_lines(lines, regex, opts, filename) +# [end-impl-sequential] + + +# [start-impl-threads] +def search_using_threads(filenames, regex, opts): + matches_by_file = queue.Queue() + + def do_background(): + MAX_FILES = 10 + MAX_MATCHES = 100 + + # Make sure we don't have too many threads at once, + # i.e. too many files open at once. + counter = threading.Semaphore(MAX_FILES) + + def search_file(filename, matches): + lines = iter_lines(filename) + for match in search_lines(lines, regex, opts, filename): + matches.put(match) # blocking + matches.put(None) # blocking + # Let a new thread start. + counter.release() + + for filename in filenames: + # Prepare for the file. + matches = queue.Queue(MAX_MATCHES) + matches_by_file.put(matches) + + # Start a thread to process the file. + t = threading.Thread(target=search_file, args=(filename, matches)) + counter.acquire() + t.start() + matches_by_file.put(None) + + background = threading.Thread(target=do_background) + background.start() + + # Yield the results as they are received, in order. + matches = matches_by_file.get() # blocking + while matches is not None: + match = matches.get() # blocking + while match is not None: + yield match + match = matches.get() # blocking + matches = matches_by_file.get() # blocking + + background.join() +# [end-impl-threads] + + +# [start-impl-cf-threads] +def search_using_threads_cf(filenames, regex, opts): + matches_by_file = queue.Queue() + + def do_background(): + MAX_FILES = 10 + MAX_MATCHES = 100 + + def search_file(filename, matches): + lines = iter_lines(filename) + for match in search_lines(lines, regex, opts, filename): + matches.put(match) # blocking + matches.put(None) # blocking + + with ThreadPoolExecutor(MAX_FILES) as workers: + for filename in filenames: + # Prepare for the file. + matches = queue.Queue(MAX_MATCHES) + matches_by_file.put(matches) + + # Start a thread to process the file. + workers.submit(search_file, filename, matches) + matches_by_file.put(None) + + background = threading.Thread(target=do_background) + background.start() + + # Yield the results as they are received, in order. + matches = matches_by_file.get() # blocking + while matches is not None: + match = matches.get() # blocking + while match is not None: + yield match + match = matches.get() # blocking + matches = matches_by_file.get() # blocking + + background.join() +# [end-impl-cf-threads] + + +def search_using_interpreters(filenames, regex, opts): + matches_by_file = queue.Queue() + + def do_background(): + MAX_FILES = 10 + MAX_MATCHES = 100 + + def new_interpreter(): + interp = interpreters.create() + interp.exec(f"""if True: + with open({__file__!r}) as infile: + text = infile.read() + ns = dict() + exec(text, ns, ns) + prep_interpreter = ns['prep_interpreter'] + del ns, text + + search_file = prep_interpreter( + {regex.pattern!r}, + {regex.flags}, + {tuple(vars(opts).items())}, + ) + """) + return interp + + ready_workers = queue.Queue(MAX_FILES) + workers = [] + + def next_worker(): + if len(workers) < MAX_FILES: + interp = new_interpreter() + workers.append(interp) + ready_workers.put(interp) + return ready_workers.get() # blocking + + def do_work(filename, matches, interp): + #interp.call(search_file, (regex, opts, filename, matches)) + interp.prepare_main(matches=matches) + interp.exec(f'search_file({filename!r}, matches)') + # Let a new thread start. + ready_workers.put(interp) + + for filename in filenames: + # Prepare for the file. + matches = interpreters.queues.create(MAX_MATCHES) + matches_by_file.put(matches) + interp = next_worker() + + # Start a thread to process the file. + t = threading.Thread( + target=do_work, + args=(filename, matches, interp), + ) + t.start() + matches_by_file.put(None) + + background = threading.Thread(target=do_background) + background.start() + + # Yield the results as they are received, in order. + matches = matches_by_file.get() # blocking + while matches is not None: + match = matches.get() # blocking + while match is not None: + yield match + match = matches.get() # blocking + matches = matches_by_file.get() # blocking + + background.join() + + +def prep_interpreter(regex_pat, regex_flags, opts): + regex = re.compile(regex_pat, regex_flags) + opts = types.SimpleNamespace(**dict(opts)) + + def search_file(filename, matches): + lines = iter_lines(filename) + for match in search_lines(lines, regex, opts, filename): + matches.put(match) # blocking + matches.put(None) # blocking + return search_file + + +async def search_using_asyncio(filenames, regex, opts): + matches_by_file = asyncio.Queue() + + async def do_background(): + MAX_FILES = 10 + MAX_MATCHES = 100 + + # Make sure we don't have too many coroutines at once, + # i.e. too many files open at once. + counter = asyncio.Semaphore(MAX_FILES) + + async def search_file(filename, matches): + # aiter_lines() opens the file too. + lines = aiter_lines(filename) + async for match in asearch_lines(lines, regex, opts, filename): + await matches.put(match) + await matches.put(None) + # Let a new coroutine start. + counter.release() + + async with asyncio.TaskGroup() as tg: + for filename in filenames: + # Prepare for the file. + matches = asyncio.Queue(MAX_MATCHES) + await matches_by_file.put(matches) + + # Start a coroutine to process the file. + tg.create_task( + search_file(filename, matches), + ) + await counter.acquire() + await matches_by_file.put(None) + + background = asyncio.create_task(do_background()) + + # Yield the results as they are received, in order. + matches = await matches_by_file.get() # blocking + while matches is not None: + match = await matches.get() # blocking + while match is not None: + yield match + match = await matches.get() # blocking + matches = await matches_by_file.get() # blocking + + await asyncio.wait([background]) + + +def search_using_multiprocessing(filenames, regex, opts): + matches_by_file = queue.Queue() + + def do_background(): + MAX_FILES = 10 + MAX_MATCHES = 100 + + # Make sure we don't have too many processes at once, + # i.e. too many files open at once. + counter = threading.Semaphore(MAX_FILES) + finished = multiprocessing.Queue() + active = {} + done = False + + def monitor_tasks(): + while not done: + try: + index = finished.get(timeout=0.1) + except queue.Empty: + continue + proc = active.pop(index) + proc.join(0.1) + if proc.is_alive(): + # It's taking too long to terminate. + # We can wait for it at the end. + active[index] = proc + # Let a new process start. + counter.release() + monitor = threading.Thread(target=monitor_tasks) + monitor.start() + + for index, filename in enumerate(filenames): + # Prepare for the file. + matches = multiprocessing.Queue(MAX_MATCHES) + matches_by_file.put(matches) + + # Start a subprocess to process the file. + proc = multiprocessing.Process( + target=search_file, + args=(filename, matches, regex, opts, index, finished), + ) + counter.acquire(blocking=True) + active[index] = proc + proc.start() + matches_by_file.put(None) + # Wait for all remaining tasks to finish. + done = True + monitor.join() + for proc in active.values(): + proc.join() + + background = threading.Thread(target=do_background) + background.start() + + # Yield the results as they are received, in order. + matches = matches_by_file.get() # blocking + while matches is not None: + match = matches.get() # blocking + while match is not None: + yield match + match = matches.get() # blocking + matches = matches_by_file.get() # blocking + + background.join() + + +def search_file(filename, matches, regex, opts, index, finished): + lines = iter_lines(filename) + for match in search_lines(lines, regex, opts, filename): + matches.put(match) # blocking + matches.put(None) # blocking + # Let a new process start. + finished.put(index) + + +IMPLS = { + 'sequential': search_sequential, + 'threads': search_using_threads, + 'threads-cf': search_using_threads_cf, + 'interpreters': search_using_interpreters, + 'asyncio': search_using_asyncio, + 'multiprocessing': search_using_multiprocessing, +} + + +####################################### +# the script + +if __name__ == '__main__': + multiprocessing.set_start_method('spawn') + + # Parse the args. + import argparse + parser = argparse.ArgumentParser(prog='grep') + + parser.add_argument('--impl', choices=sorted(IMPLS), default='threads') + + parser.add_argument('-r', '--recursive', action='store_true') + parser.add_argument('-L', '--files-without-match', dest='filesonly', + action='store_const', const='invert') + parser.add_argument('-l', '--files-with-matches', dest='filesonly', + action='store_const', const='match') + parser.add_argument('-q', '--quiet', action='store_true') + parser.set_defaults(invert=False) + + regexopts = parser.add_mutually_exclusive_group(required=True) + regexopts.add_argument('-e', '--regexp', dest='regex', metavar='REGEX') + regexopts.add_argument('regex', nargs='?', metavar='REGEX') + + parser.add_argument('files', nargs='+', metavar='FILE') + + opts = parser.parse_args() + ns = vars(opts) + + regex = ns.pop('regex') + filenames = ns.pop('files') + recursive = ns.pop('recursive') + if opts.filesonly: + if opts.filesonly == 'invert': + opts.invert = True + else: + assert opts.filesonly == 'match', opts + opts.invert = False + opts.filesonly = bool(opts.filesonly) + + search = IMPLS[ns.pop('impl')] + +# [start-high-level] + def main(regex=regex, filenames=filenames): + # step 1 + regex = re.compile(regex) + # step 2 + filenames = resolve_filenames(filenames, recursive) + # step 3 + matches = search(filenames, regex, opts) + + # step 4 + + if hasattr(type(matches), '__aiter__'): + async def iter_and_show(matches=matches): + matches = type(matches).__aiter__(matches) + + # Handle the first match. + async for filename, line in matches: + if opts.quiet: + return 0 + elif opts.filesonly: + print(filename) + else: + async for second in matches: + print(f'{filename}: {line}') + filename, line = second + print(f'{filename}: {line}') + break + else: + print(line) + break + else: + return 1 + + # Handle the remaining matches. + if opts.filesonly: + async for filename, _ in matches: + print(filename) + else: + async for filename, line in matches: + print(f'{filename}: {line}') + + return 0 + return asyncio.run(search_and_show()) + else: + matches = iter(matches) + + # Handle the first match. + for filename, line in matches: + if opts.quiet: + return 0 + elif opts.filesonly: + print(filename) + else: + for second in matches: + print(f'{filename}: {line}') + filename, line = second + print(f'{filename}: {line}') + break + else: + print(line) + break + else: + return 1 + + # Handle the remaining matches. + if opts.filesonly: + for filename, _ in matches: + print(filename) + else: + for filename, line in matches: + print(f'{filename}: {line}') + + return 0 + rc = main() + + # step 5 + sys.exit(rc) +# [end-high-level] diff --git a/Doc/includes/concurrency/grep-sequential.py b/Doc/includes/concurrency/grep-sequential.py new file mode 100644 index 00000000000000..31ef2a11bee41b --- /dev/null +++ b/Doc/includes/concurrency/grep-sequential.py @@ -0,0 +1,145 @@ +import os +import os.path +import re +import sys + + +def search(filenames, regex, opts): + for filename in filenames: + # iter_lines() opens the file too. + lines = iter_lines(filename) + yield from search_lines( + lines, regex, opts, filename) + + +def iter_lines(filename): + if filename == '-': + yield from sys.stdin + else: + with open(filename) as infile: + yield from infile + + +def search_lines(lines, regex, opts, filename): + try: + if opts.filesonly: + if opts.invert: + for line in lines: + m = regex.search(line) + if m: + break + else: + yield (filename, None) + else: + for line in lines: + m = regex.search(line) + if m: + yield (filename, None) + break + else: + assert not opts.invert, opts + for line in lines: + m = regex.search(line) + if not m: + continue + if line.endswith(os.linesep): + line = line[:-len(os.linesep)] + yield (filename, line) + except UnicodeDecodeError: + # It must be a binary file. + return + + +def resolve_filenames(filenames, recursive=False): + for filename in filenames: + assert isinstance(filename, str), repr(filename) + if filename == '-': + yield '-' + elif not os.path.isdir(filename): + yield filename + elif recursive: + for d, _, files in os.walk(filename): + for base in files: + yield os.path.join(d, base) + + +if __name__ == '__main__': + # Parse the args. + import argparse + ap = argparse.ArgumentParser(prog='grep') + + ap.add_argument('-r', '--recursive', + action='store_true') + ap.add_argument('-L', '--files-without-match', + dest='filesonly', + action='store_const', const='invert') + ap.add_argument('-l', '--files-with-matches', + dest='filesonly', + action='store_const', const='match') + ap.add_argument('-q', '--quiet', action='store_true') + ap.set_defaults(invert=False) + + reopts = ap.add_mutually_exclusive_group(required=True) + reopts.add_argument('-e', '--regexp', dest='regex', + metavar='REGEX') + reopts.add_argument('regex', nargs='?', + metavar='REGEX') + + ap.add_argument('files', nargs='+', metavar='FILE') + + opts = ap.parse_args() + ns = vars(opts) + + regex = ns.pop('regex') + filenames = ns.pop('files') + recursive = ns.pop('recursive') + if opts.filesonly: + if opts.filesonly == 'invert': + opts.invert = True + else: + assert opts.filesonly == 'match', opts + opts.invert = False + opts.filesonly = bool(opts.filesonly) + + def main(regex=regex, filenames=filenames): + # step 1 + regex = re.compile(regex) + # step 2 + filenames = resolve_filenames(filenames, recursive) + # step 3 + matches = search(filenames, regex, opts) + matches = iter(matches) + + # step 4 + + # Handle the first match. + for filename, line in matches: + if opts.quiet: + return 0 + elif opts.filesonly: + print(filename) + else: + for second in matches: + print(f'{filename}: {line}') + filename, line = second + print(f'{filename}: {line}') + break + else: + print(line) + break + else: + return 1 + + # Handle the remaining matches. + if opts.filesonly: + for filename, _ in matches: + print(filename) + else: + for filename, line in matches: + print(f'{filename}: {line}') + + return 0 + rc = main() + + # step 5 + sys.exit(rc) diff --git a/Doc/includes/concurrency/grep-threads-cf.py b/Doc/includes/concurrency/grep-threads-cf.py new file mode 100644 index 00000000000000..682d9004a9c534 --- /dev/null +++ b/Doc/includes/concurrency/grep-threads-cf.py @@ -0,0 +1,181 @@ +import os +import os.path +import re +import sys + +from concurrent.futures import ThreadPoolExector +import queue +import threading + + +def search(filenames, regex, opts): + matches_by_file = queue.Queue() + + def do_background(): + MAX_FILES = 10 + MAX_MATCHES = 100 + + def search_file(filename, matches): + lines = iter_lines(filename) + for match in search_lines( + lines, regex, opts, filename): + matches.put(match) # blocking + matches.put(None) # blocking + + with ThreadPoolExecutor(MAX_FILES) as workers: + for filename in filenames: + # Prepare for the file. + matches = queue.Queue(MAX_MATCHES) + matches_by_file.put(matches) + + # Start a thread to process the file. + workers.submit( + search_file, filename, matches) + matches_by_file.put(None) + + background = threading.Thread(target=do_background) + background.start() + + # Yield the results as they are received, in order. + matches = matches_by_file.get() # blocking + while matches is not None: + match = matches.get() # blocking + while match is not None: + yield match + match = matches.get() # blocking + matches = matches_by_file.get() # blocking + + background.join() + + +def iter_lines(filename): + if filename == '-': + yield from sys.stdin + else: + with open(filename) as infile: + yield from infile + + +def search_lines(lines, regex, opts, filename): + try: + if opts.filesonly: + if opts.invert: + for line in lines: + m = regex.search(line) + if m: + break + else: + yield (filename, None) + else: + for line in lines: + m = regex.search(line) + if m: + yield (filename, None) + break + else: + assert not opts.invert, opts + for line in lines: + m = regex.search(line) + if not m: + continue + if line.endswith(os.linesep): + line = line[:-len(os.linesep)] + yield (filename, line) + except UnicodeDecodeError: + # It must be a binary file. + return + + +def resolve_filenames(filenames, recursive=False): + for filename in filenames: + assert isinstance(filename, str), repr(filename) + if filename == '-': + yield '-' + elif not os.path.isdir(filename): + yield filename + elif recursive: + for d, _, files in os.walk(filename): + for base in files: + yield os.path.join(d, base) + + +if __name__ == '__main__': + # Parse the args. + import argparse + ap = argparse.ArgumentParser(prog='grep') + + ap.add_argument('-r', '--recursive', + action='store_true') + ap.add_argument('-L', '--files-without-match', + dest='filesonly', + action='store_const', const='invert') + ap.add_argument('-l', '--files-with-matches', + dest='filesonly', + action='store_const', const='match') + ap.add_argument('-q', '--quiet', action='store_true') + ap.set_defaults(invert=False) + + reopts = ap.add_mutually_exclusive_group(required=True) + reopts.add_argument('-e', '--regexp', dest='regex', + metavar='REGEX') + reopts.add_argument('regex', nargs='?', + metavar='REGEX') + + ap.add_argument('files', nargs='+', metavar='FILE') + + opts = ap.parse_args() + ns = vars(opts) + + regex = ns.pop('regex') + filenames = ns.pop('files') + recursive = ns.pop('recursive') + if opts.filesonly: + if opts.filesonly == 'invert': + opts.invert = True + else: + assert opts.filesonly == 'match', opts + opts.invert = False + opts.filesonly = bool(opts.filesonly) + + def main(regex=regex, filenames=filenames): + # step 1 + regex = re.compile(regex) + # step 2 + filenames = resolve_filenames(filenames, recursive) + # step 3 + matches = search(filenames, regex, opts) + matches = iter(matches) + + # step 4 + + # Handle the first match. + for filename, line in matches: + if opts.quiet: + return 0 + elif opts.filesonly: + print(filename) + else: + for second in matches: + print(f'{filename}: {line}') + filename, line = second + print(f'{filename}: {line}') + break + else: + print(line) + break + else: + return 1 + + # Handle the remaining matches. + if opts.filesonly: + for filename, _ in matches: + print(filename) + else: + for filename, line in matches: + print(f'{filename}: {line}') + + return 0 + rc = main() + + # step 5 + sys.exit(rc) diff --git a/Doc/includes/concurrency/grep-threads.py b/Doc/includes/concurrency/grep-threads.py new file mode 100644 index 00000000000000..0a6d793962722a --- /dev/null +++ b/Doc/includes/concurrency/grep-threads.py @@ -0,0 +1,187 @@ +import os +import os.path +import re +import sys + +import queue +import threading + + +def search(filenames, regex, opts): + matches_by_file = queue.Queue() + + def do_background(): + MAX_FILES = 10 + MAX_MATCHES = 100 + + # Make sure we don't have too many threads at once, + # i.e. too many files open at once. + counter = threading.Semaphore(MAX_FILES) + + def search_file(filename, matches): + lines = iter_lines(filename) + for match in search_lines( + lines, regex, opts, filename): + matches.put(match) # blocking + matches.put(None) # blocking + # Let a new thread start. + counter.release() + + for filename in filenames: + # Prepare for the file. + matches = queue.Queue(MAX_MATCHES) + matches_by_file.put(matches) + + # Start a thread to process the file. + t = threading.Thread(target=search_file, + args=(filename, matches)) + counter.acquire() + t.start() + matches_by_file.put(None) + + background = threading.Thread(target=do_background) + background.start() + + # Yield the results as they are received, in order. + matches = matches_by_file.get() # blocking + while matches is not None: + match = matches.get() # blocking + while match is not None: + yield match + match = matches.get() # blocking + matches = matches_by_file.get() # blocking + + background.join() + + +def iter_lines(filename): + if filename == '-': + yield from sys.stdin + else: + with open(filename) as infile: + yield from infile + + +def search_lines(lines, regex, opts, filename): + try: + if opts.filesonly: + if opts.invert: + for line in lines: + m = regex.search(line) + if m: + break + else: + yield (filename, None) + else: + for line in lines: + m = regex.search(line) + if m: + yield (filename, None) + break + else: + assert not opts.invert, opts + for line in lines: + m = regex.search(line) + if not m: + continue + if line.endswith(os.linesep): + line = line[:-len(os.linesep)] + yield (filename, line) + except UnicodeDecodeError: + # It must be a binary file. + return + + +def resolve_filenames(filenames, recursive=False): + for filename in filenames: + assert isinstance(filename, str), repr(filename) + if filename == '-': + yield '-' + elif not os.path.isdir(filename): + yield filename + elif recursive: + for d, _, files in os.walk(filename): + for base in files: + yield os.path.join(d, base) + + +if __name__ == '__main__': + # Parse the args. + import argparse + ap = argparse.ArgumentParser(prog='grep') + + ap.add_argument('-r', '--recursive', + action='store_true') + ap.add_argument('-L', '--files-without-match', + dest='filesonly', + action='store_const', const='invert') + ap.add_argument('-l', '--files-with-matches', + dest='filesonly', + action='store_const', const='match') + ap.add_argument('-q', '--quiet', action='store_true') + ap.set_defaults(invert=False) + + reopts = ap.add_mutually_exclusive_group(required=True) + reopts.add_argument('-e', '--regexp', dest='regex', + metavar='REGEX') + reopts.add_argument('regex', nargs='?', + metavar='REGEX') + + ap.add_argument('files', nargs='+', metavar='FILE') + + opts = ap.parse_args() + ns = vars(opts) + + regex = ns.pop('regex') + filenames = ns.pop('files') + recursive = ns.pop('recursive') + if opts.filesonly: + if opts.filesonly == 'invert': + opts.invert = True + else: + assert opts.filesonly == 'match', opts + opts.invert = False + opts.filesonly = bool(opts.filesonly) + + def main(regex=regex, filenames=filenames): + # step 1 + regex = re.compile(regex) + # step 2 + filenames = resolve_filenames(filenames, recursive) + # step 3 + matches = search(filenames, regex, opts) + matches = iter(matches) + + # step 4 + + # Handle the first match. + for filename, line in matches: + if opts.quiet: + return 0 + elif opts.filesonly: + print(filename) + else: + for second in matches: + print(f'{filename}: {line}') + filename, line = second + print(f'{filename}: {line}') + break + else: + print(line) + break + else: + return 1 + + # Handle the remaining matches. + if opts.filesonly: + for filename, _ in matches: + print(filename) + else: + for filename, line in matches: + print(f'{filename}: {line}') + + return 0 + rc = main() + + # step 5 + sys.exit(rc) diff --git a/Doc/includes/concurrency/run-examples.py b/Doc/includes/concurrency/run-examples.py new file mode 100644 index 00000000000000..85d754c3b5d5ff --- /dev/null +++ b/Doc/includes/concurrency/run-examples.py @@ -0,0 +1,698 @@ +"""Example code for howto/concurrency.rst. + +The examples take advantage of the literalinclude directive's +:start-after: and :end-before: options. +""" + +from collections import namedtuple +import os.path +import re +import shlex +import subprocess +import sys + + +IMPLS_DIR = os.path.dirname(__file__) +#if IMPLS_DIR not in sys.path: +# sys.path.insert(0, IMPLS_DIR) + + +class example(staticmethod): + """A function containing example code. + + The function will be called when this file is run as a script. + """ + + registry = [] + + def __init__(self, func): + super().__init__(func) + self.func = func + + def __set_name__(self, cls, name): + assert name == self.func.__name__, (name, self.func.__name__) + type(self).registry.append((self.func, cls)) + + +class Examples: + """Code examples for docs using "literalinclude".""" + + +class WorkloadExamples(Examples): + """Examples of a single concurrency workload.""" + + +####################################### +# workload: grep +####################################### + +class GrepExamples(WorkloadExamples): + + @staticmethod + def app(name, kind='basic', cf=False): + import shlex + from grep import Options, resolve_impl, grep + from grep.__main__ import render_matches + opts = Options( + #recursive=True, + #ignorecase = True, + #invertmatch = True, + #showfilename = True, + #showfilename = False, + #filesonly = 'invert', + #filesonly = 'match', + #showonlymatch = True, + #quiet = True, + #hideerrors = True, + ) + #opts = Options(recursive=True, filesonly='match') + impl = resolve_impl(name, kind, cf) + pat = 'help' + #filenames = '.' + filenames = ['make.bat', 'Makefile'] + print(f'# grep {opts} {shlex.quote(pat)} {shlex.join(filenames)}') + print() + matches = grep(pat, opts, *filenames, impl=impl) + if name == 'asyncio': + assert hasattr(type(matches), '__aiter__'), (name,) + async def search(): + async for line in render_matches(matches, opts): + print(line) + import asyncio + asyncio.run(search()) + else: + assert not hasattr(type(matches), '__aiter__'), (name,) + for line in render_matches(matches, opts): + print(line) + + @example + def run_sequentially(): + GrepExamples.app('sequential') + + @example + def run_using_threads(): + GrepExamples.app('threads') + + @example + def run_using_cf_threads(): + GrepExamples.app('threads', cf=True) + + @example + def run_using_subinterpreters(): + GrepExamples.app('interpreters') + + @example + def run_using_cf_subinterpreters(): + GrepExamples.app('interpreters', cf=True) + + @example + def run_using_async(): + GrepExamples.app('asyncio') + + @example + def run_using_multiprocessing(): + GrepExamples.app('multiprocessing') + + @example + def run_using_cf_multiprocessing(): + GrepExamples.app('multiprocessing', cf=True) + + +####################################### +# workload: image resizer +####################################### + +class ImageResizer(WorkloadExamples): + + @example + def run_sequentially(): + # [start-image-resizer-sequential] + # sequential 2 + ... + # [end-image-resizer-sequential] + + @example + def run_using_threads(): + # [start-image-resizer-threads] + import threading + + def task(): + ... + + t = threading.Thread(target=task) + t.start() + + ... + # [end-image-resizer-threads] + + @example + def run_using_cf_thread(): + # [start-image-resizer-cf-thread] + # concurrent.futures 2 + ... + # [end-image-resizer-cf-thread] + + @example + def run_using_subinterpreters(): + # [start-image-resizer-subinterpreters] + # subinterpreters 2 + ... + # [end-image-resizer-subinterpreters] + + @example + def run_using_cf_subinterpreters(): + # [start-image-resizer-cf-subinterpreters] + # concurrent.futures 2 + ... + # [end-image-resizer-cf-subinterpreters] + + @example + def run_using_async(): + # [start-image-resizer-async] + # async 2 + ... + # [end-image-resizer-async] + + @example + def run_using_multiprocessing(): + # [start-image-resizer-multiprocessing] + import multiprocessing + + def task(): + ... + + ... + # [end-image-resizer-multiprocessing] + + @example + def run_using_cf_multiprocessing(): + # [start-image-resizer-cf-multiprocessing] + # concurrent.futures 2 + ... + # [end-image-resizer-cf-multiprocessing] + + +####################################### +# workload 2: ... +####################################### + +class Workload2(WorkloadExamples): + + @example + def run_sequentially(): + # [start-w2-sequential] + # sequential 3 + ... + # [end-w2-sequential] + + @example + def run_using_threads(): + # [start-w2-threads] + import threading + + def task(): + ... + + t = threading.Thread(target=task) + t.start() + + ... + # [end-w2-threads] + + @example + def run_using_cf_thread(): + # [start-w2-cf-thread] + # concurrent.futures 3 + ... + # [end-w2-cf-thread] + + @example + def run_using_subinterpreters(): + # [start-w2-subinterpreters] + # subinterpreters 3 + ... + # [end-w2-subinterpreters] + + @example + def run_using_cf_subinterpreters(): + # [start-w2-cf-subinterpreters] + # concurrent.futures 3 + ... + # [end-w2-cf-subinterpreters] + + @example + def run_using_async(): + # [start-w2-async] + # async 3 + ... + # [end-w2-async] + + @example + def run_using_multiprocessing(): + # [start-w2-multiprocessing] + import multiprocessing + + def task(): + ... + + ... + # [end-w2-multiprocessing] + + @example + def run_using_cf_multiprocessing(): + # [start-w2-cf-multiprocessing] + # concurrent.futures 3 + ... + # [end-w2-cf-multiprocessing] + + +# [start-w2-cf] +# concurrent.futures 2 +... +# [end-w2-cf] + + +####################################### +# workload 3: ... +####################################### + +class Workload3(WorkloadExamples): + + @example + def run_sequentially(): + # [start-w3-sequential] + # sequential 3 + ... + # [end-w3-sequential] + + @example + def run_using_threads(): + # [start-w3-threads] + import threading + + def task(): + ... + + t = threading.Thread(target=task) + t.start() + + ... + # [end-w3-threads] + + @example + def run_using_cf_thread(): + # [start-w3-cf-thread] + # concurrent.futures 3 + ... + # [end-w3-cf-thread] + + @example + def run_using_subinterpreters(): + # [start-w3-subinterpreters] + # subinterpreters 3 + ... + # [end-w3-subinterpreters] + + @example + def run_using_cf_subinterpreters(): + # [start-w3-cf-subinterpreters] + # concurrent.futures 3 + ... + # [end-w3-cf-subinterpreters] + + @example + def run_using_async(): + # [start-w3-async] + # async 3 + ... + # [end-w3-async] + + @example + def run_using_multiprocessing(): + # [start-w3-multiprocessing] + import multiprocessing + + def task(): + ... + + ... + # [end-w3-multiprocessing] + + @example + def run_using_cf_multiprocessing(): + # [start-w3-cf-multiprocessing] + # concurrent.futures 3 + ... + # [end-w3-cf-multiprocessing] + + +# [start-w3-cf] +# concurrent.futures 3 +... +# [end-w3-cf] + + +ALL = object() + + +def registry_resolve(registry, name, kind=None): + orig = name + if name is None: + name = '' + while True: + try: + return name, registry[name] + except KeyError: + # Try aliases. + try: + name = registry[f'|{name}|'] + except KeyError: + raise ValueError(f'unsupported {kind or "???"} registry key {name!r}') + + +def registry_resolve_all(registry, name=ALL, kind=None): + if name is ALL or name is registry: + for pair in registry.items(): + name, _ = pair + if name.startswith('|'): + continue + yield pair + else: + yield registry_resolve(registry, name, kind) + + +def registry_render(registry, kind=None, indent='', vfmt=None, parent=None): + kind = f'{kind} - ' if kind else '' + entries = {} + aliases_by_target = {} + default = None + for name, value in registry.items(): + if name == '||': + default, _ = registry_resolve(registry, value) + elif name.startswith('|'): + target, _ = registry_resolve(registry, value) + try: + aliases = aliases_by_target[target] + except KeyError: + aliases = aliases_by_target[target] = [] + aliases.append(name.strip('|')) + else: + entries[name] = value + for name, spec in entries.items(): + label = f'{parent}.{name}' if parent else name + line = f'{indent}* {kind}{label}' + if vfmt: + specstr = vfmt(spec, name) if callable(vfmt) else vfmt.format(spec) + line = f'{line:30} => {specstr}' + if name == default: + line = f'{line:60} (default)' + aliases = aliases_by_target.get(name) + if aliases: + line = f'{line:70} (aliases: {", ".join(aliases)})' + yield line, name, spec + + +EXAMPLES = { + 'grep': { + 'IMPLS': { + '||': 'sequential', + 'sequential': { + '||': 'basic', + 'basic': '<>', + 'parts': ['grep-parts.py', '--impl', 'sequential'], + }, + 'threads': { + '||': 'basic', + 'basic': '<>', + 'cf': '<>', + 'parts': ['grep-parts.py', '--impl', 'threads'], + 'parts-cf': ['grep-parts.py', '--impl', 'threads-cf'], + }, + 'interpreters': { + '||': 'basic', + 'basic': '<>', + #'cf': '<>', + 'parts': ['grep-parts.py', '--impl', 'interpreters'], + }, + '|async|': 'asyncio', + 'asyncio': { + '||': 'basic', + 'basic': '<>', + 'parts': ['grep-parts.py', '--impl', 'asyncio'], + }, + 'multiprocessing': { + '||': 'basic', + 'basic': '<>', + #'cf': '<>', + 'parts': ['grep-parts.py', '--impl', 'multiprocessing'], + }, + }, + 'OPTS': { + '||': 'min', + '|min|': 'min.all', + 'min.all': "-r '.. index::' .", + 'min.mixed': 'help Makefile make.bat requirements.txt', + }, + }, +} + + +class Selection(namedtuple('Selection', 'app model impl')): + + _opts = None + _cmd = None + _executable = None + _executable_argv = None + _subargv = None + _subargvstr = None + _defaultimpl = None + + REGEX = re.compile(rf""" + ^ + ( \* | \w+ ) # + (?: + \. + ( \* | \w+ ) # + (?: + \. + ( \* | \w+ ) # + )? + )? + (?: + : + ( \S+ (?: \s+ \S+ )? ) # + )? + \s* + $ + """, re.VERBOSE) + + @classmethod + def parse(cls, text): + m = cls.REGEX.match(text) + if not m: + raise ValueError(f'unsupported selection {text!r}') + app, model, impl, opts = m.groups() + + if app == '*': + app = ALL + if model == '*': + model = ALL + if impl == '*': + impl = ALL + if opts == '*': + opts = ALL + self = cls(app, model, impl, opts) + return self + + @classmethod + def _from_resolved(cls, app, model, impl, opts, cmd, argv, + defaultimpl=False, + ): + self = cls.__new__(cls, app, model, impl, opts) + executable_argv = None + if cmd == '<>': + cmd = f'{app}-{model}' if defaultimpl else f'{app}-{model}-{impl}' + executable = cmd + '.py' + elif isinstance(cmd, str): + executable = cmd + else: + cmd = tuple(cmd) + executable, *executable_argv = cmd + if os.path.basename(executable) == executable: + executable = os.path.join(IMPLS_DIR, executable) + if not os.path.exists(executable): + executable = os.path.basename(executable) + self._cmd = cmd + self._executable = executable + self._executable_argv = executable_argv + if isinstance(argv, str): + self._subargvstr = argv + else: + self._subargv = tuple(argv) + self._defaultimpl = defaultimpl + return self + + def __new__(cls, app, model, impl, opts=None): + self = super().__new__(cls, app, model or None, impl or None) + self._opts = opts or None + return self + + @property + def opts(self): + return self._opts + + @property + def cmd(self): + for bad in (ALL, None): + if bad in self or self._opts is bad: + raise RuntimeError('not resolved') + return self._cmd + + @property + def executable(self): + for bad in (ALL, None): + if bad in self or self._opts is bad: + raise RuntimeError('not resolved') + return self._executable + + @property + def executable_argv(self): + if self._executable_argv is None: + return () + return self._executable_argv + + @property + def subargv(self): + for bad in (ALL, None): + if bad in self or self._opts is bad: + raise RuntimeError('not resolved') + if self._subargv is None: + self._subargv = tuple(shlex.split(self._subargvstr)) + return self._subargv + + @property + def argv(self): + for bad in (ALL, None): + if bad in self or self._opts is bad: + raise RuntimeError('not resolved') + argv = [self._executable] + if self._executable.endswith('.py'): + argv.insert(0, sys.executable) + if self._executable_argv: + argv.extend(self._executable_argv) + if self._subargv is None: + self._subargv = shlex.split(self._subargvstr) + argv.extend(self._subargv) + return argv + + def resolve(self, argv=None): + cls = type(self) + if self._cmd and self._cmd is not ALL: + assert self._argv is not ALL + if argv is None: + resolved = self + else: + default = self._defaultimpl + resolved = cls._from_resolved( + *self, self._opts, self._cmd, argv, default) + yield resolved + else: + assert self.app is not None + requested = argv + resolve = registry_resolve_all + for app, app_spec in resolve(EXAMPLES, self.app, 'app'): + models = app_spec['IMPLS'] + all_opts = app_spec['OPTS'] + for model, impls in resolve(models, self.model, 'model'): + default, _ = registry_resolve(impls, '') + for impl, cmd in resolve(impls, self.impl, 'impl'): + if requested: + argvs = [('???', requested)] + else: + argvs = resolve(all_opts, self._opts, 'opts') + isdefault = (impl == default) + for opts, argv in argvs: + yield cls._from_resolved( + app, model, impl, opts, cmd, argv, isdefault) + + +####################################### +# A script to run the examples +####################################### + +USAGE = 'run-examples [-h|--help] [SELECTION ..] [-- ..]' +HELP = """ +SELECTION is one of the following: + + APP + APP.MODEL + APP.MODEL.IMPL + APP:OPTS + APP.MODEL:OPTS + APP.MODEL.IMPL:OPTS + +where each field is a known name or the wildcard (*). +"""[:-1] + +if __name__ == '__main__': + argv = sys.argv[1:] + remainder = None + if '--' in argv: + index = argv.index('--') + remainder = argv[index+1:] + argv = argv[:index] + if '-h' in argv or '--help' in argv: + print(USAGE) + print(HELP) + print() + print('Supported selection field values:') + print() + indent = ' ' * 2 + def render(reg, kind, depth=0, vfmt=None, parent=None): + for l, _, s in registry_render( + reg, kind, indent*depth, vfmt, parent): + yield l, s + for line, app, app_spec in registry_render(EXAMPLES, 'app'): + print(line) + app_impls = app_spec['IMPLS'] + print(f'{indent}* concurrency models:') + for line, _ in render(app_impls, None, 2): + print(line.format('')) + print(f'{indent}* implementations:') + for model, impls in registry_resolve_all(app_impls, kind='model'): + default, _ = registry_resolve(impls, '') + def vfmt_impl(cmd, impl, app=app, model=model): + opts, argv = '???', () + sel = Selection._from_resolved( + app, model, impl, opts, cmd, argv, impl==default) + if sel.cmd != sel.executable: + return os.path.basename(sel.executable) + else: + return sel.executable + for line, cmd in render(impls, None, 2, vfmt_impl, model): + print(line) + print(f'{indent}* argv options:') + for line, _ in render(app_spec['OPTS'], None, 2, f'{app} {{}}'): + print(line) + sys.exit(0) + + if not argv: + argv = ['*.*.*:*'] + + div = '#' * 40 + for text in argv: + sel = Selection.parse(text) + for sel in sel.resolve(remainder or None): + print() + print(div) + print('# app: ', sel.app) + print('# model: ', sel.model) + print('# impl: ', sel.impl) + print('# opts: ', sel.opts) + print('# script: ', sel.executable, *sel.executable_argv) + print('# argv: ', sel.app, shlex.join(sel.subargv)) + print(div) + print() + proc = subprocess.run(sel.argv) + print() + print('# returncode:', proc.returncode) diff --git a/Doc/library/asyncio.rst b/Doc/library/asyncio.rst index 7d368dae49dc1d..f84e0cbc5549b2 100644 --- a/Doc/library/asyncio.rst +++ b/Doc/library/asyncio.rst @@ -60,6 +60,12 @@ Additionally, there are **low-level** APIs for .. _asyncio-cli: +.. seealso:: + + The :ref:`concurrency-howto` offers explanations about concurrency + and different concurrency models, along with examples for each + of those models. + .. rubric:: asyncio REPL You can experiment with an ``asyncio`` concurrent context in the :term:`REPL`: diff --git a/Doc/library/concurrency.rst b/Doc/library/concurrency.rst index 5be1a1106b09a0..1c8676f9ff4f9d 100644 --- a/Doc/library/concurrency.rst +++ b/Doc/library/concurrency.rst @@ -8,8 +8,9 @@ The modules described in this chapter provide support for concurrent execution of code. The appropriate choice of tool will depend on the task to be executed (CPU bound vs IO bound) and preferred style of development (event driven cooperative multitasking vs preemptive -multitasking). Here's an overview: +multitasking). See the :ref:`concurrency-howto`. +Here's an overview of the modules: .. toctree:: diff --git a/Doc/library/concurrent.futures.rst b/Doc/library/concurrent.futures.rst index 7efae9e628b828..3a3fb1e461582d 100644 --- a/Doc/library/concurrent.futures.rst +++ b/Doc/library/concurrent.futures.rst @@ -22,6 +22,12 @@ by the abstract :class:`Executor` class. .. include:: ../includes/wasm-notavail.rst +.. seealso:: + + The :ref:`concurrency-howto` offers explanations about concurrency + and different concurrency models, along with examples for each + of those models. + Executor Objects ---------------- diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index e44142a8ed3106..5a625ec0be186b 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -54,6 +54,10 @@ will print to standard output :: the submission of work to the underlying process pool to be separated from waiting for the results. + The :ref:`concurrency-howto` offers explanations about concurrency + and different concurrency models, along with examples for each + of those models. + The :class:`Process` class ^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/Doc/library/threading.rst b/Doc/library/threading.rst index d948493c2103df..8e59cfd6e4bb41 100644 --- a/Doc/library/threading.rst +++ b/Doc/library/threading.rst @@ -26,6 +26,10 @@ level :mod:`_thread` module. :mod:`asyncio` offers an alternative approach to achieving task level concurrency without requiring the use of multiple operating system threads. + The :ref:`concurrency-howto` offers explanations about concurrency + and different concurrency models, along with examples for each + of those models. + .. note:: In the Python 2.x series, this module contained ``camelCase`` names diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index b3096a9f0c3820..0e3b1cc1a99bb7 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -3706,6 +3706,8 @@ object itself in order to be consistently invoked by the interpreter). .. index:: single: coroutine +.. _coroutine-protocol: + Coroutines ========== diff --git a/Misc/NEWS.d/next/Documentation/2024-08-20-14-43-05.gh-issue-123152.8J0smG.rst b/Misc/NEWS.d/next/Documentation/2024-08-20-14-43-05.gh-issue-123152.8J0smG.rst new file mode 100644 index 00000000000000..ccbb089a5ada77 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2024-08-20-14-43-05.gh-issue-123152.8J0smG.rst @@ -0,0 +1 @@ +Add a new concurrency HOWTO page to the docs.