gh-99633: New generator/coroutine _context property

Add a new `_context` property to generator (and coroutine) objects to
get/set the "current context" that is observed by (and only by) the
generator and the functions it calls.

When `generator._context` is set to `None` (the default), the
generator is called a "dependent generator".  It behaves the same as
it always has: the "current context" observed by the generator is the
thread's context.  This means that the observed context can change
arbitrarily during a `yield`; the generator *depends* on the sender to
enter the appropriate context before it calls `generator.send`.

When `generator._context` is set to a `contextvars.Context` object,
the generator is called an "independent generator".  It acts more like
a separate thread with its own independent context stack.  The value
of `_context` is the head of that independent stack.  Whenever the
generator starts or resumes execution (via `generator.send`), the
current context temporarily becomes the generator's associated
context.  When the generator yields, returns, or propagates an
exception, the current context reverts back to what it was before.
The generator's context is *independent* from the sender's context.

If an independent generator calls `contextvars.Context.run`, then the
value of the `_context` property will (temporarily) change to the
newly entered context.

If an independent generator sends a value to a second independent
generator, the second generator's context will shadow the first
generator's context until the second generator returns or yields.

The `generator._context` property is private for now until experience
and feedback is collected.  Nothing is using this yet, but that will
change in future commits.

Motivations for this change:

  * First, this change makes it possible for a future commit to add
    context manager support to `contextvars.Context`.  A `yield` after
    entering a context causes execution to leave the generator with a
    different context at the top of the context stack than when
    execution started.  Swapping contexts in and out when execution
    suspends and resumes can only be done by the generator itself.

  * Second, this paves the way for a public API that will enable
    developers to guarantee that the context remains consistent
    throughout a generator's execution.  Right now the context can
    change arbitrarily during a `yield`, which can lead to subtle bugs
    that are difficult to root cause.  (Coroutines run by an asyncio
    event loop do not suffer from this same problem because asyncio
    manually sets the context each time it executes a step of an
    asynchronous function.  See the call to `contextvars.Context.run`
    in `asyncio.Handle._run`.)

  * Finally, this makes it possible to move the responsibility for
    activating an async coroutine's context from the event loop to the
    coroutine, where it more naturally belongs (alongside the rest of
    the execution state such as local variable bindings and the
    instruction pointer).  This ensures consistent behavior between
    different event loop implementations.

Example:

```python
import contextvars

cvar = contextvars.ContextVar('cvar', default='initial')

def make_generator():
    yield cvar.get()
    yield cvar.get()
    yield cvar.get()
    yield cvar.get()
    cvar.set('updated by generator')
    yield cvar.get()

gen = make_generator()
print('1.', next(gen))

def callback():
    cvar.set('updated by callback')
    print('2.', next(gen))

contextvars.copy_context().run(callback)
print('3.', next(gen))
cvar.set('updated at top level')
print('4.', next(gen))
print('5.', next(gen))
print('6.', cvar.get())
```

The above prints:

```
1. initial
2. updated by callback
3. initial
4. updated at top level
5. updated by generator
6. updated by generator
```

Now add the following line after the creation of the generator:

```python
gen._context = contextvars.copy_context()
```

With that change, the script now outputs:

```
1. initial
2. initial
3. initial
4. initial
5. updated by generator
6. updated by top level
```

Author: rhansen

Include/cpython/pystate.h

@@ -164,7 +164,6 @@ struct _ts {
     PyObject *async_gen_firstiter;
     PyObject *async_gen_finalizer;
 
-    PyObject *context;
     uint64_t context_ver;
 
     /* Unique thread state id. */

Include/internal/pycore_context.h

@@ -5,6 +5,8 @@
 #  error "this header requires Py_BUILD_CORE define"
 #endif
 
+#include "cpython/context.h"
+#include "cpython/genobject.h"    // PyGenObject
 #include "pycore_hamt.h"          // PyHamtObject
 
 #define CONTEXT_MAX_WATCHERS 8
@@ -30,6 +32,46 @@ struct _pycontextobject {
     int ctx_entered;
 };
 
+// Resets a coroutine's independent context stack to ctx.  If ctx is NULL or
+// Py_None, the coroutine will be a dependent coroutine (its context stack will
+// be empty) upon successful return.  Otherwise, the coroutine will be an
+// independent coroutine upon successful return, with ctx as the sole item on
+// its context stack.
+//
+// The coroutine's existing stack must be empty (NULL) or contain only a single
+// entry (from a previous call to this function).  If the coroutine is
+// currently executing, this function must be called from the coroutine's
+// thread.
+//
+// Unless ctx already equals the coroutine's existing context stack, the
+// context on the existing stack (if one exists) is immediately exited and ctx
+// (if non-NULL) is immediately entered.
+int _PyGen_ResetContext(PyThreadState *ts, PyGenObject *self, PyObject *ctx);
+
+// Makes the given coroutine's context stack the active context stack for the
+// thread, shadowing (temporarily deactivating) the thread's previously active
+// context stack.  The context stack remains active until deactivated with a
+// call to _PyGen_DeactivateContext, as long as it is not shadowed by another
+// activated context stack.
+//
+// Each activated context stack must eventually be deactivated by calling
+// _PyGen_DeactivateContext.  The same context stack cannot be activated again
+// until deactivated.
+//
+// If the coroutine's context stack is empty this function has no effect.
+void _PyGen_ActivateContext(PyThreadState *ts, PyGenObject *self);
+
+// Deactivates the given coroutine's context stack, un-shadowing (reactivating)
+// the thread's previously active context stack.  Does not affect any contexts
+// in the coroutine's context stack (they remain entered).
+//
+// Must not be called if a different context stack is currently shadowing the
+// coroutine's context stack.
+//
+// If the coroutine's context stack is not the active context stack this
+// function has no effect.
+void _PyGen_DeactivateContext(PyThreadState *ts, PyGenObject *self);
+
 
 struct _pycontextvarobject {
     PyObject_HEAD

Include/internal/pycore_contextchain.h

@@ -0,0 +1,79 @@
+#ifndef Py_INTERNAL_CONTEXTCHAIN_H
+#define Py_INTERNAL_CONTEXTCHAIN_H
+
+#ifndef Py_BUILD_CORE
+#  error "this header requires Py_BUILD_CORE define"
+#endif
+
+#include "pytypedefs.h"  // PyObject
+
+
+// Circularly linked chain of multiple independent context stacks, used to give
+// coroutines (including generators) their own (optional) independent context
+// stacks.
+//
+// Detailed notes on how this chain is used:
+//   * The chain is circular simply to save a pointer's worth of memory in
+//     _PyThreadStateImpl.  It is actually used as an ordinary linear linked
+//     list.  It is called "chain" instead of "stack" or "list" to evoke "call
+//     chain", which it is related to, and to avoid confusion with "context
+//     stack".
+//   * There is one chain per thread, and _PyThreadStateImpl::_ctx_chain::prev
+//     points to the head of the thread's chain.
+//   * A thread's chain is never empty.
+//   * _PyThreadStateImpl::_ctx_chain is always the tail entry of the thread's
+//     chain.
+//   * _PyThreadStateImpl::_ctx_chain is usually the only link in the thread's
+//     chain, so _PyThreadStateImpl::_ctx_chain::prev usually points to the
+//     _PyThreadStateImpl::_ctx_chain itself.
+//   * The "active context stack" is always at the head link in a thread's
+//     context chain.  Contexts are entered by pushing onto the active context
+//     stack and exited by popping off of the active context stack.
+//   * The "current context" is the top context in the active context stack.
+//     Context variable accesses (reads/writes) use the current context.
+//   * A *dependent* coroutine or generator is a coroutine or generator that
+//     does not have its own independent context stack.  When a dependent
+//     coroutine starts or resumes execution, the current context -- as
+//     observed by the coroutine -- is the same context that was current just
+//     before the coroutine's `send` method was called.  This means that the
+//     current context as observed by a dependent coroutine can change
+//     arbitrarily during a yield/await.  Dependent coroutines are so-named
+//     because they depend on their senders to enter the appropriate context
+//     before each send.  Coroutines and generators are dependent by default
+//     for backwards compatibility.
+//   * The purpose of the context chain is to enable *independent* coroutines
+//     and generators, which have their own context stacks.  Whenever an
+//     independent coroutine starts or resumes execution, the current context
+//     automatically switches to the context associated with the coroutine.
+//     This is accomplished by linking the coroutine's chain link (at
+//     PyGenObject::_ctx_chain) to the head of the thread's chain.  Independent
+//     coroutines are so-named because they do not depend on their senders to
+//     enter the appropriate context before each send.
+//   * The head link is unlinked from the thread's chain when its associated
+//     independent coroutine or generator stops executing (yields, awaits,
+//     returns, or throws).
+//   * A running dependent coroutine's chain link is linked into the thread's
+//     chain if the coroutine is upgraded from dependent to independent by
+//     assigning a context to the coroutine's `_context` property.  The chain
+//     link is inserted at the position corresponding to the coroutine's
+//     position in the call chain relative to any other currently running
+//     independent coroutines.  For example, if dependent coroutine `coro_a`
+//     calls function `func_b` which resumes independent coroutine `coro_c`
+//     which assigns a context to `coro_a._context`, then `coro_a` becomes an
+//     independent coroutine with its chain link inserted after `coro_c`'s
+//     chain link (which remains the head link).
+//   * A running independent coroutine's chain link is unlinked from the
+//     thread's chain if the coroutine is downgraded from independent to
+//     dependent by assigning `None` to its `_context` property.
+//   * The references to the object at the `prev` link in the chain are
+//     implicit (borrowed).
+typedef struct _PyContextChain {
+    // NULL for dependent coroutines/generators, non-NULL for independent
+    // coroutines/generators.
+    PyObject *ctx;
+    // NULL if unlinked from the thread's context chain, non-NULL otherwise.
+    struct _PyContextChain *prev;
+} _PyContextChain;
+
+
+#endif /* !Py_INTERNAL_CONTEXTCHAIN_H */

Include/internal/pycore_genobject.h

@@ -1,5 +1,8 @@
 #ifndef Py_INTERNAL_GENOBJECT_H
 #define Py_INTERNAL_GENOBJECT_H
+
+#include "pycore_contextchain.h"  // _PyContextChain
+
 #ifdef __cplusplus
 extern "C" {
 #endif
@@ -22,6 +25,7 @@ extern "C" {
     PyObject *prefix##_qualname;                                            \
     _PyErr_StackItem prefix##_exc_state;                                    \
     PyObject *prefix##_origin_or_finalizer;                                 \
+    _PyContextChain _ctx_chain;                                             \
     char prefix##_hooks_inited;                                             \
     char prefix##_closed;                                                   \
     char prefix##_running_async;                                            \

Include/internal/pycore_tstate.h

@@ -9,6 +9,7 @@ extern "C" {
 #endif
 
 #include "pycore_brc.h"             // struct _brc_thread_state
+#include "pycore_contextchain.h"    // _PyContextChain
 #include "pycore_freelist_state.h"  // struct _Py_freelists
 #include "pycore_mimalloc.h"        // struct _mimalloc_thread_state
 #include "pycore_qsbr.h"            // struct qsbr
@@ -21,6 +22,9 @@ typedef struct _PyThreadStateImpl {
     // semi-public fields are in PyThreadState.
     PyThreadState base;
 
+    // Lazily initialized (must be zeroed at startup).
+    _PyContextChain _ctx_chain;
+
     PyObject *asyncio_running_loop; // Strong reference
 
     struct _qsbr_thread_state *qsbr;  // only used by free-threaded build