Merge remote-tracking branch 'origin/main' into c23-math-pifuns

Author: jepler

Doc/c-api/threads.rst

@@ -736,10 +736,10 @@ Low-level APIs
 .. c:function:: PyObject* PyThreadState_GetDict()
 
    Return a dictionary in which extensions can store thread-specific state
-   information.  Each extension should use a unique key to use to store state in
+   information.  Each extension should use a unique key to store a state in
    the dictionary.  It is okay to call this function when no :term:`thread state`
    is :term:`attached <attached thread state>`. If this function returns
-   ``NULL``, no exception has been raised and the caller should assume no
+   ``NULL`` and no exception has been raised, then the caller should assume no
    thread state is attached.
 
 

Doc/library/ctypes.rst

@@ -14,6 +14,14 @@ used to wrap these libraries in pure Python.
 
 .. include:: ../includes/optional-module.rst
 
+.. warning::
+
+   :mod:`!ctypes` provides low-level access to native libraries and the
+   process's memory, bypassing Python's safety mechanisms and allowing
+   execution of arbitrary native code.
+   Incorrect use can corrupt data and objects, reveal sensitive information,
+   cause crashes, or otherwise compromise the running process.
+
 
 .. _ctypes-ctypes-tutorial:
 
@@ -198,10 +206,8 @@ argument values::
    OSError: exception: access violation reading 0x00000020
    >>>
 
-There are, however, enough ways to crash Python with :mod:`!ctypes`, so you
-should be careful anyway.  The :mod:`faulthandler` module can be helpful in
-debugging crashes (e.g. from segmentation faults produced by erroneous C library
-calls).
+The :mod:`faulthandler` module can help debug crashes,
+such as segmentation faults produced by erroneous C library calls.
 
 ``None``, integers, bytes objects and (unicode) strings are the only native
 Python objects that can directly be used as parameters in these function calls.

Doc/library/enum.rst

@@ -222,7 +222,7 @@ Data types
       names of the members in *cls*::
 
         >>> dir(Color)
-        ['BLUE', 'GREEN', 'RED', '__class__', '__contains__', '__doc__', '__getitem__', '__init_subclass__', '__iter__', '__len__', '__members__', '__module__', '__name__', '__qualname__']
+        ['BLUE', 'GREEN', 'RED', '__class__', '__contains__', '__doc__', '__getitem__', '__init_subclass__', '__iter__', '__len__', '__members__', '__module__', '__name__', '__qualname__', '_generate_next_value_', '_missing_']
 
    .. method:: EnumType.__getitem__(cls, name)
 
@@ -355,7 +355,7 @@ Data types
          ...         print(f'today is {cls(dt.date.today().isoweekday()).name}')
          ...
          >>> dir(Weekday.SATURDAY)
-         ['__class__', '__doc__', '__eq__', '__hash__', '__module__', 'name', 'today', 'value']
+         ['__class__', '__doc__', '__eq__', '__hash__', '__module__', '_add_alias_', '_add_value_alias_', '_generate_next_value_', '_missing_', 'name', 'today', 'value']
 
    .. method:: Enum._generate_next_value_(name, start, count, last_values)
 

Doc/library/lzma.rst

@@ -356,12 +356,26 @@ options. Valid filter IDs are as follows:
 
 * Branch-Call-Jump (BCJ) filters:
 
-  * :const:`FILTER_X86`
-  * :const:`FILTER_IA64`
-  * :const:`FILTER_ARM`
-  * :const:`FILTER_ARMTHUMB`
-  * :const:`FILTER_POWERPC`
-  * :const:`FILTER_SPARC`
+  * :const:`!FILTER_X86`
+  * :const:`!FILTER_IA64`
+  * :const:`!FILTER_ARM`
+  * :const:`!FILTER_ARMTHUMB`
+  * :const:`!FILTER_POWERPC`
+  * :const:`!FILTER_SPARC`
+
+    The above work on all lzma runtime library versions.
+
+  * :const:`!FILTER_ARM64`
+
+    Only works if the lzma version is 5.4.0 or later.
+
+    .. versionadded:: next
+
+  * :const:`!FILTER_RISCV`
+
+    Only works if the lzma version is 5.6.0 or later.
+
+    .. versionadded:: next
 
 A filter chain can consist of up to 4 filters, and cannot be empty. The last
 filter in the chain must be a compression filter, and any other filters must be

Doc/library/pydoc.rst

@@ -68,6 +68,11 @@ will start a HTTP server on port 1234, allowing you to browse the
 documentation at ``http://localhost:1234/`` in your preferred web browser.
 Specifying ``0`` as the port number will select an arbitrary unused port.
 
+.. warning::
+
+   The :mod:`!pydoc` HTTP server is intended for local use during
+   development and is not suitable for production use.
+
 :program:`python -m pydoc -n <hostname>` will start the server listening at the given
 hostname.  By default the hostname is 'localhost' but if you want the server to
 be reached from other machines, you may want to change the host name that the

Doc/whatsnew/3.16.rst

@@ -93,6 +93,15 @@ gzip
   which is passed on to the constructor of the :class:`~gzip.GzipFile` class.
   (Contributed by Marin Misur in :gh:`91372`.)
 
+lzma
+----
+
+* Add support of new BCJ filters ARM64 and RISC-V via
+  :const:`!lzma.FILTER_ARM64` and :const:`!lzma.FILTER_RISCV`.  Note that the
+  new filters will work only if runtime library supports them. ARM64 filter
+  requires ``lzma`` 5.4.0 or newer while RISC-V requires 5.6.0 or newer.
+  (Contributed by Chien Wong in :gh:`115988`.)
+
 math
 ----
 

Include/internal/pycore_interp_structs.h

@@ -349,7 +349,15 @@ struct _import_state {
     int lazy_imports_mode;
     PyObject *lazy_imports_filter;
     PyObject *lazy_importing_modules;
+    // The set stored in sys.lazy_modules if values that have been
+    // lazily imported. This value is only for debugging/introspection
+    // purposes and is not used by the runtime.
     PyObject *lazy_modules;
+    // A dict mapping package names to a set of submodule names that
+    // have been imported lazily from packages which have been imported
+    // lazily. When the package is reified we need to add a
+    // LazyImportObject which refers to the submodule on the module.
+    PyObject *lazy_pending_submodules;
 #ifdef Py_GIL_DISABLED
     PyMutex lazy_mutex;
 #endif
@@ -1001,7 +1009,7 @@ struct _is {
     struct ast_state ast;
     struct types_state types;
     struct callable_cache callable_cache;
-    PyObject *common_consts[NUM_COMMON_CONSTANTS];
+    _PyStackRef common_consts[NUM_COMMON_CONSTANTS];
     bool jit;
     bool compiling;
 

Include/internal/pycore_stackref.h

@@ -263,6 +263,18 @@ _PyStackRef_DUP(_PyStackRef ref, const char *filename, int linenumber)
 }
 #define PyStackRef_DUP(REF) _PyStackRef_DUP(REF, __FILE__, __LINE__)
 
+static inline _PyStackRef
+_PyStackRef_DupImmortal(_PyStackRef ref, const char *filename, int linenumber)
+{
+    assert(!PyStackRef_IsError(ref));
+    assert(!PyStackRef_IsTaggedInt(ref));
+    assert(!PyStackRef_RefcountOnObject(ref));
+    PyObject *obj = _Py_stackref_get_object(ref);
+    assert(_Py_IsImmortal(obj));
+    return _Py_stackref_create(obj, Py_TAG_REFCNT, filename, linenumber);
+}
+#define PyStackRef_DupImmortal(REF) _PyStackRef_DupImmortal((REF), __FILE__, __LINE__)
+
 static inline void
 _PyStackRef_CLOSE_SPECIALIZED(_PyStackRef ref, destructor destruct, const char *filename, int linenumber)
 {
@@ -633,6 +645,15 @@ PyStackRef_DUP(_PyStackRef ref)
 }
 #endif
 
+static inline _PyStackRef
+PyStackRef_DupImmortal(_PyStackRef ref)
+{
+    assert(!PyStackRef_IsNull(ref));
+    assert(!PyStackRef_RefcountOnObject(ref));
+    assert(_Py_IsImmortal(BITS_TO_PTR_MASKED(ref)));
+    return ref;
+}
+
 static inline bool
 PyStackRef_IsHeapSafe(_PyStackRef ref)
 {

InternalDocs/compiler.md

@@ -359,19 +359,19 @@ in [Python/compile.c](../Python/compile.c) into a sequence of pseudo instruction
 These are similar to bytecode, but in some cases they are more abstract, and are
 resolved later into actual bytecode. The construction of this instruction sequence
 is handled by several functions that break the task down by various AST node types.
-The functions are all named `compiler_visit_{xx}` where *xx* is the name of the node
+The functions are all named `codegen_visit_{xx}` where *xx* is the name of the node
 type (such as `stmt`, `expr`, etc.).  Each function receives a `struct compiler *`
 and `{xx}_ty` where *xx* is the AST node type.  Typically these functions
 consist of a large 'switch' statement, branching based on the kind of
 node type passed to it.  Simple things are handled inline in the
 'switch' statement with more complex transformations farmed out to other
-functions named `compiler_{xx}` with *xx* being a descriptive name of what is
+functions named `codegen_{xx}` with *xx* being a descriptive name of what is
 being handled.
 
 When transforming an arbitrary AST node, use the `VISIT()` macro.
-The appropriate `compiler_visit_{xx}` function is called, based on the value
+The appropriate `codegen_visit_{xx}` function is called, based on the value
 passed in for <node type> (so `VISIT({c}, expr, {node})` calls
-`compiler_visit_expr({c}, {node})`).  The `VISIT_SEQ()` macro is very similar,
+`codegen_visit_expr({c}, {node})`).  The `VISIT_SEQ()` macro is very similar,
 but is called on AST node sequences (those values that were created as
 arguments to a node that used the '*' modifier).
 
@@ -414,8 +414,8 @@ which is added at the end of a function is not associated with any
 line in the source code.
 
 There are several helper functions that will emit pseudo-instructions
-and are named `compiler_{xx}()` where *xx* is what the function helps
-with (`list`, `boolop`, etc.).  A rather useful one is `compiler_nameop()`.
+and are named `codegen_{xx}()` where *xx* is what the function helps
+with (`list`, `boolop`, etc.).  A rather useful one is `codegen_nameop()`.
 This function looks up the scope of a variable and, based on the
 expression context, emits the proper opcode to load, store, or delete
 the variable.

InternalDocs/qsbr.md

@@ -101,15 +101,39 @@ Periodically, a polling mechanism processes this deferred-free list:
 
 To reduce memory contention from frequent updates to the global `wr_seq`, its
 advancement is sometimes deferred. Instead of incrementing `wr_seq` on every
-reclamation request, each thread tracks its number of deferrals locally. Once
-the deferral count reaches a limit (QSBR_DEFERRED_LIMIT, currently 10), the
-thread advances the global `wr_seq` and resets its local count.
-
-When an object is added to the deferred-free list, its qsbr_goal is set to
-`wr_seq` + 2. By setting the goal to the next sequence value, we ensure it's safe
-to defer the global counter advancement. This optimization improves runtime
-speed but may increase peak memory usage by slightly delaying when memory can
-be reclaimed.
+reclamation request, the object's qsbr_goal is set to `wr_seq` + 2 (the value
+the counter *would* take on its next advance) without actually advancing the
+global counter. This is safe because the goal still corresponds to a future
+sequence value that no thread has yet observed as quiescent.
+
+Whether to actually advance `wr_seq` is decided per request, based on how
+much memory and how many items the calling thread has already deferred since
+its last advance:
+
+* For deferred object frees (`_PyMem_FreeDelayed`), the thread tracks both a
+  count (`deferred_count`) and an estimate of the held memory
+  (`deferred_memory`). The global `wr_seq` is advanced when the freed block
+  is larger than `QSBR_FREE_MEM_LIMIT` (1 MiB), when the accumulated deferred
+  memory exceeds that limit, or when the count exceeds `QSBR_DEFERRED_LIMIT`
+  (127, sized so a chunk of work items is processed before it overflows).
+  Crossing any of these thresholds also sets a per-thread `should_process`
+  flag, signalling that the deferred-free list should be drained.
+
+* For mimalloc pages held by QSBR, the thread tracks `deferred_page_memory`
+  and advances `wr_seq` when either the individual page or the accumulated
+  page memory exceeds `QSBR_PAGE_MEM_LIMIT` (4096 * 20 bytes). Advancing
+  promptly here matters because a held page cannot be reused for a different
+  size class or by a different thread.
+
+Processing of the deferred-free list normally happens from the eval breaker
+(rather than from inside `_PyMem_FreeDelayed`), which gives the global
+`rd_seq` a better chance to have advanced far enough that items can actually
+be freed. `_PyMem_ProcessDelayed` is still called from the free path as a
+safety valve when a work-item chunk fills up.
+
+This optimization improves runtime speed but may increase peak memory usage
+by slightly delaying when memory can be reclaimed; the size-based thresholds
+above bound that extra memory.
 
 
 ## Limitations