gh-132983: Introduce _zstd bindings module (GH-133027)

* Add _zstd module for https://peps.python.org/pep-0784/

This commit introduces the `_zstd` module, with bindings to libzstd from
the pyzstd project. It also includes the unix build system configuration.
Windows build system support will be integrated independently as it
depends on integration with cpython-source-deps.

* Add _zstd to modules

* Fix path for compression.zstd module

* Ignore _zstd module like _io

* Expand module state macros to improve code quality

Also removes module state references from the classes in the _zstd
module and instead uses PyType_GetModuleState()

* Remove backticks suggested in review

Co-authored-by: Stan Ulbrych <[email protected]>

* Use critical sections to lock object state

This should avoid races and deadlocks.

* Remove compress/decompress and mark module as not reliant on the GIL

The `compress`/`decompress` functions will be moved to Python code for simplicity.
C implementations can always be re-added in the future.

Also, mark _zstd as not requiring the GIL.

* Lift critical section to avoid clang warning

* Respond to comments by picnixz

* Call out pyzstd explicitly in license description

Co-authored-by: Adam Turner <[email protected]>

* Use a much more robust implementation...

... for `get_zstd_state_from_type`

Co-authored-by: Bénédikt Tran <[email protected]>

* Use PyList_GetItemRef for thread safety purposes

* Use a macro for the minimum supported version

* remove const from primivite types

* Use PyMem_New in another spot

* Simplify error handling in _get_frame_size

* Another simplification of error handling in get_frame_info

* Rename _module_state to mod_state

* Rewrite comment explaining the context of the code

* Add link to pyzstd

* Add TODO about refactoring dict training code

* Use PyModule_AddObjectRef over PyModule_AddObject

PyModule_AddObject is soft-deprecated, so we should use PyModule_AddObjectRef

* Check result of OutputBufferGrow

* Simplify return logic in `add_constant_to_type`

Co-authored-by: Bénédikt Tran <[email protected]>

* Ignore return value of _zstd_clear()

Co-authored-by: Bénédikt Tran <[email protected]>

* Remove redundant comments

* Remove __reduce__ from ZstdDict

We should instead document that to pickle a dictionary a user should use
the `.dict_content` attribute.

* Use PyUnicode_FromFormat instead of a buffer

* Don't use C constants/types in error messages

* Make error messages easier to understand for Python users

* Lower minimum required version 1.4.0

* Use casts and make slot function signatures correct

* Be consistent with CPython on const usage

* Make else clauses in line with PEP 7

* Fix over-indented blocks in argument clinic

* Add critical section around ZSTD_DCtx_setParameter

* Add a TODO about refactoring critical sections

* Use Py_UNREACHABLE

* Move bytes operations out of Py_BEGIN_ALLOW_THREADS

* Add TODO about ensuring a lock is held

* Remove asserts that may not be correct

* Add TODO to make ZstdDict and others GC objects

* Make objects GC tracked

* Remove unused include

* Fix some memory issues

* Fix refleaks on module and in ZstdDict

* Update configure to check for ZDICT_finalizeDictionary

* Properly check version in configure

* exit(1) if check fails

* Use AC_RUN_IFELSE

* Use a define() to re-use version check

* Actually properly set _zstd module status based on version

---------

Co-authored-by: Stan Ulbrych <[email protected]>
Co-authored-by: Adam Turner <[email protected]>
Co-authored-by: Bénédikt Tran <[email protected]>

Author: 4 people

Doc/license.rst

@@ -1132,3 +1132,40 @@ The file is distributed under the 2-Clause BSD License::
   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
   THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+
+Zstandard bindings
+------------------
+
+Zstandard bindings in :file:`Modules/_zstd` and :file:`Lib/compression/zstd`
+are based on code from the
+`pyzstd library <https://github.com/Rogdham/pyzstd/>`_, copyright Ma Lin and
+contributors. The pyzstd code is distributed under the 3-Clause BSD License::
+
+  Copyright (c) 2020-present, Ma Lin and contributors.
+  All rights reserved.
+
+  Redistribution and use in source and binary forms, with or without
+  modification, are permitted provided that the following conditions are met:
+
+  1. Redistributions of source code must retain the above copyright notice, this
+     list of conditions and the following disclaimer.
+
+  2. Redistributions in binary form must reproduce the above copyright notice,
+     this list of conditions and the following disclaimer in the documentation
+     and/or other materials provided with the distribution.
+
+  3. Neither the name of the copyright holder nor the names of its
+     contributors may be used to endorse or promote products derived from
+     this software without specific prior written permission.
+
+  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+  AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+  IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+  DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+  FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+  DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+  SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+  CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+  OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Include/internal/pycore_global_objects_fini_generated.h

@@ -325,6 +325,7 @@ struct _Py_global_strings {
         STRUCT_FOR_ID(bytes_per_sep)
         STRUCT_FOR_ID(c_call)
         STRUCT_FOR_ID(c_exception)
+        STRUCT_FOR_ID(c_parameter_type)
         STRUCT_FOR_ID(c_return)
         STRUCT_FOR_ID(cached_datetime_module)
         STRUCT_FOR_ID(cached_statements)
@@ -379,6 +380,7 @@ struct _Py_global_strings {
         STRUCT_FOR_ID(count)
         STRUCT_FOR_ID(covariant)
         STRUCT_FOR_ID(cwd)
+        STRUCT_FOR_ID(d_parameter_type)
         STRUCT_FOR_ID(data)
         STRUCT_FOR_ID(database)
         STRUCT_FOR_ID(day)
@@ -393,6 +395,7 @@ struct _Py_global_strings {
         STRUCT_FOR_ID(deterministic)
         STRUCT_FOR_ID(device)
         STRUCT_FOR_ID(dict)
+        STRUCT_FOR_ID(dict_content)
         STRUCT_FOR_ID(dictcomp)
         STRUCT_FOR_ID(difference_update)
         STRUCT_FOR_ID(digest)
@@ -459,6 +462,7 @@ struct _Py_global_strings {
         STRUCT_FOR_ID(follow_symlinks)
         STRUCT_FOR_ID(format)
         STRUCT_FOR_ID(format_spec)
+        STRUCT_FOR_ID(frame_buffer)
         STRUCT_FOR_ID(from_param)
         STRUCT_FOR_ID(fromlist)
         STRUCT_FOR_ID(fromtimestamp)
@@ -517,6 +521,8 @@ struct _Py_global_strings {
         STRUCT_FOR_ID(intersection)
         STRUCT_FOR_ID(interval)
         STRUCT_FOR_ID(io)
+        STRUCT_FOR_ID(is_compress)
+        STRUCT_FOR_ID(is_raw)
         STRUCT_FOR_ID(is_running)
         STRUCT_FOR_ID(is_struct)
         STRUCT_FOR_ID(isatty)
@@ -640,6 +646,7 @@ struct _Py_global_strings {
         STRUCT_FOR_ID(overlapped)
         STRUCT_FOR_ID(owner)
         STRUCT_FOR_ID(pages)
+        STRUCT_FOR_ID(parameter)
         STRUCT_FOR_ID(parent)
         STRUCT_FOR_ID(password)
         STRUCT_FOR_ID(path)
@@ -801,6 +808,7 @@ struct _Py_global_strings {
         STRUCT_FOR_ID(write_through)
         STRUCT_FOR_ID(year)
         STRUCT_FOR_ID(zdict)
+        STRUCT_FOR_ID(zstd_dict)
     } identifiers;
     struct {
         PyASCIIObject _ascii;

Include/internal/pycore_global_strings.h

@@ -3341,6 +3341,7 @@ MODULE__TESTCAPI_DEPS=$(srcdir)/Modules/_testcapi/parts.h $(srcdir)/Modules/_tes
 MODULE__TESTLIMITEDCAPI_DEPS=$(srcdir)/Modules/_testlimitedcapi/testcapi_long.h $(srcdir)/Modules/_testlimitedcapi/parts.h $(srcdir)/Modules/_testlimitedcapi/util.h
 MODULE__TESTINTERNALCAPI_DEPS=$(srcdir)/Modules/_testinternalcapi/parts.h
 MODULE__SQLITE3_DEPS=$(srcdir)/Modules/_sqlite/connection.h $(srcdir)/Modules/_sqlite/cursor.h $(srcdir)/Modules/_sqlite/microprotocols.h $(srcdir)/Modules/_sqlite/module.h $(srcdir)/Modules/_sqlite/prepare_protocol.h $(srcdir)/Modules/_sqlite/row.h $(srcdir)/Modules/_sqlite/util.h
+MODULE__ZSTD_DEPS=$(srcdir)/Modules/_zstd/_zstdmodule.h $(srcdir)/Modules/_zstd/buffer.h
 
 CODECS_COMMON_HEADERS=$(srcdir)/Modules/cjkcodecs/multibytecodec.h $(srcdir)/Modules/cjkcodecs/cjkcodecs.h
 MODULE__CODECS_CN_DEPS=$(srcdir)/Modules/cjkcodecs/mappings_cn.h $(CODECS_COMMON_HEADERS)

Include/internal/pycore_runtime_init_generated.h

@@ -200,6 +200,7 @@ PYTHONPATH=$(COREPYTHONPATH)
 #_dbm _dbmmodule.c -lgdbm_compat -DUSE_GDBM_COMPAT
 #_gdbm _gdbmmodule.c -lgdbm
 #_lzma _lzmamodule.c -llzma
+#_zstd _zstd/_zstdmodule.c -lzstd -I$(srcdir)/Modules/_zstd
 #_uuid _uuidmodule.c -luuid
 #zlib  zlibmodule.c -lz
 

Include/internal/pycore_unicodeobject_generated.h

@@ -65,10 +65,11 @@
 @MODULE__DECIMAL_TRUE@_decimal _decimal/_decimal.c
 
 # compression libs and binascii (optional CRC32 from zlib)
-# bindings need -lbz2, -lz, or -llzma, respectively
+# bindings need -lbz2, -llzma, -lzstd, or -lz, respectively
 @MODULE_BINASCII_TRUE@binascii binascii.c
 @MODULE__BZ2_TRUE@_bz2 _bz2module.c
 @MODULE__LZMA_TRUE@_lzma _lzmamodule.c
+@MODULE__ZSTD_TRUE@_zstd _zstd/_zstdmodule.c _zstd/zdict.c _zstd/compressor.c _zstd/decompressor.c
 @MODULE_ZLIB_TRUE@zlib zlibmodule.c
 
 # dbm/gdbm