+++++++++++ Python News +++++++++++ What's New in Python 3.11.0 final? ================================== *Release date: 2022-10-24* Security -------- - gh-issue-97616: Fix multiplying a list by an integer (``list *= int``): detect the integer overflow when the new allocated length is close to the maximum size. Issue reported by Jordan Limor. Patch by Victor Stinner. - gh-issue-97514: On Linux the :mod:`multiprocessing` module returns to using filesystem backed unix domain sockets for communication with the *forkserver* process instead of the Linux abstract socket namespace. Only code that chooses to use the :ref:`"forkserver" start method <multiprocessing-start-methods>` is affected. Abstract sockets have no permissions and could allow any user on the system in the same `network namespace <https://man7.org/linux/man-pages/man7/network_namespaces.7.html>`_ (often the whole system) to inject code into the multiprocessing *forkserver* process. This was a potential privilege escalation. Filesystem based socket permissions restrict this to the *forkserver* process user as was the default in Python 3.8 and earlier. This prevents Linux `CVE-2022-42919 <https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-42919>`_. Core and Builtins ----------------- - gh-issue-97002: Fix an issue where several frame objects could be backed by the same interpreter frame, possibly leading to corrupted memory and hard crashes of the interpreter. - gh-issue-97752: Fix possible data corruption or crashes when accessing the ``f_back`` member of newly-created generator or coroutine frames. - gh-issue-96975: Fix a crash occurring when :c:func:`PyEval_GetFrame` is called while the topmost Python frame is in a partially-initialized state. - gh-issue-96848: Fix command line parsing: reject :option:`-X int_max_str_digits <-X>` option with no value (invalid) when the :envvar:`PYTHONINTMAXSTRDIGITS` environment variable is set to a valid limit. Patch by Victor Stinner. - gh-issue-96821: Fix undefined behaviour in ``_testcapimodule.c``. - gh-issue-95778: When :exc:`ValueError` is raised if an integer is larger than the limit, mention the :func:`sys.set_int_max_str_digits` function in the error message. Patch by Victor Stinner. - gh-issue-96587: Correctly raise ``SyntaxError`` on exception groups (:pep:`654`) on python versions prior to 3.11 - bpo-42316: Document some places where an assignment expression needs parentheses. Library ------- - gh-issue-98331: Update the bundled copies of pip and setuptools to versions 22.3 and 65.5.0 respectively. - gh-issue-90985: Earlier in 3.11 we deprecated ``asyncio.Task.cancel("message")``. We realized we were too harsh, and have undeprecated it. - gh-issue-97545: Make Semaphore run faster. - gh-issue-96865: fix Flag to use boundary CONFORM This restores previous Flag behavior of allowing flags with non-sequential values to be combined; e.g. class Skip(Flag): TWO = 2 EIGHT = 8 Skip.TWO | Skip.EIGHT -> <Skip.TWO|EIGHT: 10> - gh-issue-90155: Fix broken :class:`asyncio.Semaphore` when acquire is cancelled. Documentation ------------- - gh-issue-97741: Fix ``!`` in c domain ref target syntax via a ``conf.py`` patch, so it works as intended to disable ref target resolution. - gh-issue-93031: Update tutorial introduction output to use 3.10+ SyntaxError invalid range. Tests ----- - gh-issue-95027: On Windows, when the Python test suite is run with the ``-jN`` option, the ANSI code page is now used as the encoding for the stdout temporary file, rather than using UTF-8 which can lead to decoding errors. Patch by Victor Stinner. Build ----- - gh-issue-96729: Ensure that Windows releases built with ``Tools\msi\buildrelease.bat`` are upgradable to and from official Python releases. Windows ------- - gh-issue-98360: Fixes :mod:`multiprocessing` spawning child processes on Windows from a virtual environment to ensure that child processes that also use :mod:`multiprocessing` to spawn more children will recognize that they are in a virtual environment. - gh-issue-98414: Fix :file:`py.exe` launcher handling of ``-V:<company>/`` option when default preferences have been set in environment variables or configuration files. - gh-issue-90989: Clarify some text in the Windows installer. macOS ----- - gh-issue-97897: The macOS 13 SDK includes support for the ``mkfifoat`` and ``mknodat`` system calls. Using the ``dir_fd`` option with either :func:`os.mkfifo` or :func:`os.mknod` could result in a segfault if cpython is built with the macOS 13 SDK but run on an earlier version of macOS. Prevent this by adding runtime support for detection of these system calls ("weaklinking") as is done for other newer syscalls on macOS. What's New in Python 3.11.0 release candidate 2? ================================================ *Release date: 2022-09-11* Security -------- - gh-issue-95778: Converting between :class:`int` and :class:`str` in bases other than 2 (binary), 4, 8 (octal), 16 (hexadecimal), or 32 such as base 10 (decimal) now raises a :exc:`ValueError` if the number of digits in string form is above a limit to avoid potential denial of service attacks due to the algorithmic complexity. This is a mitigation for `CVE-2020-10735 <https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10735>`_. This new limit can be configured or disabled by environment variable, command line flag, or :mod:`sys` APIs. See the :ref:`integer string conversion length limitation <int_max_str_digits>` documentation. The default limit is 4300 digits in string form. Patch by Gregory P. Smith [Google] and Christian Heimes [Red Hat] with feedback from Victor Stinner, Thomas Wouters, Steve Dower, Ned Deily, and Mark Dickinson. Core and Builtins ----------------- - gh-issue-96678: Fix case of undefined behavior in ceval.c - gh-issue-96641: Do not expose ``KeyWrapper`` in :mod:`_functools`. - gh-issue-96636: Ensure that tracing, ``sys.setrace()``, is turned on immediately. In pre-release versions of 3.11, some tracing events might have been lost when turning on tracing in a ``__del__`` method or interrupt. - gh-issue-96572: Fix use after free in trace refs build mode. Patch by Kumar Aditya. - gh-issue-96611: When loading a file with invalid UTF-8 inside a multi-line string, a correct SyntaxError is emitted. - gh-issue-96612: Make sure that incomplete frames do not show up in tracemalloc traces. - gh-issue-96569: Remove two cases of undefined behavior, by adding NULL checks. - gh-issue-96582: Fix possible ``NULL`` pointer dereference in ``_PyThread_CurrentFrames``. Patch by Kumar Aditya. - gh-issue-96352: Fix :exc:`AttributeError` missing ``name`` and ``obj`` attributes in :meth:`object.__getattribute__`. Patch by Philip Georgi. - gh-issue-96268: Loading a file with invalid UTF-8 will now report the broken character at the correct location. - gh-issue-96187: Fixed a bug that caused ``_PyCode_GetExtra`` to return garbage for negative indexes. Patch by Pablo Galindo - gh-issue-96071: Fix a deadlock in :c:func:`PyGILState_Ensure` when allocating new thread state. Patch by Kumar Aditya. - gh-issue-96046: :c:func:`PyType_Ready` now initializes ``ht_cached_keys`` and performs additional checks to ensure that type objects are properly configured. This avoids crashes in 3rd party packages that don't use regular API to create new types. - gh-issue-95818: Skip over incomplete frames in :c:func:`PyThreadState_GetFrame`. - gh-issue-95876: Fix format string in ``_PyPegen_raise_error_known_location`` that can lead to memory corruption on some 64bit systems. The function was building a tuple with ``i`` (int) instead of ``n`` (Py_ssize_t) for Py_ssize_t arguments. - gh-issue-95605: Fix misleading contents of error message when converting an all-whitespace string to :class:`float`. - gh-issue-94996: :func:`ast.parse` will no longer parse function definitions with positional-only params when passed ``feature_version`` less than ``(3, 8)``. Patch by Shantanu Jain. Library ------- - gh-issue-96700: Fix incorrect error message in the :mod:`io` module. - gh-issue-96652: Fix the faulthandler implementation of ``faulthandler.register(signal, chain=True)`` if the ``sigaction()`` function is not available: don't call the previous signal handler if it's NULL. Patch by Victor Stinner. - gh-issue-68163: Correct conversion of :class:`numbers.Rational`'s to :class:`float`. - gh-issue-96385: Fix ``TypeVarTuple.__typing_prepare_subst__``. ``TypeError`` was not raised when using more than one ``TypeVarTuple``, like ``[*T, *V]`` in type alias substitutions. - gh-issue-90467: Fix :class:`asyncio.streams.StreamReaderProtocol` to keep a strong reference to the created task, so that it's not garbage collected - gh-issue-96159: Fix a performance regression in logging TimedRotatingFileHandler. Only check for special files when the rollover time has passed. - gh-issue-96175: Fix unused ``localName`` parameter in the ``Attr`` class in :mod:`xml.dom.minidom`. - gh-issue-96125: Fix incorrect condition that causes ``sys.thread_info.name`` to be wrong on pthread platforms. - gh-issue-95463: Remove an incompatible change from :issue:`28080` that caused a regression that ignored the utf8 in ``ZipInfo.flag_bits``. Patch by Pablo Galindo. - gh-issue-95899: Fix :class:`asyncio.Runner` to call :func:`asyncio.set_event_loop` only once to avoid calling :meth:`~asyncio.AbstractChildWatcher.attach_loop` multiple times on child watchers. Patch by Kumar Aditya. - gh-issue-95736: Fix :class:`unittest.IsolatedAsyncioTestCase` to set event loop before calling setup functions. Patch by Kumar Aditya. - gh-issue-95704: When a task catches :exc:`asyncio.CancelledError` and raises some other error, the other error should generally not silently be suppressed. - gh-issue-95231: Fail gracefully if :data:`~errno.EPERM` or :data:`~errno.ENOSYS` is raised when loading :mod:`crypt` methods. This may happen when trying to load ``MD5`` on a Linux kernel with :abbr:`FIPS (Federal Information Processing Standard)` enabled. - gh-issue-74116: Allow :meth:`asyncio.StreamWriter.drain` to be awaited concurrently by multiple tasks. Patch by Kumar Aditya. - gh-issue-92986: Fix :func:`ast.unparse` when ``ImportFrom.level`` is None Documentation ------------- - gh-issue-96098: Improve discoverability of the higher level concurrent.futures module by providing clearer links from the lower level threading and multiprocessing modules. - gh-issue-95957: What's New 3.11 now has instructions for how to provide compiler and linker flags for Tcl/Tk and OpenSSL on RHEL 7 and CentOS 7. Tests ----- - gh-issue-95243: Mitigate the inherent race condition from using find_unused_port() in testSockName() by trying to find an unused port a few times before failing. Patch by Ross Burton. Build ----- - gh-issue-94682: Build and test with OpenSSL 1.1.1q Windows ------- - gh-issue-96577: Fixes a potential buffer overrun in :mod:`msilib`. - gh-issue-96559: Fixes the Windows launcher not using the compatible interpretation of default tags found in configuration files when no tag was passed to the command. What's New in Python 3.11.0 release candidate 1? ================================================ *Release date: 2022-08-05* Core and Builtins ----------------- - gh-issue-95150: Update code object hashing and equality to consider all debugging and exception handling tables. This fixes an issue where certain non-identical code objects could be "deduplicated" during compilation. - gh-issue-95355: ``_PyPegen_Parser_New`` now properly detects token memory allocation errors. Patch by Honglin Zhu. - gh-issue-90081: Run Python code in tracer/profiler function at full speed. Fixes slowdown in earlier versions of 3.11. - gh-issue-95324: Emit a warning in debug mode if an object does not call :c:func:`PyObject_GC_UnTrack` before deallocation. Patch by Pablo Galindo. - gh-issue-95185: Prevented crashes in the AST constructor when compiling some absurdly long expressions like ``"+0"*1000000``. :exc:`RecursionError` is now raised instead. Patch by Pablo Galindo - gh-issue-93351: :class:`ast.AST` node positions are now validated when provided to :func:`compile` and other related functions. If invalid positions are detected, a :exc:`ValueError` will be raised. - gh-issue-94938: Fix error detection in some builtin functions when keyword argument name is an instance of a str subclass with overloaded ``__eq__`` and ``__hash__``. Previously it could cause SystemError or other undesired behavior. Library ------- - gh-issue-95609: Update bundled pip to 22.2.2. - gh-issue-95289: Fix :class:`asyncio.TaskGroup` to propagate exception when :exc:`asyncio.CancelledError` was replaced with another exception by a context manger. Patch by Kumar Aditya and Guido van Rossum. - gh-issue-95339: Update bundled pip to 22.2.1. - gh-issue-95045: Fix GC crash when deallocating ``_lsprof.Profiler`` by untracking it before calling any callbacks. Patch by Kumar Aditya. - gh-issue-95097: Fix :func:`asyncio.run` for :class:`asyncio.Task` implementations without :meth:`~asyncio.Task.uncancel` method. Patch by Kumar Aditya. - gh-issue-93899: Fix check for existence of :data:`os.EFD_CLOEXEC`, :data:`os.EFD_NONBLOCK` and :data:`os.EFD_SEMAPHORE` flags on older kernel versions where these flags are not present. Patch by Kumar Aditya. - gh-issue-95166: Fix :meth:`concurrent.futures.Executor.map` to cancel the currently waiting on future on an error - e.g. TimeoutError or KeyboardInterrupt. - gh-issue-95109: Ensure that timeouts scheduled with :class:`asyncio.Timeout` that have already expired are delivered promptly. - gh-issue-91810: Suppress writing an XML declaration in open files in ``ElementTree.write()`` with ``encoding='unicode'`` and ``xml_declaration=None``. - gh-issue-91447: Fix findtext in the xml module to only give an empty string when the text attribute is set to None. Documentation ------------- - gh-issue-91207: Fix stylesheet not working in Windows CHM htmlhelp docs and add warning that they are deprecated. Contributed by C.A.M. Gerlach. - gh-issue-95451: Update library documentation with :ref:`availability information <wasm-availability>` on WebAssembly platforms ``wasm32-emscripten`` and ``wasm32-wasi``. - gh-issue-95415: Use consistent syntax for platform availability. The directive now supports a content body and emits a warning when it encounters an unknown platform. - gh-issue-86128: Document a limitation in ThreadPoolExecutor where its exit handler is executed before any handlers in atexit. Tests ----- - gh-issue-95573: :source:`Lib/test/test_asyncio/test_ssl.py` exposed a bug in the macOS kernel where intense concurrent load on non-blocking sockets occasionally causes :const:`errno.ENOBUFS` ("No buffer space available") to be emitted. FB11063974 filed with Apple, in the mean time as a workaround buffer size used in tests on macOS is decreased to avoid intermittent failures. Patch by Fantix King. - gh-issue-95280: Fix problem with ``test_ssl`` ``test_get_ciphers`` on systems that require perfect forward secrecy (PFS) ciphers. - gh-issue-94675: Add a regression test for :mod:`re` exponentional slowdown when using rjsmin. Build ----- - gh-issue-94801: Fix a regression in ``configure`` script that caused some header checks to ignore custom ``CPPFLAGS``. The regression was introduced in :gh:`94802`. - gh-issue-95145: wasm32-wasi builds no longer depend on WASIX's pthread stubs. Python now has its own stubbed pthread API. - gh-issue-95174: Python now detects missing ``dup`` function in WASI and wor