+++++++++++ Python News +++++++++++ What's New in Python 3.11.1 final? ================================== *Release date: 2022-12-06* Security -------- - gh-issue-100001: ``python -m http.server`` no longer allows terminal control characters sent within a garbage request to be printed to the stderr server log. This is done by changing the :mod:`http.server` :class:`BaseHTTPRequestHandler` ``.log_message`` method to replace control characters with a ``\xHH`` hex escape before printing. - gh-issue-87604: Avoid publishing list of active per-interpreter audit hooks via the :mod:`gc` module - gh-issue-98433: The IDNA codec decoder used on DNS hostnames by :mod:`socket` or :mod:`asyncio` related name resolution functions no longer involves a quadratic algorithm. This prevents a potential CPU denial of service if an out-of-spec excessive length hostname involving bidirectional characters were decoded. Some protocols such as :mod:`urllib` http ``3xx`` redirects potentially allow for an attacker to supply such a name. - gh-issue-98739: Update bundled libexpat to 2.5.0 - gh-issue-97612: Fix a shell code injection vulnerability in the ``get-remote-certificate.py`` example script. The script no longer uses a shell to run ``openssl`` commands. Issue reported and initial fix by Caleb Shortt. Patch by Victor Stinner. Core and Builtins ----------------- - gh-issue-99886: Fix a crash when an object which does not have a dictionary frees its instance values. - gh-issue-99891: Fix a bug in the tokenizer that could cause infinite recursion when showing syntax warnings that happen in the first line of the source. Patch by Pablo Galindo - gh-issue-99729: Fix an issue that could cause frames to be visible to Python code as they are being torn down, possibly leading to memory corruption or hard crashes of the interpreter. - gh-issue-99578: Fix a reference bug in :func:`_imp.create_builtin()` after the creation of the first sub-interpreter for modules ``builtins`` and ``sys``. Patch by Victor Stinner. - gh-issue-99581: Fixed a bug that was causing a buffer overflow if the tokenizer copies a line missing the newline caracter from a file that is as long as the available tokenizer buffer. Patch by Pablo galindo - gh-issue-99553: Fix bug where an :exc:`ExceptionGroup` subclass can wrap a :exc:`BaseException`. - gh-issue-99370: Fix zip path for venv created from a non-installed python on POSIX platforms. - gh-issue-99298: Fix an issue that could potentially cause incorrect error handling for some bytecode instructions. - gh-issue-99205: Fix an issue that prevented :c:type:`PyThreadState` and :c:type:`PyInterpreterState` memory from being freed properly. - gh-issue-99181: Fix failure in :keyword:`except* <except_star>` with unhashable exceptions. - gh-issue-99204: Fix calculation of :data:`sys._base_executable` when inside a POSIX virtual environment using copies of the python binary when the base installation does not provide the executable name used by the venv. Calculation will fall back to alternative names ("python<MAJOR>", "python<MAJOR>.<MINOR>"). - gh-issue-96055: Update :mod:`faulthandler` to emit an error message with the proper unexpected signal number. Patch by Dong-hee Na. - gh-issue-99153: Fix location of :exc:`SyntaxError` for a :keyword:`try` block with both :keyword:`except` and :keyword:`except* <except_star>`. - gh-issue-99103: Fix the error reporting positions of specialized traceback anchors when the source line contains Unicode characters. - gh-issue-98852: Fix subscription of type aliases containing bare generic types or types like :class:`~typing.TypeVar`: for example ``tuple[A, T][int]`` and ``tuple[TypeVar, T][int]``, where ``A`` is a generic type, and ``T`` is a type variable. - gh-issue-98925: Lower the recursion depth for marshal on WASI to support wasmtime 2.0/main. - gh-issue-98783: Fix multiple crashes in debug mode when ``str`` subclasses are used instead of ``str`` itself. - gh-issue-99257: Fix an issue where member descriptors (such as those for :attr:`~object.__slots__`) could behave incorrectly or crash instead of raising a :exc:`TypeError` when accessed via an instance of an invalid type. - gh-issue-98374: Suppress ImportError for invalid query for help() command. Patch by Dong-hee Na. - gh-issue-98415: Fix detection of MAC addresses for :mod:`uuid` on certain OSs. Patch by Chaim Sanders - gh-issue-92119: Print exception class name instead of its string representation when raising errors from :mod:`ctypes` calls. - gh-issue-96078: :func:`os.sched_yield` now release the GIL while calling sched_yield(2). Patch by Dong-hee Na. - gh-issue-93354: Fix an issue that could delay the specialization of :opcode:`PRECALL` instructions. - gh-issue-97943: Bugfix: :func:`PyFunction_GetAnnotations` should return a borrowed reference. It was returning a new reference. - gh-issue-97779: Ensure that all Python frame objects are backed by "complete" frames. - gh-issue-97591: Fixed a missing incref/decref pair in ``Exception.__setstate__()``. Patch by Ofey Chan. - gh-issue-94526: Fix the Python path configuration used to initialized :data:`sys.path` at Python startup. Paths are no longer encoded to UTF-8/strict to avoid encoding errors if it contains surrogate characters (bytes paths are decoded with the surrogateescape error handler). Patch by Victor Stinner. - gh-issue-95921: Fix overly-broad source position information for chained comparisons used as branching conditions. - gh-issue-96387: At Python exit, sometimes a thread holding the GIL can wait forever for a thread (usually a daemon thread) which requested to drop the GIL, whereas the thread already exited. To fix the race condition, the thread which requested the GIL drop now resets its request before exiting. Issue discovered and analyzed by Mingliang ZHAO. Patch by Victor Stinner. - gh-issue-96864: Fix a possible assertion failure, fatal error, or :exc:`SystemError` if a line tracing event raises an exception while opcode tracing is enabled. - gh-issue-96678: Fix undefined behaviour in C code of null pointer arithmetic. - gh-issue-96754: Make sure that all frame objects created are created from valid interpreter frames. Prevents the possibility of invalid frames in backtraces and signal handlers. - gh-issue-95196: Disable incorrect pickling of the C implemented classmethod descriptors. - gh-issue-96005: On WASI :data:`~errno.ENOTCAPABLE` is now mapped to :exc:`PermissionError`. The :mod:`errno` modules exposes the new error number. ``getpath.py`` now ignores :exc:`PermissionError` when it cannot open landmark files ``pybuilddir.txt`` and ``pyenv.cfg``. - gh-issue-93696: Allow :mod:`pdb` to locate source for frozen modules in the standard library. - bpo-31718: Raise :exc:`ValueError` instead of :exc:`SystemError` when methods of uninitialized :class:`io.IncrementalNewlineDecoder` objects are called. Patch by Oren Milman. - bpo-38031: Fix a possible assertion failure in :class:`io.FileIO` when the opener returns an invalid file descriptor. Library ------- - gh-issue-100001: Also \ escape \s in the http.server BaseHTTPRequestHandler.log_message so that it is technically possible to parse the line and reconstruct what the original data was. Without this a \xHH is ambiguious as to if it is a hex replacement we put in or the characters r"\x" came through in the original request line. - gh-issue-93453: :func:`asyncio.get_event_loop` now only emits a deprecation warning when a new event loop was created implicitly. It no longer emits a deprecation warning if the current event loop was set. - gh-issue-51524: Fix bug when calling trace.CoverageResults with valid infile. - gh-issue-99645: Fix a bug in handling class cleanups in :class:`unittest.TestCase`. Now ``addClassCleanup()`` uses separate lists for different ``TestCase`` subclasses, and ``doClassCleanups()`` only cleans up the particular class. - gh-issue-97001: Release the GIL when calling termios APIs to avoid blocking threads. - gh-issue-99341: Fix :func:`ast.increment_lineno` to also cover :class:`ast.TypeIgnore` when changing line numbers. - gh-issue-99418: Fix bug in :func:`urllib.parse.urlparse` that causes URL schemes that begin with a digit, a plus sign, or a minus sign to be parsed incorrectly. - gh-issue-99382: Check the number of arguments in substitution in user generics containing a :class:`~typing.TypeVarTuple` and one or more :class:`~typing.TypeVar`. - gh-issue-99379: Fix substitution of :class:`~typing.ParamSpec` followed by :class:`~typing.TypeVarTuple` in generic aliases. - gh-issue-99344: Fix substitution of :class:`~typing.TypeVarTuple` and :class:`~typing.ParamSpec` together in user generics. - gh-issue-74044: Fixed bug where :func:`inspect.signature` reported incorrect arguments for decorated methods. - gh-issue-99275: Fix ``SystemError`` in :mod:`ctypes` when exception was not set during ``__initsubclass__``. - gh-issue-99277: Remove older version of ``_SSLProtocolTransport.get_write_buffer_limits`` in :mod:`!asyncio.sslproto` - gh-issue-99248: fix negative numbers failing in verify() - gh-issue-99155: Fix :class:`statistics.NormalDist` pickle with ``0`` and ``1`` protocols. - gh-issue-93464: ``enum.auto()`` is now correctly activated when combined with other assignment values. E.g. ``ONE = auto(), 'some text'`` will now evaluate as ``(1, 'some text')``. - gh-issue-99134: Update the bundled copy of pip to version 22.3.1. - gh-issue-83004: Clean up refleak on failed module initialisation in :mod:`_zoneinfo` - gh-issue-83004: Clean up refleaks on failed module initialisation in in :mod:`_pickle` - gh-issue-83004: Clean up refleak on failed module initialisation in :mod:`_io`. - gh-issue-98897: Fix memory leak in :func:`math.dist` when both points don't have the same dimension. Patch by Kumar Aditya. - gh-issue-98706: [3.11] Applied changes from importlib_metadata `4.11.4 through 4.13 <https://importlib-metadata.readthedocs.io/en/latest/history.html#v4-13-0>`_, including compatibility and robustness fixes for ``Distribution`` objects without ``_normalized_name``, disallowing invalid inputs to ``Distribution.from_name``, and refined behaviors in ``PathDistribution._name_from_stem`` and ``PathDistribution._normalized_name``. - gh-issue-98793: Fix argument typechecks in :func:`!_overlapped.WSAConnect` and :func:`!_overlapped.Overlapped.WSASendTo` functions. - gh-issue-98744: Prevent crashing in :mod:`traceback` when retrieving the byte-offset for some source files that contain certain unicode characters. - gh-issue-98740: Fix internal error in the :mod:`re` module which in very rare circumstances prevented compilation of a regular expression containing a :ref:`conditional expression <re-conditional-expression>` without the "else" branch. - gh-issue-98703: Fix :meth:`asyncio.StreamWriter.drain` to call ``protocol.connection_lost`` callback only once on Windows. - gh-issue-98624: Add a mutex to unittest.mock.NonCallableMock to protect concurrent access to mock attributes. - gh-issue-89237: Fix hang on Windows in ``subprocess.wait_closed()`` in :mod:`asyncio` with :class:`~asyncio.ProactorEventLoop`. Patch by Kumar Aditya. - gh-issue-98458: Fix infinite loop in unittest when a self-referencing chained exception is raised - gh-issue-97928: :meth:`tkinter.Text.count` raises now an exception for options starting with "-" instead of silently ignoring them. - gh-issue-97966: On ``uname_result``, restored expectation that ``_fields`` and ``_asdict`` would include all six properties including ``processor``. - gh-issue-98307: A :meth:`~logging.handlers.SysLogHandler.createSocket` method was added to :class:`~logging.handlers.SysLogHandler`. - gh-issue-96035: Fix bug in :func:`urllib.parse.urlparse` that causes certain port numbers containing whitespace, underscores, plus and minus signs, or non-ASCII digits to be incorrectly accepted. - gh-issue-98251: Allow :mod:`venv` to pass along :envvar:`PYTHON*` variables to ``ensurepip`` and ``pip`` when they do not impact path resolution - gh-issue-98178: On macOS, fix a crash in :func:`syslog.syslog` in multi-threaded applications. On macOS, the libc ``syslog()`` function is not thread-safe, so :func:`syslog.syslog` no longer releases the GIL to call it. Patch by Victor Stinner. - gh-issue-96151: Allow ``BUILTINS`` to be a valid field name for frozen dataclasses. - gh-issue-87730: Wrap network errors consistently in urllib FTP support, so the test suite doesn't fail when a network is available but the public internet is not reachable. - gh-issue-98086: Make sure ``patch.dict()`` can be applied on async functions. - gh-issue-90985: Earlier in 3.11 we deprecated ``asyncio.Task.cancel("message")``. We realized we were too harsh, and have undeprecated it. - gh-issue-97837: Change deprecate warning message in :mod:`unittest` from ``It is deprecated to return a value!=None`` to ``It is deprecated to return a value that is not None from a test case`` - gh-issue-97825: Fixes :exc:`AttributeError` when :meth:`subprocess.check_output` is used with argument ``input=None`` and either of the arguments *encoding* or *errors* are used. - gh-issue-82836: Fix :attr:`~ipaddress.IPv4Address.is_private` properties in the :mod:`ipaddress` module. Previously non-private networks (0.0.0.0/0) would return True from this method; now they correctly return False. - gh-issue-96827: Avoid spurious tracebacks from :mod:`asyncio` when default executor cleanup is delayed until after the event loop is closed (e.g. as the result of a keyboard interrupt). - gh-issue-97592: Avoid a crash in the C version of :meth:`asyncio.Future.remove_done_callback` when an evil argument is passed. - gh-issue-97639: Remove ``tokenize.NL`` check from :mod:`tabnanny`. - gh-issue-73588: Fix generation of the default name of :class:`tkinter.Checkbutton`. Previously, checkbuttons in different parent widgets could have the same short name and share the same state if arguments "name" and "variable" are not specified. Now they are globally unique. - gh-issue-97005: Update bundled libexpat to 2.4.9 - gh-issue-85760: Fix race condition in :mod:`asyncio` where :meth:`~asyncio.SubprocessProtocol.process_exited` called before the :meth:`~asyncio.SubprocessProtocol.pipe_data_received` leading to inconsistent output. Patch by Kumar Aditya. - gh-issue-96819: Fixed check in :mod:`multiprocessing.resource_tracker` that guarantees that the length of a write to a pipe is not greater than ``PIPE_BUF``. - gh-issue-96741: Corrected type annotation for dataclass attribute ``pstats.FunctionProfile.ncalls`` to be ``str``. - gh-issue-95987: Fix ``repr`` of ``Any`` subclasses. - gh-issue-96388: Work around missing socket functions in :class:`~socket.socket`'s ``__repr__``. - gh-issue-96073: In :mod:`inspect`, fix overeager replacement of "``typing.``" in formatting annotations. - gh-issue-96192: Fix handling of ``bytes`` :term:`path-like objects <path-like object>` in :func:`os.ismount()`. - gh-issue-96052: Fix handling compiler warnings (SyntaxWarning and DeprecationWarning) in :func:`codeop.compile_command` when checking for incomplete input. Previously it emitted warnings and raised a SyntaxError. Now it always returns ``None`` for incomplete input without emitting any warnings. - gh-issue-88863: To avoid apparent memory leaks when :func:`asyncio.open_connection` raises, break reference cycles generated by local exception and future instances (which has exception instance as its member var). Patch by Dong Uk, Kang. - gh-issue-91212: Fixed flickering of the turtle window when the trac