| Store | Cart

[Python-Dev] RFC: PEP 490 - Chain exceptions at C level

From: Victor Stinner <vict...@gmail.com>
Thu, 26 Mar 2015 13:30:18 +0100
Hi,

Here is the first draft of my PEP to chain automatically exceptions at
C level in Python 3.5. Plain text version of the PEP below. HTML
version of the PEP:

   https://www.python.org/dev/peps/pep-0490/

It has already an implementation, the pyerr_chain.patch file attached to
http://bugs.python.org/issue23763

The patch is very short.

We may modify more functions to hide the previous exception when the
new exception contains as much or more information. Keeping the
previous exception waste memory, make reference cycles more likely and
is useless when displaying the full exception chain.

I wrote a huge patch modifying a lot of functions to explicitly *not*
chain exceptions: pyerr_match_clear-2.patch of the issue #23763. I
wrote the patch to show where exceptions will be chained with
pyerr_chain.patch applied. But we might apply it if we want to keep
exactly the same behaviour between Python 3.4 and 3.5 for C modules
the stdlib.

pyerr_match_clear-2.patch adds also checks on the type of the previous
exception to not replace an "unexpected" exception. For example, we
can replace an expected TypeError with a ValueError (without chaining
them), but we should not raise a ValueError if we got a
KeyboardInterrupt, a MemoryError or other unexpected exceptions. But
such changes are unrelated to the PEP and may be done in a separated
issue.

The starting point of my PEP is my work on CPython done with the
pyfailmalloc module. This module injects MemoryError. I fixed a ton of
issues in the code handling errors in CPython (see the issue #18408).
I noticed that Python didn't complain when exceptions are lost. I
added many assertions to detect these mistakes. Recently, I modified
Python 3.5 to raise an exception even in release mode (issue #23571)
if a C function returns NULL without an exception set, or if a C
function returns a result with an exception set.


Victor


PEP: 490
Title: Chain exceptions at C level
Version: $Revision$
Last-Modified: $Date$
Author: Victor Stinner <vict...@gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 25-March-2015
Python-Version: 3.5


Abstract
========

Chain exceptions at C level, as already done at Python level.


Rationale
=========

Python 3 introduced a new killer feature: exceptions are chained by
default, PEP 3134.

Example::

    try:
        raise TypeError("err1")
    except TypeError:
        raise ValueError("err2")

Output::

    Traceback (most recent call last):
      File "test.py", line 2, in <module>
        raise TypeError("err1")
    TypeError: err1

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "test.py", line 4, in <module>
        raise ValueError("err2")
    ValueError: err2

Exceptions are chained by default in Python code, but not in
extensions written in C.

A new private ``_PyErr_ChainExceptions()`` function was introduced in
Python 3.4.3 and 3.5 to chain exceptions. Currently, it must be called
explicitly to chain exceptions and its usage is not trivial.

Example of ``_PyErr_ChainExceptions()`` usage from the ``zipimport``
module to chain the previous ``OSError`` to a new ``ZipImportError``
exception::

    PyObject *exc, *val, *tb;
    PyErr_Fetch(&exc, &val, &tb);
    PyErr_Format(ZipImportError, "can't open Zip file: %R", archive);
    _PyErr_ChainExceptions(exc, val, tb);

This PEP proposes to also chain exceptions automatically at C level to
stay consistent and give more information on failures to help
debugging. The previous example becomes simply::

    PyErr_Format(ZipImportError, "can't open Zip file: %R", archive);


Proposal
========

Modify PyErr_*() functions to chain exceptions
----------------------------------------------

Modify C functions raising exceptions of the Python C API to
automatically chain exceptions: modify ``PyErr_SetString()``,
``PyErr_Format()``, ``PyErr_SetNone()``, etc.


Modify functions to not chain exceptions
----------------------------------------

Keeping the previous exception is not always interesting when the new
exception contains information of the previous exception or even more
information, especially when the two exceptions have the same type.

Example of an useless exception chain with ``int(str)``::

    TypeError: a bytes-like object is required, not 'type'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: int() argument must be a string, a bytes-like object or
a number, not 'type'

The new ``TypeError`` exception contains more information than the
previous exception. The previous exception should be hidden.

The ``PyErr_Clear()`` function can be called to clear the current
exception before raising a new exception, to not chain the current
exception with a new exception.


Modify functions to chain exceptions
------------------------------------

Some functions save and then restore the current exception. If a new
exception is raised, the exception is currently displayed into
sys.stderr or ignored depending on the function.  Some of these
functions should be modified to chain exceptions instead.

Examples of function ignoring the new exception(s):

* ``ptrace_enter_call()``: ignore exception
* ``subprocess_fork_exec()``: ignore exception raised by enable_gc()
* ``t_bootstrap()`` of the ``_thread`` module: ignore exception raised
  by trying to display the bootstrap function to ``sys.stderr``
* ``PyDict_GetItem()``, ``_PyDict_GetItem_KnownHash()``: ignore
  exception raised by looking for a key in the dictionary
* ``_PyErr_TrySetFromCause()``: ignore exception
* ``PyFrame_LocalsToFast()``: ignore exception raised by
  ``dict_to_map()``
* ``_PyObject_Dump()``: ignore exception. ``_PyObject_Dump()`` is used
  to debug, to inspect a running process, it should not modify the
  Python state.
* ``Py_ReprLeave()``: ignore exception "because there is no way to
  report them"
* ``type_dealloc()``: ignore exception raised by
  ``remove_all_subclasses()``
* ``PyObject_ClearWeakRefs()``: ignore exception?
* ``call_exc_trace()``, ``call_trace_protected()``: ignore exception
* ``remove_importlib_frames()``: ignore exception
* ``do_mktuple()``, helper used by ``Py_BuildValue()`` for example:
  ignore exception?
* ``flush_io()``: ignore exception
* ``sys_write()``, ``sys_format()``: ignore exception
* ``_PyTraceback_Add()``: ignore exception
* ``PyTraceBack_Print()``: ignore exception

Examples of function displaying the new exception to ``sys.stderr``:

* ``atexit_callfuncs()``: display exceptions with
  ``PyErr_Display()`` and return the latest exception, the function
  calls multiple callbacks and only returns the latest exception
* ``sock_dealloc()``: log the ``ResourceWarning`` exception with
  ``PyErr_WriteUnraisable()``
* ``slot_tp_del()``: display exception with
  ``PyErr_WriteUnraisable()``
* ``_PyGen_Finalize()``: display ``gen_close()`` exception with
  ``PyErr_WriteUnraisable()``
* ``slot_tp_finalize()``: display exception raised by the
  ``__del__()`` method with ``PyErr_WriteUnraisable()``
* ``PyErr_GivenExceptionMatches()``: display exception raised by
  ``PyType_IsSubtype()`` with ``PyErr_WriteUnraisable()``


Backward compatibility
======================

A side effect of chaining exceptions is that exceptions store
traceback objects which store frame objects which store local
variables.  Local variables are kept alive by exceptions. A common
issue is a reference cycle between local variables and exceptions: an
exception is stored in a local variable and the frame indirectly
stored in the exception. The cycle only impacts applications storing
exceptions.

The reference cycle can now be fixed with the new
``traceback.TracebackException`` object introduced in Python 3.5. It
stores informations required to format a full textual traceback without
storing local variables.

The ``asyncio`` is impacted by the reference cycle issue. This module
is also maintained outside Python standard library to release a
version for Python 3.3.  ``traceback.TracebackException`` will maybe
be backported in a private ``asyncio`` module to fix reference cycle
issues.


Alternatives
============

No change
---------

A new private ``_PyErr_ChainExceptions()`` function is enough to chain
manually exceptions.

Exceptions will only be chained explicitly where it makes sense.


New helpers to chain exceptions
-------------------------------

Functions like ``PyErr_SetString()`` don't chain automatically
exceptions. To make the usage of ``_PyErr_ChainExceptions()`` easier,
new private functions are added:

* ``_PyErr_SetStringChain(exc_type, message)``
* ``_PyErr_FormatChain(exc_type, format, ...)``
* ``_PyErr_SetNoneChain(exc_type)``
* ``_PyErr_SetObjectChain(exc_type, exc_value)``

Helper functions to raise specific exceptions like
``_PyErr_SetKeyError(key)`` or ``PyErr_SetImportError(message, name,
path)`` don't chain exceptions.  The generic
``_PyErr_ChainExceptions(exc_type, exc_value, exc_tb)`` should be used
to chain exceptions with these helper functions.


Appendix
========

PEPs
----

* `PEP 3134 -- Exception Chaining and Embedded Tracebacks
  <https://www.python.org/dev/peps/pep-3134/>`_ (Python 3.0):
  new ``__context__`` and ``__cause__`` attributes for exceptions
* `PEP 415 - Implement context suppression with exception attributes
  <https://www.python.org/dev/peps/pep-0415/>`_ (Python 3.3):
  ``raise exc from None``
* `PEP 409 - Suppressing exception context
  <https://www.python.org/dev/peps/pep-0409/>`_
  (superseded by the PEP 415)


Python C API
------------

The header file ``Include/pyerror.h`` declares functions related to
exceptions.

Functions raising exceptions:

* ``PyErr_SetNone(exc_type)``
* ``PyErr_SetObject(exc_type, exc_value)``
* ``PyErr_SetString(exc_type, message)``
* ``PyErr_Format(exc, format, ...)``

Helpers to raise specific exceptions:

* ``PyErr_BadArgument()``
* ``PyErr_BadInternalCall()``
* ``PyErr_NoMemory()``
* ``PyErr_SetFromErrno(exc)``
* ``PyErr_SetFromWindowsErr(err)``
* ``PyErr_SetImportError(message, name, path)``
* ``_PyErr_SetKeyError(key)``
* ``_PyErr_TrySetFromCause(prefix_format, ...)``

Manage the current exception:

* ``PyErr_Clear()``: clear the current exception,
  like ``except: pass``
* ``PyErr_Fetch(exc_type, exc_value, exc_tb)``
* ``PyErr_Restore(exc_type, exc_value, exc_tb)``
* ``PyErr_GetExcInfo(exc_type, exc_value, exc_tb)``
* ``PyErr_SetExcInfo(exc_type, exc_value, exc_tb)``

Others function to handle exceptions:

* ``PyErr_ExceptionMatches(exc)``: check to implement
  ``except exc:  ...``
* ``PyErr_GivenExceptionMatches(exc1, exc2)``
* ``PyErr_NormalizeException(exc_type, exc_value, exc_tb)``
* ``_PyErr_ChainExceptions(exc_type, exc_value, exc_tb)``


Python Issues
-------------

Chain exceptions:

* `Issue #23763: Chain exceptions in C
  <http://bugs.python.org/issue23763>`_
* `Issue #23696: zipimport: chain ImportError to OSError
  <http://bugs.python.org/issue23696>`_
* `Issue #21715: Chaining exceptions at C level
  <http://bugs.python.org/issue21715>`_: added
  ``_PyErr_ChainExceptions()``
* `Issue #18488: sqlite: finalize() method of user function may be
  called with an exception set if a call to step() method failed
  <http://bugs.python.org/issue18488>`_
* `Issue #23781: Add private _PyErr_ReplaceException() in 2.7
  <http://bugs.python.org/issue23781>`_
* `Issue #23782: Leak in _PyTraceback_Add
  <http://bugs.python.org/issue23782>`_

Changes preventing to loose exceptions:

* `Issue #23571: Raise SystemError if a function returns a result with an
  exception set <http://bugs.python.org/issue23571>`_
* `Issue #18408: Fixes crashes found by pyfailmalloc
  <http://bugs.python.org/issue18408>`_


Copyright
=========

This document has been placed in the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:
_______________________________________________
Python-Dev mailing list
Pyth...@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/python-dev-ml%40activestate.com

Recent Messages in this Thread
Victor Stinner Mar 26, 2015 12:30 pm
Nick Coghlan Mar 30, 2015 01:11 pm
Messages in this thread