Python Thread based parallelism: excepthook














































Python Thread based parallelism: excepthook



Python threading.excepthook() Method


threading.excepthook() Add a new threading.excepthook() function which handles
uncaught  Thread.run().It can be overridden to control how uncaught exceptions
are handled. 
Threading.ExceptHookArgs is not documented on purpose :it
should not be used directly.

The args argument has the following attributes:

  • exc_type: Exception type.

  • exc_value: Exception value, can be None.

  • exc_traceback: Exception traceback, can be None.

  • thread: Thread which raised the exception, can be None.

If exc_type is SystemExit, the exception is silently ignored. Otherwise, the exception is printed out on sys.stderr.

If this function raises an exception, sys.excepthook() is called to handle it.

threading.excepthook() can be overridden to control how uncaught exceptions raised by Thread.run() are handled.

Storing exc_value using a custom hook can create a reference cycle. It should be cleared explicitly to break the reference cycle when the exception is no longer needed.

Storing thread using a custom hook can resurrect it if it is set to an object which is being finalized. Avoid storing thread after the custom hook completes to avoid resurrecting objects.


{ sys.excepthook(type, value, traceback) .This function prints out a given traceback   and exception to sys.stderr.}

You might think of executing everything in a try-catch to make sure any uncaught exception gets logged in the catch. This is better than nothing as it will capture exceptions on the main thread (more about threading later) but it requires modifying existing code. For example you could modify your code to run it in a big exception block by doing something like:

import sys

try:
# all your code that gets executed here
except:
print("Unhandled exception:", sys.exc_info()[0])
raise

One really important thing to note with this approach is that you must re-raise the exception via raise here otherwise you have altered the program substantially. Silently catching exceptions leads to all sorts of nasty bugs that can cause a large amount of damage.

(Note that there are some exceptions that you want to not catch such as a KeyboardInterrupt when people are running your code from the REPL. We will account for this in the next step)

A better approach that doesn't require modifying existing code is to make use of the sys module which provides excepthook to allow you to attach a handler for any unhandled exception.

import sys
import logging

logger = logging.getLogger(__name__)
logging.basicConfig(filename='example.log', filemode='w', level=logging.DEBUG)

def handle_unhandled_exception(exc_type, exc_value, exc_traceback):
"""Handler for unhandled exceptions that will write to the logs"""
if issubclass(exc_type, KeyboardInterrupt):
# call the default excepthook saved at __excepthook__
sys.__excepthook__(exc_type, exc_value, exc_traceback)
return
logger.critical("Unhandled exception", exc_info=(exc_type, exc_value, exc_traceback))

sys.excepthook = handle_unhandled_exception


This is something we can use with logging to make sure any unhandled exception will end up in our log files without us having to modify the source of other modules. Unfortunately this won't work well with threading or multiprocessing without a few modifications that we will explain shortly. Take for example the situation where some code runs on another thread such as a GUI thread:


Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/home/janis/python_examples/excepthook_logging_example/threaded_exception.py", line 9, in run
raise ExceptionFromThread(self.error_message)
threaded_exception.ExceptionFromThread: Runs on thread


We get this console output without anything being written to the logs. This clearly isn't what we want with a global exception handler and there's a bug report about this on Python's issue tracker. The reason this happens is because because both threading and multiprocessing have their own unhandled exception machinery that is a bit customized. An unhandled exception on a thread gets handled by the threading code so no unhandled exception exists at the top level. So if an exception gets raised in a thread but not handled sys.excepthook never gets a chance to be called as the thread will handle that first. This is the same machinery that will print out what thread the exception occurs on (the "Exception in thread Thread-1:" bit). We can patch this behavior as follows:

def patch_threading_excepthook():
"""Installs our exception handler into the threading modules Thread object
Inspired by https://bugs.python.org/issue1230540
"""

old_init = threading.Thread.__init__
def new_init(self, *args, **kwargs):
old_init(self, *args, **kwargs)
old_run = self.run
def run_with_our_excepthook(*args, **kwargs):
try:
old_run(*args, **kwargs)
except (KeyboardInterrupt, SystemExit):
raise
except:
sys.excepthook(*sys.exc_info())
self.run = run_with_our_excepthook
threading.Thread.__init__ = new_init

patch_threading_excepthook()


If maintaining the threading information matters we can modify how our excepthook is called in the patch code to as follows:

sys.excepthook(*sys.exc_info(), thread_identifier=threading.get_ident())


This assumes a slight modification to our excepthook handler to take a default argument that will handle the threading identifier passed into it.

Now that we have done this the excepthook approach for logging unhandled exceptions works great. Note that we can't get this behavior with the naive try-catch-everything approach since any exception that occurs on a different thread cannot be logged in the main thread of execution.

So that's it, you should be able to log all unhandled exceptions now with your function specified in sys.excepthook even if they are running in other threads (unless you have code similar to what Thread does by default handling it before you get a chance to first).


In next article, we will read about threading.get_ident()

Link -javascript:nicTemp();


More Articles of Diksha Kumari:

Name Views Likes
Python : py_compile | Generate byte code 308 0
Python : py_compiler |Convert py to pyc 1156 0
Python : py_compile |Compile python sources file 287 0
Python :How to lock Critical Sections(Thread based parallelism) 249 0
Python : Locking without Deadlocks(Thread based Parallelism ) 180 0
Python : Communicating Between Threads-2(Thread based parallelism) 389 0
Python : Communicating Between Threads -1(Thread based parallelism) 176 0
Python Start and Stop a thread (Thread based parallelism) 165 0
Python: Check if a Thread has started (thread based parallelism) 144 0
Python Thread-based parallelism :main_thread() 229 0
Python Lock Class | acquire() Method(thread based parallelism) 294 0
Python Thread-based parallelism : get_native_id() 460 0
Python Thread-based parellelism: active_count() 269 1
Python Lock Class | lock() Method(thread based parallelism) 246 0
Python Thread-based parallelism : stack_size([size]) 331 1
Python Thread Local Data(thread based parallelism 265 1
Python Semaphore Objects (thread based parallelism) 264 0
Python Event class - set() method | Thread based parallelism 252 1
Python Condition object (thread based parallelism) 411 0
Python Thread based parallelism : Run() 228 0
Python Thread-based parallelism : current_thread() 253 1
Python Condition object - acquire()(thread based parallelism) 190 0
Python Thread-based parallelism : TIMEOUT_MAX () 423 0
Python Condition object - notify_all() ( thread based parallelism ) 192 0
Python Thread-based parallelism :.settrace(func) 308 0
Python difference lock and RLock object : Thread based parallelism 1082 0
Python Event class - clear() method | Thread based parallelism 188 0
Python Thread object -start() (thread based parallelism) 335 0
Python RLock Class - acquire() Method (thread based parallelism) 226 0
Python Thread-based parallelism : enumerate() 246 0
Python Thread based parallelism: get_ident() 302 1
Python Thread-based parallelism : setprofile(func) 215 0
Python RLock Class - release() Method (thread based parallelism) 194 0
Python Barrier object | Thread based parallelism 228 0
Python Condition object - notify () ( thread based parallelism ) 217 0
Python Event class - is_set() method | Thread based parallelism 173 0
Python Thread based parallelism : join() 317 0
Python thread based parallelism : set.name() 200 0
Python Event class - wait() method | Thread based parallelism 243 0
PYTHON Thread based parallism 292 1
Python Barrier method | Thread based parallelism 228 0
Python Condition object - release() ( thread based parallelism ) 184 0
Python Thread based parallelism: excepthook 1487 1
Python thread object : is_alive() (thread based parallelism) 996 0
Python Condition object - wait() ( thread based parallelism ) 215 0
Python Lock Class | release() Method(thread based parallelism) 254 0
Python Thread object (thread based parallelism) 259 0
Python Timer class - start() method | Thread based parallelism 175 1
Python thread based parallelism : get.name() 322 1
Python Timer class - cancel() method | Thread based parallelism 206 1
Python thread object : Deamon() (thread based parallelism) 241 0

Comments

  • sanjot
    25-May-2021 10:13:03 PM
    This articled help me in clearing many doubts.