2011
07.08

Python and Threading (Also, Maya)

Category: Maya, Python / Tag: ,  / Add Comment

So, we noticed some odd behavior with several threaded scripts at work: They ran slower than the initial single threaded version.. Off to the Maya docs!

Per the Maya docs on threading and python:

“One important difference between Python threads and C++-based threads is that python threads do not run concurrently since the Python interpreter itself is not currently threadsafe. For this reason, they are not useful for data parallel applications. However, they may be useful, for example, where polling of resources is done that might otherwise have to wait for timeouts.”

After doing a bit of research I thought I would share some of what I found on python and Threading, this sparked a few conversations at work about the usefulness and specific use cases of threads.

Short version:

  • Maya Python and standalone Python handle threads differently: False
  • Python instructions cannot run in parallel: True
  • Python threads are not “true” threads: False
  • Threads provide no speed benefit or parallelism: False
  • Should you use threads speed up potentially parallel operations: Sometimes

Python’s natively limited via the Global Interpreter Lock (GIL) to prevent concurrent Python byte code instructions, this prevents separate threads from simultaneously modifying the same memory space in order to prevent  memory access issues. If one thread were able to delete a shared object while another thread tried to append to it, the result would be a crash without the GIL. On the internal implementation side it get’s even more important as internal data like weakref’s and reference counts could become out of sync.

The silver lining to this limitation is that it only effect’s pure python bytecode, anything happening via a compiled c extension or a linked C++ library like PyQt will generally try to release the GIL while the compiled code is running, as that code can’t modify the internal Python interpreter state anyways (C++ code couldn’t delete your python object for example). Most of the major and intensive operations that you would want to do in a thread are already careful to release the GIL when performing a blocking operation: file read/write, time.sleep, sql select/execute, xml parse, etc..

Per the python documentation on the GIL:

” The GIL is controversial because it prevents multithreaded CPython programs from taking full advantage of multiprocessor systems in certain situations. Note that potentially blocking or long-running operations, such as I/O, image processing, and NumPy number crunching, happen outside the GIL. Therefore it is only in multithreaded programs that spend a lot of time inside the GIL, interpreting CPython bytecode, that the GIL becomes a bottleneck.”

PyQt and the GIL:

“when the threaded code is calling long running c/c++ functions, such as a database select, it will release the GIL until the call completes.  This is all done automatically within python and PyQt.  This is “real” multithreading.”

As for the original issue raised by the Maya api docs, the real issue isn’t as straightforward as the docs would lead you to believe: Yes, using threads to number crunch in pure python threads is pointless, they will not run in parallel and will yeild no speed increase. However, file operations, numpy, sql access, and many other functions can be used safely within Maya and yield speed increases and truly parallel work. The docs seem to be primarily focused on parallelism in the context of deformers or number crunching/simulation plugins, in which case that computation should be offloaded to a custom c extension or something like numpy.

References:

  1. Seth said: 2011.07.18 21:17

    There are some great pycon talks about this on blip.tv i’d recommend checking out as well. Good stuff!

Your Comment

You must be logged in to post a comment.