Python Multitasking – MultiThreading and MultiProcessing

Modern operating systems allow a user to run several applications simultaneously, so that some tasks can be run in the background while the user continues with other work in the foreground. The user can also run multiple copies of the same program at the same time.

To create a task we can use process or thread. Process has its private resources including memory mapping, files and other os objects. Multiple threads can run on the same process and share all its resources but if one thread fail it will kill all other threads in its process.


Python has many packages to handle multi tasking, in this post i will cover some


Unix/Linux/OS X specific (i.e. all but windows). The function creates a child process that start running after the fork return. We call fork once but it returns twice on the parent and on the child. Consider the following code:

If you run it , you will see both prints:

On fork call, the OS creates a child process and fork returns twice – on the parent the return value is the child process id and on the child fork returns 0. The point behind this weird behaviour is the Join-Fork parallel pattern – you run a code and want to use another processor for a complex task – fork a child, divide the work and join again:

Before the fork call and after the if statement only one process exists , The child does its task and finish returning a value to the parent, If the parent finish before the child it will wait for him

It is important to call wait otherwise the child remains zombie


Runs a command in the shell (bash, cmd, etc)

sometimes you only need to run another program or command for example you want to do something that easily done with a command like format


Run a process and read its output as a file for example:


The subprocess package

An attempt to replace os specific function with platform independent. You can run a process and open a pipe in any os for example:

to communicate with the output use:


The multiprocessing package

Another useful package for process creation and other methods.

The multiprocessing module is suitable for sharing data or tasks between processor cores. It does not use threading, but processes instead. Processes are inherently more “expensive” that threads, so they are not worth using for trivial data sets or tasks


Working with Threads

Threads run on the same process address space – it is easy to share data between them but if one thread fails all other threads in the same process killed. To create a thread we use the threading package. We write a class derived from Thread and declare the function run. To create a thread we need to create an object from the class and call start method:


You can also define the thread function as global (without class):

Synchronization Objects

Accessing resources needs synchronization. We can find the following objects in threading package:

  • Condition variables – similar to linux posix condition variables
  • Events – similar to windows events
  • Lock – similar to mutex
  • Semaphore

Lock Example:

Inter Process Communication (IPC)

Threads share the same address space so its easy to send data from one thread to another but processes live in a different address spaces and thats why we need an IPC object. You can use pipe, fifo, message queue and more

Message Queue Example:

In the following example we create 2 processes and use a queue for communication. One process send message to the queue and the other receive:


You can find more methods for creating threads and processes, using synchronization objects and IPC




Leave a Reply

Your email address will not be published. Required fields are marked *