Python Multitasking – MultiThreading and MultiProcessing

Modern operating systems allow a user to run several applications simultaneously, so that some tasks can be run in the background while the user continues with other work in the foreground. The user can also run multiple copies of the same program at the same time.

To create a task we can use process or thread. Process has its private resources including memory mapping, files and other os objects. Multiple threads can run on the same process and share all its resources but if one thread fail it will kill all other threads in its process.

 

Python has many packages to handle multi tasking, in this post i will cover some

os.fork

Unix/Linux/OS X specific (i.e. all but windows). The function creates a child process that start running after the fork return. We call fork once but it returns twice on the parent and on the child. Consider the following code:

import os

x=os.fork()
if x:
    print('hello parent')
else:
    print('hello child')

If you run it , you will see both prints:

hello parent
hello child

On fork call, the OS creates a child process and fork returns twice – on the parent the return value is the child process id and on the child fork returns 0. The point behind this weird behaviour is the Join-Fork parallel pattern – you run a code and want to use another processor for a complex task – fork a child, divide the work and join again:

import os

ret=0
# we need to do something complex so lets create a child
x=os.fork()
if x:
    print('do half work by parent')
    ret = os.wait()
else:
    print('do half work by child')
    os._exit(10)

print("only parent here res=" + str(os.WEXITSTATUS(ret[1])))

Before the fork call and after the if statement only one process exists , The child does its task and finish returning a value to the parent, If the parent finish before the child it will wait for him

It is important to call wait otherwise the child remains zombie

os.system

Runs a command in the shell (bash, cmd, etc)

sometimes you only need to run another program or command for example you want to do something that easily done with a command like format

import os

status = os.system('script2.py')
print ("exit status",status)

os.popen

Run a process and read its output as a file for example:

import os

for line in os.popen('ps aux').readlines():
    print(line)

 

The subprocess package

An attempt to replace os specific function with platform independent. You can run a process and open a pipe in any os for example:

import subprocess

subprocess.run('script2.py')
pr = subprocess.Popen('ls')
pr.wait();

to communicate with the output use:

import subprocess
import sys

proc = subprocess.Popen([sys.executable, 'myapp.py', 'data'],
                        stdout = subprocess.PIPE,
                        stderr = subprocess.PIPE)
(output, error) = proc.communicate()
if error != None:
    print("error:", error.decode())
print("output:", output.decode())

 

The multiprocessing package

Another useful package for process creation and other methods.

The multiprocessing module is suitable for sharing data or tasks between processor cores. It does not use threading, but processes instead. Processes are inherently more “expensive” that threads, so they are not worth using for trivial data sets or tasks

import multiprocessing
def fn(a,b):
    print ("vals=",a,b)

proc1 = multiprocessing.Process(target=fn, args=(10,20))
proc2 = multiprocessing.Process(target=fn, args=(10,20))
proc1.start()
proc2.start()
proc1.join()
proc2.join()

 

Working with Threads

Threads run on the same process address space – it is easy to share data between them but if one thread fails all other threads in the same process killed. To create a thread we use the threading package. We write a class derived from Thread and declare the function run. To create a thread we need to create an object from the class and call start method:

import threading
import time
class MyThread (threading.Thread):
    def run (self):
        for i in range(5):
            print ("In thread", self.name)
            time.sleep(2)

th1 = MyThread()
th2 = MyThread()
th1.setName("t1")
th2.setName("t2")
th1.start()
th2.start()
print ("From main")
th1.join()
th2.join()

output:

In thread t1
In thread t2
From main
In thread t1
In thread t2
In thread t1
In thread t2
....

You can also define the thread function as global (without class):

from threading import Thread
import time

def myfunc(*args):
    print "From thread", args
    time.sleep(5)
   
tid1 = Thread(target=myfunc, args='T1')
tid2 = Thread(target=myfunc, args='T2')
tid1.start()
tid2.start()
print "From main"
tid1.join()
tid2.join()

Synchronization Objects

Accessing resources needs synchronization. We can find the following objects in threading package:

  • Condition variables – similar to linux posix condition variables
  • Events – similar to windows events
  • Lock – similar to mutex
  • Semaphore

Lock Example:

import threading
import time

lc = threading.Lock()


def myfunc1(*args):
    global lc
    lc.acquire()
    print("From thread", args)
    lc.release()

def myfunc2(*args):
    global lc
    lc.acquire()
    print("From thread", args)
    lc.release()


tid1 = threading.Thread(target=myfunc1, args='T1')
tid2 = threading.Thread(target=myfunc2, args='T2')
tid1.start()
tid2.start()
print("From main")
tid1.join()
tid2.join()

Inter Process Communication (IPC)

Threads share the same address space so its easy to send data from one thread to another but processes live in a different address spaces and thats why we need an IPC object. You can use pipe, fifo, message queue and more

Message Queue Example:

In the following example we create 2 processes and use a queue for communication. One process send message to the queue and the other receive:

import multiprocessing
import time

def fn1(q):
    while True:
        x=q.get()
        print ("val=",x)

def fn2(q):
    while True:
        time.sleep(2)
        q.put("hello")

queue = multiprocessing.Queue()
proc1 = multiprocessing.Process(target=fn1, args=(queue,))
proc2 = multiprocessing.Process(target=fn2, args=(queue,))
proc1.start()
proc2.start()

 

You can find more methods for creating threads and processes, using synchronization objects and IPC

 

 

Tagged

3 thoughts on “Python Multitasking – MultiThreading and MultiProcessing

  1. Multiple threading are useful create program small size its use full to workout.

  2. Many thanks, very useful post!

  3. Hi friends, its fantastic post on the topic of teachingand fully defined, keep
    it up all the time.

Comments are closed.