Python has many packages to handle multi tasking, in this post i will cover some

os.fork

Unix/Linux/OS X specific (i.e. all but windows). The function creates a child process that start running after the fork return. We call fork once but it returns twice on the parent and on the child. Consider the following code:

import os

x=os.fork()
if x:
    print('hello parent')
else:
    print('hello child')

If you run it , you will see both prints:

hello parent
hello child

On fork call, the OS creates a child process and fork returns twice – on the parent the return value is the child process id and on the child fork returns 0. The point behind this weird behaviour is the Join-Fork parallel pattern – you run a code and want to use another processor for a complex task – fork a child, divide the work and join again:

import os

ret=0
# we need to do something complex so lets create a child
x=os.fork()
if x:
    print('do half work by parent')
    ret = os.wait()
else:
    print('do half work by child')
    os._exit(10)

print("only parent here res=" + str(os.WEXITSTATUS(ret[1])))

Before the fork call and after the if statement only one process exists , The child does its task and finish returning a value to the parent, If the parent finish before the child it will wait for him

It is important to call wait otherwise the child remains zombie

os.system

Runs a command in the shell (bash, cmd, etc)

sometimes you only need to run another program or command for example you want to do something that easily done with a command like format

import os

status = os.system('script2.py')
print ("exit status",status)

os.popen

Run a process and read its output as a file for example:

import os

for line in os.popen('ps aux').readlines():
    print(line)

The subprocess package

An attempt to replace os specific function with platform independent. You can run a process and open a pipe in any os for example:

import subprocess

subprocess.run('script2.py')
pr = subprocess.Popen('ls')
pr.wait();

to communicate with the output use:

import subprocess
import sys

proc = subprocess.Popen([sys.executable, 'myapp.py', 'data'],
                        stdout = subprocess.PIPE,
                        stderr = subprocess.PIPE)
(output, error) = proc.communicate()
if error != None:
    print("error:", error.decode())
print("output:", output.decode())

The multiprocessing package

Another useful package for process creation and other methods.

import multiprocessing
def fn(a,b):
    print ("vals=",a,b)

proc1 = multiprocessing.Process(target=fn, args=(10,20))
proc2 = multiprocessing.Process(target=fn, args=(10,20))
proc1.start()
proc2.start()
proc1.join()
proc2.join()

Working with Threads

Threads run on the same process address space – it is easy to share data between them but if one thread fails all other threads in the same process killed. To create a thread we use the threading package. We write a class derived from Thread and declare the function run. To create a thread we need to create an object from the class and call start method:

import threading
import time
class MyThread (threading.Thread):
    def run (self):
        for i in range(5):
            print ("In thread", self.name)
            time.sleep(2)

th1 = MyThread()
th2 = MyThread()
th1.setName("t1")
th2.setName("t2")
th1.start()
th2.start()
print ("From main")
th1.join()
th2.join()

output:

In thread t1
In thread t2
From main
In thread t1
In thread t2
In thread t1
In thread t2
....

You can also define the thread function as global (without class):

from threading import Thread
import time

def myfunc(*args):
    print "From thread", args
    time.sleep(5)
   
tid1 = Thread(target=myfunc, args='T1')
tid2 = Thread(target=myfunc, args='T2')
tid1.start()
tid2.start()
print "From main"
tid1.join()
tid2.join()

Synchronization Objects

Accessing resources needs synchronization. We can find the following objects in threading package:

• Condition variables – similar to linux posix condition variables
• Events – similar to windows events
• Lock – similar to mutex
• Semaphore

Lock Example:

import threading
import time

lc = threading.Lock()


def myfunc1(*args):
    global lc
    lc.acquire()
    print("From thread", args)
    lc.release()

def myfunc2(*args):
    global lc
    lc.acquire()
    print("From thread", args)
    lc.release()


tid1 = threading.Thread(target=myfunc1, args='T1')
tid2 = threading.Thread(target=myfunc2, args='T2')
tid1.start()
tid2.start()
print("From main")
tid1.join()
tid2.join()

Inter Process Communication (IPC)

Threads share the same address space so its easy to send data from one thread to another but processes live in a different address spaces and thats why we need an IPC object. You can use pipe, fifo, message queue and more

Message Queue Example:

In the following example we create 2 processes and use a queue for communication. One process send message to the queue and the other receive:

import multiprocessing
import time

def fn1(q):
    while True:
        x=q.get()
        print ("val=",x)

def fn2(q):
    while True:
        time.sleep(2)
        q.put("hello")

queue = multiprocessing.Queue()
proc1 = multiprocessing.Process(target=fn1, args=(queue,))
proc2 = multiprocessing.Process(target=fn2, args=(queue,))
proc1.start()
proc2.start()

You can find more methods for creating threads and processes, using synchronization objects and IPC