There are three ways to implement python multi-tasking. They are multi-processing, multi-threading and coroutine. Multi-processing is the most cumbersome way among these three and coroutine is the lightest.

Table of Contents

Multi-threading

  • Use threading.Thread method to create threads
python 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
 
import threading

def test1(n):
  for i in range(n):
    print("----test1---%d---"%i)
    
def test2(n):
  for i in range(n):
    print("----test2---%d---"%i)
    
def main():
  t1 = threading.Thread(target=test1, args=(5,))
  t2 = threading.Thread(target=test2, args=(5,))
  
  t1.start()
  t2.start()

if __name__ == "__main__":
  main()
  • Create thread by using a class to inherit from threading.Thread
python 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
 
import threading
import time

class MyThread(threading.Thread):
  def __init__(self,n):
    super().__init__()
    self.n = n
    
  def run(self):
    for i in range(self.n):
      time.sleep(1)
      msg = "I'm "+self.name+' @ '+str(i)
      print(msg)

if __name__ == "__main__":
  t = MyThread(5)
  t.start()

Multi-processing

  • The usage of multiprocessing module is very similar to the multithreading module.
python 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
 
import multiprocessing

def test1(n):
  for i in range(n):
    print("----test1---%d---"%i)
    
def test2(n):
  for i in range(n):
    print("----test2---%d---"%i)
    
def main():
  p1 = multiprocessing.Process(target=test1, args=(5,))
  p2 = multiprocessing.Process(target=test2, args=(5,))
  
  p1.start()
  p2.start()

if __name__ == "__main__":
  main()

Coroutine

Coroutine originated from generator. There is always one thread that is running. Coroutine makes one thread switch between different functions so that it looks like multi-tasking.

  • Here is an example that a generator can switch from one function to another
python 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
 
def task_1():
  while True:
    print("---1---")
    time.sleep(0.1)
    yield
    
def task_2():
  while True:
    print("---2---")
    time.sleep(0.1)
    yield

def main():
  t1 = task_1()
  t2 = task_2()
  while True:
    next(t1)
    next(t2)
    
if __name__ == "__main__":
  main()
  • Generator can make two functions switch from one to another. However it is too complicated to write like this. In python, we have two packages, greenlet and gevent to implement coroutine. No need to write any yield statement in your function and then manage them in the main thread.
python 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
 
from greenlet import greenlet
import time

def test1():
  while True:
    print("---A---")
    gr2.switch()
    time.sleep(0.5)
    
def test2():
  while True:
    print("---B---")
    gr1.switch()
    time.sleep(0.5)
    
gr1 = greenlet(test1)
gr2 = greenlet(test2)

#switch to gr1 to run
gr1.switch()

  • gevent is a wrapper around greenlet. When you use greenlet, you still need to manually switch by explicit calling greenlet.switch() method.

  • Comparatively, gevent will switch whenever it encountered a blocking function, such as time.sleep() and various of socket functions. But the sleep function must use the one in gevent package.

  • Luckily we have monkey.patch_all() that will change the blocking functions in other packages to the blocking functions in gevent package. Here is an example:

python 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
 
import gevent
import time
from gevent import monkey

#use this to avoid manual change "time.sleep(0.5)" to "gevent.sleep(0.5)"
monkey.patch_all() 

def f1(n):
  for i in range(n):
    print(gevent.getcurrent(),i)
    time.sleep(0.5)

def f2(n):
  for i in range(n):
    print(gevent.getcurrent(),i)
    time.sleep(0.5)

def f3(n):
  for i in range(n):
    print(gevent.getcurrent(),i)
    time.sleep(0.5)

print("-----1-----")
g1 = gevent.spawn(f1,5)
print("-----2-----")
g2 = gevent.spawn(f2,5)
print("-----3-----")
g3 = gevent.spawn(f3,5)
print("-----4-----")
g1.join()
g2.join()
g3.join()