2018-11-21 回答

python提供了非常好用的多进程包multiprocessing，你只需要定义一个函数，python会替你完成其他所有事情。

借助这个包，可以轻松完成从单进程到并发执行的转换。

1、新建单一进程

如果我们新建少量进程，可以如下：

import multiprocessing

import time

def func(msg):

for i in xrange(3):

print msg

time.sleep(1)

if __name__ == "__main__":

p = multiprocessing.process(target=func, args=("hello", ))

p.start()

p.join()

print "sub-process done."12345678910111213

2、使用进程池

是的，你没有看错，不是线程池。它可以让你跑满多核cpu，而且使用方法非常简单。

注意要用apply_async，如果落下async，就变成阻塞版本了。

processes=4是最多并发进程数量。

import multiprocessing

import time

def func(msg):

for i in xrange(3):

print msg

time.sleep(1)

if __name__ == "__main__":

pool = multiprocessing.pool(processes=4)

for i in xrange(10):

msg = "hello %d" %(i)

pool.apply_async(func, (msg, ))

pool.close()

pool.join()

print "sub-process(es) done."12345678910111213141516

3、使用pool，并需要关注结果

更多的时候，我们不仅需要多进程执行，还需要关注每个进程的执行结果，如下：

import multiprocessing

import time

def func(msg):

for i in xrange(3):

print msg

time.sleep(1)

return "done " + msg

if __name__ == "__main__":

pool = multiprocessing.pool(processes=4)

result = []

for i in xrange(10):

msg = "hello %d" %(i)

result.append(pool.apply_async(func, (msg, )))

pool.close()

pool.join()

for res in result:

print res.get()

print "sub-process(es) done."1234567891011121314151617181920

2014.12.25更新

根据网友评论中的反馈，在windows下运行有可能崩溃(开启了一大堆新窗口、进程)，可以通过如下调用来解决：

multiprocessing.freeze_support()1

附录(自己的脚本)：

#!/usr/bin/python

import threading

import subprocess

import datetime

import multiprocessing

def dd_test(round, th):

test_file_arg = 'of=/zbkc/test_mds_crash/1m_%s_%s_{}' %(round, th)

command = "seq 100 | xargs -i dd if=/dev/zero %s bs=1m count=1" %test_file_arg

print command

subprocess.call(command,shell=true,stdout=open('/dev/null','w'),stderr=subprocess.stdout)

def mds_stat(round):

p = subprocess.popen("zbkc mds stat", shell = true, stdout = subprocess.pipe)

out = p.stdout.readlines()

if out[0].find('active') != -1:

command = "echo '0205pm %s round mds status ok, %s' >> /round_record" %(round, datetime.datetime.now())

command_2 = "time (ls /zbkc/test_mds_crash/) 2>>/round_record"

command_3 = "ls /zbkc/test_mds_crash | wc -l >> /round_record"

subprocess.call(command,shell=true)

subprocess.call(command_2,shell=true)

subprocess.call(command_3,shell=true)

return 1

else:

command = "echo '0205 %s round mds status abnormal, %s, %s' >> /round_record" %(round, out[0], datetime.datetime.now())

subprocess.call(command,shell=true)

return 0

#threads = []

for round in range(1, 1600):

pool = multiprocessing.pool(processes = 10) #使用进程池

for th in range(10):

# th_name = "thread-" + str(th)

# threads.append(th_name) #添加线程到线程列表

# threading.thread(target = dd_test, args = (round, th), name = th_name).start() #创建多线程任务

pool.apply_async(dd_test, (round, th))

pool.close()

pool.join()

#等待线程完成

# for t in threads:

# t.join()

if mds_stat(round) == 0:

subprocess.call("zbkc -s",shell=true)

break