harnessing the power of the multi-core processor in Python

The following is a very basic way to get Python to launch a number of processes to run in parallel, allowing me to make better use of the 8 CPUs sitting under the hood of my desktop machine:

1. Install the "subprocess" module.

2. set up a "master" script that uses the function subprocess.Popen to launch all desired processes. Simplified example:

import subprocess

import sys

slave_file = "slave.py"

for param in params:

log_file = open("results_%s.log" % param, "w")

subprocess.Popen([sys.executable, slave_file, param], 0, None, None, log_file)

log_file.close()

The first arg to Popen is an array containing the individual strings in the command line call to python, and the last is the file handle to which std_out from the slave process will be redirected.

3. to avoid the spawned processes devouring all 8 CPUs --- causing everything to grind to a halt --- it is a good idea to limit the processors that can be used. This can be done by setting the "processor affinities" in Task Manager:

a. With TM select the "Processes" tab

b. Find the shell process from which the master script will be launched (for me it will be a "bash.exe" instance)

c. Right-click on this, and select "Set process affinity..."

d. Uncheck one or two processors to prevent them being used.

These "affinities" should then be inherited by all the python scripts that are spawned when the master script is run.