How to Fully Utilize All Cores of a UNIX Compute Node

Parallelizing tasks can drastically improve computation time of a program.  One way to do this is to ensure your code is utilizing all available cores of a processor.  To do this you need to write your code in such a way that background tasks are being carried out simultaneously.  This is done by inserting the ampersand (&) at the end of a line of code.  If the number of background tasks running equals the number of cores of the computer node, then you are efficiently and fully utilizing the resource.  The final necessary piece of code is to use the wait command.  This tells the computer to wait until all the background tasks are completed before moving on to the next line of code.  The wait command comes in handy to ensure the number of background tasks submitted does not exceed the number of processor cores.  If this happens you will likely overwhelm the processor with too many tasks.  To prevent this the idea is to simultaneously submit a number of tasks equal to the number of cores and then use the wait command to wait for the jobs to finish before submitting more tasks.  Here is some example code I put together to utilize all 8 cores of a node when running 1,000 permutations of a program:

	#!/bin/bash
	# Create new run_jobs.sh file
	echo "#!/bin/bash" > run_jobs.sh
	# Add entries for each permutation (1-1000) in groups of 8 (ie: number of cores)
	for i in {1..125}
	do
	for j in {7..0}
	do
	echo "python permutation.py $(($i*8-$j)) &" >> run_jobs.sh
	done
	echo "wait" >> run_jobs.sh
	done
	# Submit program
	qsub -l nodes=1,walltime=4:00:00 run_jobs.sh
	wait
	# Remove run list
	rm run_jobs.sh

view raw parallel_jobs.sh hosted with ❤ by GitHub

This will run a script with the following commands:

	#!/bin/bash
	python permutation.py 1 &
	python permutation.py 2 &
	python permutation.py 3 &
	python permutation.py 4 &
	python permutation.py 5 &
	python permutation.py 6 &
	python permutation.py 7 &
	python permutation.py 8 &
	wait
	python permutation.py 9 &
	python permutation.py 10 &
	python permutation.py 11 &
	python permutation.py 12 &
	python permutation.py 13 &
	python permutation.py 14 &
	python permutation.py 15 &
	python permutation.py 16 &
	wait
	...
	...
	...
	...
	...
	...
	...
	...
	wait
	python permutation.py 993 &
	python permutation.py 994 &
	python permutation.py 995 &
	python permutation.py 996 &
	python permutation.py 997 &
	python permutation.py 998 &
	python permutation.py 999 &
	python permutation.py 1000 &
	wait

view raw run_jobs.sh hosted with ❤ by GitHub

Of course, this could further be parallelized to run permutations simultaneously on different compute nodes as well to further speed up run time. Hope this is a helpful example to help you fully utilize compute nodes and speed up your processing time.