Multithreaded Bash Script

How to Use Multi-Threaded Processing in Bash Scripts

@RoelVandePaar
Jul 27, 2021, 5:00 am EDT
| 5 min read
Aleksey Mnogosmyslov/
Multi-threaded programming has always been of interest to developers to increase application performance and optimize resource usage. This guide will introduce you to Bash multi-threaded coding basics.
What Is multi-threaded programming?
A picture is worth a thousand words, and this holds when it comes to showing the difference between single (1) thread programming and multi-threaded (>1) programming in Bash:
sleep 1
sleep 1 & sleep 1
Our first multi-threaded programming setup or mini one-liner script could not have been simpler; in the first line, we sleep for one second using the sleep 1 command. As far as the user is concerned, a single thread was executing a single sleep of one second.
In the second line, we have two one-second sleep commands. We join them by using a & separator, which does not only act as a separator between the two sleep commands, but also as an indicator to Bash to start the first command in a background thread.
Normally, one would terminate a command by using a semicolon (;). Doing so would execute the command and only then proceed to the next command listed behind the semicolon. For example, executing sleep 1; sleep 1 would take just over two seconds – exactly one second for the first command, one second for the second, and a tiny amount of system overhead for each of the two commands.
However, instead of terminating a command with a semicolon one can use other command terminators which Bash recognizes like &, && and ||. The && syntax is quite unrelated to multi-threaded programming, it simply does this; proceed with executing the second command only if the first command was successful. The || is the opposite of && and will execute the second command only if the first command failed.
Returning to multi-threaded programming, using & as our command terminator will initiate a background process executing the command preceding it. It then immediately proceeds with executing the next command in the current shell while leaving the background process (thread) to execute by itself.
In the output of the command we can see a background process being started (as indicated by [1] 445317 where 445317 is the Process ID or PID of the just started background process and [1] is an indicated that this is our first background process) and it subsequently being terminated (as indicated by [1]+ Done sleep 1).
If you would like to view an additional example of background process handling, please see our Bash Automation and Scripting Basics (Part 3) article. Additionally, Bash Process Termination Hacks may be of interest.
Let’s now proof that we are effectively running two sleep processes at the same time:
time sleep 1; echo ‘done’
time $(sleep 1 & sleep 1); echo ‘done’
Here we start our sleep process under time and we can see how our single threaded command ran for exactly 1. 003 seconds before our command line prompt was returned.
However, in the second example, it took about the same time (1. 005 seconds) even though we were executing two periods (and processes) of sleep, though not consecutively. Again we used a background process for the first sleep command, leading to (semi-)parallel execution, i. e., multi-threaded.
We also used a subshell wrapper ($(… )) around our two sleep commands to combine them together under time. As we can see our done output shows in 1. 005 seconds and thus the two sleep 1 commands must have run simultaneously. Interesting is the very small increase in overall processing time (0. 002 seconds) which can be easily explained by the time required to start a subshell and the time required to initiate a background process.
Multi-threaded (and Background) Process Management
In Bash, multi-threaded coding will normally involve background threads from a main one-line script or full Bash script. In essence, one may think about multi-threaded coding in Bash as starting several background threads. When one starts to code using multiple threads, it quickly becomes clear that such threads will usually require some handling. For example, take the fictive example where we start five concurrent periods (and processes) of sleep in a Bash script;
#! /bin/bash
sleep 10 &
sleep 600 &
sleep 1200 &
sleep 1800 &
sleep 3600 &
When we start the script (after making it executable using chmod +x), we see no output! Even if we execute jobs (the command which shows any background jobs in progress), there is no output. Why?
The reason is that the shell which was used to start this script (i. e., the current shell) is not the same shell (nor the same thread; to start thinking in terms of subshells as threads in and by themselves) that executed the actual sleep commands or placed them into the background. It was rather the (sub)shell which was started when. / was executed.
Let’s change our script by adding jobs inside the script. This will ensure that jobs is executed from within the (sub)shell where it is relevant, the same one as to where the periods (and processes) of sleep were started.
This time we can see the list of background processes being started thanks to the jobs command at the end of the script. We can also see their PID’s (Process Identifiers). These PIDs are very important when it comes to handling and managing background processes.
Another way to obtain the background Process Identifier is to query for it immediately after placing a program/process into the background:
echo ${! }
Similar to our jobs command (with new PID’s now as we restarted our script), thanks to the Bash ${! } variable being echoed, we will now see the five PID’s being displayed almost immediately after the script starts: the various sleep processes were placed into background threads one after the other.
The wait Command
Once we have started our background processes, we have nothing further to do than wait for them to be finished. However, when each background process is executing a complex subtask, and we need the main script (which started the background processes) to resume executing when one or more of the background processes terminates, we need additional code to handle this.
Let’s expand our script now with the wait command to handle our background threads:
T1=${! }
T2=${! }
T3=${! }
T4=${! }
T5=${! }
echo “This script started 5 background threads which are currently executing with PID’s ${T1}, ${T2}, ${T3}, ${T4}, ${T5}. ”
wait ${T1}
echo “Thread 1 (sleep 10) with PID ${T1} has finished! ”
wait ${T2}
echo “Thread 2 (sleep 600) with PID ${T2} has finished! ”
Here we expanded our script with two wait commands which wait for the PID attached to the first and second threads to terminate. After 10 seconds, our first thread exists, and we are notified of the same. Step by step, this script will do the following: start five threads at almost the same time (though the starting of the threads itself is still sequential and not parallel) where each of the five sleep‘s will execute in parallel.
The main script then (sequentially) reports on the thread created and subsequently waits for the Process ID of the first thread to terminate. When that happens, it will sequentially report on the first thread finishing and commence a wait for the second thread to finish, etc.
Using the Bash idioms &, ${! } and the wait command give us great flexibility when it comes to running multiple threads in parallel (as background threads) in Bash.
Wrapping up
In this article we explored Bash multi-threaded scripting basics. We introduced the background process operator (&) using some easy-to-follow examples showing both single and multi-threaded sleep commands. Next, we looked at how to handle background processes via the commonly used Bash idioms ${! } and wait. We also explored the jobs command to see running background threads/processes.
If you enjoyed reading this article, have a look at our Bash Process Termination Hacks article.

Multi-threaded Bash scripting & process management…

The things you can do using Bash script are limitless. Once you begin to developed advanced scripts, you’ll soon find you will start to run into operating system limits. For example, does your computer have 2 CPU threads or more (many modern machines have 8-32 threads)? If so, then you will likely benefit from multi-threaded Bash scripting and coding. Continue reading and find out why!
In this tutorial you will learn:
How to implement multi-threaded Bash one-liners directly from the command line
Why multi-threaded coding almost always can and will increase the performance of your scripts
How background and foreground processes work and how to manipulate job queues
Multi-threaded Bash scripting & process management
Software requirements and conventions used
Software Requirements and Linux Command Line Conventions
Category
Requirements, Conventions or Software Version Used
System
Distribution-independent, Bash version-dependent
Software
Bash command line interface (bash)
Conventions
# – requires given linux commands to be executed with root privileges either directly as a root user or by use of sudo command
$ – requires given linux commands to be executed as a regular non-privileged user
When you execute a Bash script, it will at maximum use a single CPU thread, unless you start subshells/threads. If your machine has at least two CPU threads, you will be able to max-out CPU resources using multi-threaded scripting in Bash. The reason for this is simple; as soon as a secondary ‘thread’ (read: subshell) is started, then that subsequent thread can (and often will) use a different CPU thread.
Assume for a moment that you have a modern machine with 8 or more threads. Can you start seeing how if we would be able to execute code – eight parallel threads all at the same time, each running on a different CPU thread (or shared across all threads) – this way it would execute much faster then a single-threaded process running on a single CPU thread (which may be co-shared with other running processes)? The gains realized will depend a bit on what is being executed, but gains there will be, almost always!
Excited? Great. Let’s dive into it.
First we need to understand what a subshell is, how it is started, why you would use one, and how it can be used to implement multi-threaded Bash code.
A subshell is another Bash client process executed/started from within the current one. Let’s do something easy, and start one from within an opened Bash terminal prompt:
$ bash
$ exit
exit
$
What happened here? First we started another Bash shell (bash) which started and in turn yielded a command prompt ($). So the second $ in the example above is actually a different Bash shell, with a different PID (PID is the process identifier; a unique number identifier which uniquely identifies each running process in an operating system). Finally we exited from the subshell via exit and returned to the parent subshell! Can we somehow proof this is really what happened? Yes:
$ echo $
220250
222629
There is a special variable in bash $$, which contains the PID of the current shell in use. Can you see how the process identifier changed once we were inside a subshell?
Great! Now that we know what subshells are, and a little about how they work, let’s dive into some multi-threaded coding examples and learn more!
Simple multi-threading in Bash
Let us start with a simple one-liner multi-threaded example, of which the output may look somewhat confusing at first:
$ for i in $(seq 1 2); do echo $i; done
1
2
$ for i in $(seq 1 2); do echo $i & done
[1] 223561
[2] 223562
$ 2
[1]- Done echo $i
[2]+ Done echo $i
In the first for loop (see our article on Bash loops to learn how to code loops), we simply output the variable $i which will range from 1 to 2 (due to our use of the seq command), which – interestingly – is started in a subshell!
NOTE
You can use the $(… ) syntax anywhere within a command line to start a subshell: it is a very powerful and versatile way to code subshells directly into other command lines!
In the second for loop, we have changed only one character. Instead of using; – an EOL (end of line) Bash syntax idiom which terminates a given command (you may think about it like Enter/Execute/Go ahead), we used &. This simple change makes for an almost completely different program, and our code is now multi-threaded! Both echo’s will process more or less at the same time, with a small delay in the operating system still having to execute the second loop run (to echo ‘2’).
You can think about & in a similar way to; with the difference that & will tell the operating system to ‘keep running the next command, keep processing the code’ whereas; will wait for the current executing command (terminated by;) to terminate/finish before returning to the command prompt / before continuing to process and execute the next code.
Let’s now examine the output. We see:
At first, followed by:
And there is also an empty line in between, which is the result of background processes still running whilst waiting for the next command input (try this command a few times at the command line, as well as some light variations, and you will get a feel how this works).
The first output ([1] 223561) shows us that a background process was started, with PID 223561 and the identifier number 1 was given to it. Then, already before the script reached the second echo (an echo likely being an expensive code statement to run), the output 1 was shown.
Our background process did not finish completely as the next output indicates we started a second subshell/thread (as indicated by [2]) with PID 223562. Subsequently the second process outputs the 2 (“indicatively”: OS mechanisms may affect this) before the second thread finalizes.
Finally, in the second block of output, we see the two processes terminating (as indicated by Done), as well as what they were executing last (as indicated by echo $i). Note that the same numbers 1 and 2 are used to indicate the background processes.
More multi-threading in Bash
Next, let’s execute three sleep commands, all terminated by & (so they start as background processes), and let us vary their sleep duration lengths, so we can more clearly see how background processing works.
$ sleep 10 & sleep 1 & sleep 5 &
[1] 7129
[2] 7130
[3] 7131
[2]- Done sleep 1
[3]+ Done sleep 5
[1]+ Done sleep 10
The output in this case should be self-explanatory. The command line immediately returns after our sleep 10 & sleep 1 & sleep 5 & command, and 3 background processes, with their respective PID’s are shown. I hit enter a few times in between. After 1 second the first command completed yielding the Done for process identifier [2]. Subsequently the third and first process terminated, according to their respective sleep durations. Also note that this example show clearly that multiple jobs are effectively running, simultaneously, in the background.
You may have also picked up the + sign in the output examples above. This is all about job control. We will look at job control in the next example, but for the moment it’s important to understand that + indicates is the job which will be controlled if we were to use/execute job control commands. It is always the job which was added to the list of running jobs most recently. This is the default job, which is always the one most recently added to the list of jobs.
A – indicates the job which would become the next default for job control commands if the current job (the job with the + sign) would terminate. Job control (or in other words; background thread handling) may sound a bit daunting at first, but it is actually very handy and easy to use once you get used to it. Let’s dive in!
Job control in Bash
$ sleep 10 & sleep 5 &
[1] 7468
[2] 7469
$ jobs
[1]- Running sleep 10 &
[2]+ Running sleep 5 &
$ fg 2
sleep 5
$ fg 1
sleep 10
Here we placed two sleeps in the background. Once they were started, we examined the currently running jobs by using the jobs command. Next, the second thread was placed into the foreground by using the fg command followed by the job number. You can think about it like this; the & in the sleep 5 command was turned into a;. In other words, a background process (not waited upon) became a foreground process.
We then waited for the sleep 5 command to finalize and subsequently placed the sleep 10 command into the foreground. Note that each time we did this we had to wait for the foreground process to finish before we would receive our command line back, which is not the case when using only background processes (as they are literally ‘running in the background’).
Job control in Bash: job interruption
$ sleep 10
^Z
[1]+ Stopped sleep 10
$ bg 1
[1]+ sleep 10 &
Here we press CTRL+z to interrupt a running sleep 10 (which stops as indicated by Stopped). We then place the process into the background and finally placed it into the foreground and wait for it to finish.
$ sleep 100
[1]+ Stopped sleep 100
$ kill%1
[1]+ Terminated sleep 100
Having started a 100 second sleep, we next interrupt the running process by CTRL+z, and then kill the first started/running background process by using the kill command. Note how we use%1 in this case, instead of simply 1. This is because we are now working with a utility which is not natively tied to background processes, like fg and bg are. Thus, to indicate to kill that we want to effect the first background process, we use% followed by the background process number.
Job control in Bash: process disown
$ bg%1
[1]+ sleep 100 &
$ disown
In this final example, we again terminate a running sleep, and place it into the background. Finally we execute the disown command which you can read as: disassociate all background processes (jobs) from the current shell. They will keep running, but are no longer ‘owned’ by the current shell. Even if you close your current shell and logout, these processes will keep running until they naturally terminate.
This is a very powerful way to interrupt a process, place it into the background, disown it and then logout from the machine you were using, provided you will not need to interact with the process anymore. Ideal for those long running processes over SSH which cannot be interrupted. Simply CTRL+z the process (which temporarily interrupts it), place it into the background, disown all jobs, and logout! Go home and have a nice relaxed evening knowing your job will keep running!
Multi-threaded Bash scripting & process management command line examples
Conclusion
In this tutorial we saw how to implement multi-threaded Bash one-liners directly from the command line, and explored why multi-threaded coding often increases the performance of your scripts. We also examined how background and foreground processes work, and we manipulated job queues. Finally we explored how to disown our job queue from the current process, providing us with additional control over running processes. Enjoy your new found skills, and leave us a comment below with your job control experiences!

Forking / Multi-Threaded Processes | Bash – Stack Overflow

I would like to make a section of my code more efficient. I’m thinking of making it fork off into multiple processes and have them execute 50/100 times at once, instead of just once.
For example (pseudo):
for line in file;
do
foo;
foo2;
foo3;
done
I would like this for loop to run multiple times. I know this can be done with forking. Would it look something like this?
while(x <= 50) parent(child pid) { fork child()} child foo; foo2; foo3; return child_pid()} Or am I thinking about this the wrong way? Thanks! Paul Dixon282k48 gold badges304 silver badges337 bronze badges asked Sep 21 '09 at 17:19 In bash scripts (non-interactive) by default JOB CONTROL is disabled so you can't do the the commands: job, fg, and bg. Here is what works well for me: #! /bin/sh set -m # Enable Job Control for i in `seq 30`; do # start 30 jobs in parallel sleep 3 & # Wait for all parallel jobs to finish while [ 1]; do fg 2> /dev/null; [ $? == 1] && break; done
The last line uses “fg” to bring a background job into the foreground. It does this in a loop until fg returns 1 ($? == 1), which it does when there are no longer any more background jobs.
answered Sep 4 ’10 at 20:50
5
I don’t know of any explicit fork call in bash. What you probably want to do is append &
to a command that you want to run in the background. You can also use & on functions that you define within a bash script:
do_something_with_line()
line=$1
foo
foo2
foo3}
for line in file
do_something_with_line $line &
EDIT: to put a limit on the number of simultaneous background processes, you could try something like this:
while [`jobs | wc -l` -ge 50]
sleep 5
answered Sep 21 ’09 at 17:32
mobmob112k17 gold badges140 silver badges268 bronze badges
8
I don’t like using wait because it gets blocked until the process exits, which is not ideal when there are multiple process to wait on as I can’t get a status update until the current process is done. I prefer to use a combination of kill -0 and sleep to this.
Given an array of pids to wait on, I use the below waitPids() function to get a continuous feedback on what pids are still pending to finish.
declare -a pids
waitPids() {
while [ ${#pids[@]} -ne 0]; do
echo “Waiting for pids: ${pids[@]}”
local range=$(eval echo {0.. $((${#pids[@]}-1))})
local i
for i in $range; do
if! kill -0 ${pids[$i]} 2> /dev/null; then
echo “Done — ${pids[$i]}”
unset pids[$i]
fi
pids=(“${pids[@]}”) # Expunge nulls created by unset.
sleep 1
echo “Done! “}
When I start a process in the background, I add its pid immediately to the pids array by using this below utility function:
addPid() {
local desc=$1
local pid=$2
echo “$desc — $pid”
pids=(${pids[@]} $pid)}
Here is a sample that shows how to use:
for i in {2.. 5}; do
sleep $i &
addPid “Sleep for $i” $!
waitPids
And here is how the feedback looks:
Sleep for 2 — 36271
Sleep for 3 — 36272
Sleep for 4 — 36273
Sleep for 5 — 36274
Waiting for pids: 36271 36272 36273 36274
Done — 36271
Waiting for pids: 36272 36273 36274
Done — 36272
Waiting for pids: 36273 36274
Done — 36273
Waiting for pids: 36274
Done — 36274
Done!
answered Nov 5 ’14 at 12:07
haridsvharidsv7, 5154 gold badges57 silver badges59 bronze badges
1
With GNU Parallel you can do:
cat file | parallel ‘foo {}; foo2 {}; foo3 {}’
This will run one job on each cpu core. To run 50 do:
cat file | parallel -j 50 ‘foo {}; foo2 {}; foo3 {}’
Watch the intro videos to learn more:
answered Jan 5 ’12 at 9:51
Ole TangeOle Tange1, 85615 silver badges10 bronze badges
2
haridsv’s approach is great, it gives the flexibility to run a processor slots setup where a number of processes can be kept running with new jobs submitting as jobs complete, keeping the overall load up. Here are my mods to haridsv’s code for an n-slot processor for a ‘grid’ of ngrid ‘jobs’ ( I use it for grids of simulation models) Followed by test output for 8 jobs 3 at a time, with running totals of running, submitted, completed and remaining
#! /bin/bash
########################################################################
# see haridsv on forking-multi-threaded-processes-bash
# loop over grid, submitting jobs in the background.
# As jobs complete new ones are set going to keep the number running
# up to n as much as possible, until it tapers off at the end.
#
# 8 jobs
ngrid=8
# 3 at a time
n=3
# running counts
running=0
completed=0
# previous values
prunning=0
pcompleted=0
# process monitoring functions
function checkPids() {
echo ${#pids[@]}
if [ ${#pids[@]} -ne 0]
then
echo “Checking for pids: ${pids[@]}”
completed=$(expr $completed + 1)
running=$((${#pids[@]}))
echo “#PIDS:”$running
fi}
function addPid() {
desc=$1
pid=$2
echo ” ${desc} – “$pid
# Loop and report when job changes happen,
# keep going until all are completed.
idx=0
while [ $completed -lt ${ngrid}]
if [ $running -lt $n] && [ $idx -lt ${ngrid}]
####################################################################
# submit a new process if less than n
# are running and we haven’t finished…
# get desc for process
name=”job_”${idx}
# background execution
addPid $name $!
idx=$(expr $idx + 1)
checkPids
# if something changes…
if [ ${running} -gt ${prunning}] || \
[ ${completed} -gt ${pcompleted}]
remain=$(expr $ngrid – $completed)
echo ” Running: “${running}” Submitted: “${idx}\
” Completed: “$completed” Remaining: “$remain
# save counts to prev values
prunning=${running}
pcompleted=${completed}
Test output:
job_0 – 75257
Checking for pids: 75257
#PIDS:1
Running: 1 Submitted: 1 Completed: 0 Remaining: 8
job_1 – 75262
Checking for pids: 75257 75262
#PIDS:2
Running: 2 Submitted: 2 Completed: 0 Remaining: 8
job_2 – 75267
3
Checking for pids: 75257 75262 75267
#PIDS:3
Running: 3 Submitted: 3 Completed: 0 Remaining: 8
Done — 75257
Running: 2 Submitted: 3 Completed: 1 Remaining: 7
job_3 – 75277
Checking for pids: 75262 75267 75277
Done — 75262
Running: 2 Submitted: 4 Completed: 2 Remaining: 6
job_4 – 75283
Checking for pids: 75267 75277 75283
Done — 75267
Running: 2 Submitted: 5 Completed: 3 Remaining: 5
job_5 – 75289
Checking for pids: 75277 75283 75289
Running: 3 Submitted: 6 Completed: 3 Remaining: 5
Done — 75277
Running: 2 Submitted: 6 Completed: 4 Remaining: 4
job_6 – 75298
Checking for pids: 75283 75289 75298
Done — 75283
Running: 2 Submitted: 7 Completed: 5 Remaining: 3
job_7 – 75304
Checking for pids: 75289 75298 75304
Done — 75289
Running: 2 Submitted: 8 Completed: 6 Remaining: 2
Checking for pids: 75298 75304
Done — 75298
Running: 1 Submitted: 8 Completed: 7 Remaining: 1
Checking for pids: 75304
Done — 75304
#PIDS:0
Running: 0 Submitted: 8 Completed: 8 Remaining: 0
answered Jun 19 ’16 at 1:41
RalphRalph4015 silver badges3 bronze badges
Let me try example
for x in 1 2 3; do { echo a $x; sleep 1; echo b $x;} & done; sleep 10
And use jobs to see what’s running.
answered Sep 21 ’09 at 17:23
Based on what you all shared I was able to put this together:
#! /usr/bin/env bash
VAR1=”192. 168. 1. 20 192. 126 192. 36”
for a in $VAR1; do { ssh -t -t $a -l Administrator “sudo softwareupdate -l”;} & done;
WAITPIDS=”$WAITPIDS “$! ;… ; wait $WAITPIDS
echo “Script has finished”
Exit 1
This lists all the updates on the mac on three machines at once. Later on I used it to perform a software update for all machines when i CAT my
answered Dec 13 ’12 at 22:10
KamalKamal172 bronze badges
Here’s my thread control function:
# This function just checks jobs in background, don’t do more things.
# if jobs number is lower than MAX, then return to get more jobs;
# if jobs number is greater or equal to MAX, then wait, until someone finished.
# Usage:
# thread_max 8
# thread_max 0 # wait, until all jobs completed
thread_max() {
local CHECK_INTERVAL=”3s”
local CUR_THREADS=
local MAX=
[[ $1]] && MAX=$1 || return 127
# reset MAX value, 0 is easy to remember
[ $MAX -eq 0] && {
MAX=1
DEBUG “waiting for all tasks finish”}
while true; do
CUR_THREADS=`jobs -p | wc -w`
# workaround about jobs bug. If don’t execute it explicitily,
# CUR_THREADS will stick at 1, even no jobs running anymore.
jobs &>/dev/null
DEBUG “current thread amount: $CUR_THREADS”
if [ $CUR_THREADS -ge $MAX]; then
sleep $CHECK_INTERVAL
else
return 0
done}
answered Dec 4 ’13 at 16:08
0
Not the answer you’re looking for? Browse other questions tagged bash shell fork or ask your own question.

Frequently Asked Questions about multithreaded bash script

Is multithreading possible in bash?

Parallel executes Bash scripts in parallel via a concept called multi-threading. This utility allows you to run different jobs per CPU instead of only one, cutting down on time to run a script.Jun 2, 2021

How do I run multiple shell scripts in parallel?

How can I run multiple programs in parallel from a bash script? You have various options to run programs or commands in parallel on a Linux or Unix-like systems: => Use GNU/parallel or xargs command. => Use wait built-in command with &.Jul 25, 2018

What is forking in bash?

When a command or the shell itself initiates (or spawns) a new subprocess to carry out a task, this is called forking. This new process is the child, and the process that forked it off is the parent. While the child process is doing its work, the parent process is still executing.