Running Bash Commands in Parallel — urmaul.com

Let’s imagine you have a script like below and you want to make it faster.

#!/bin/bash

echo "Started at `date +%Y-%m-%d\ %H:%M:%S`"

echo "Starting job 1"
sleep 5

echo "Starting job 2"
sleep 5

echo "Finished at `date +%Y-%m-%d\ %H:%M:%S`"

If it’s ok to run jobs in parallel, you can easily do it with bash background jobs.

By adding and ampersand sign (&) in the end of command you tell bash to run this command in background. Then you can wait command to wait until child background jobs finish.

#!/bin/bash

echo "Started at `date +%Y-%m-%d\ %H:%M:%S`"

echo "Starting job 1"
sleep 5 &

echo "Starting job 2"
sleep 5 &

wait

echo "Finished at `date +%Y-%m-%d\ %H:%M:%S`"

That worked fine with single command, but can you run several sequential commands in one subprocess? Yes, by grouping them with { ... }.

#!/bin/bash

echo "Started at `date +%Y-%m-%d\ %H:%M:%S`"

{
    echo "Starting job 1"
    sleep 5
    echo "Finished job 1"
} &

{
    echo "Starting job 2"
    sleep 5
    echo "Finished job 2"
} &

wait

echo "Finished at `date +%Y-%m-%d\ %H:%M:%S`"

Now let’s imagine these commands are heavy and running them all will cause big load on your server. In that case it’s possible to to limit background processes to a certain number.

The best way I found to do this is just to wait until there are less than JOBS_LIMIT jobs before running next one. We use jobs -rp command to list running child processes and to see their count.

#!/bin/bash

JOBS_LIMIT=5

echo "Started at `date +%Y-%m-%d\ %H:%M:%S`"

jobs=(1 2 3 4 5 6 7 8 9 10)
for job in "${jobs[@]}"
do
    while [ `jobs -rp | wc -l` -ge $JOBS_LIMIT ]
    do
        sleep 1
    done

    {
        echo "Starting job $job"
        sleep $(( ( RANDOM % 5 )  + 1 ))
        echo "Finished job $job"
    } &
done

wait

echo "Finished at `date +%Y-%m-%d\ %H:%M:%S`"

These tricks give you power to parallelize your scripts and make them much faster. If you want more tricks when running commands manually – check this article. If you have better ideas about running commands from bash scripts – don’t hesitate to enlighten me in cooments.

And here’s a bit of Nicholas Cage for you: .