Themes Syntax Bash

Bash Mastery

Sample article showcasing basic Bash syntax.

Revision: March 2, 2025

Variables and Shell Expansions

Bash Reference Manual - Shell Parameters

User-defined Variables and Parameter Expansion

The standard syntax for parameter expansion is ${parameter} as this allows for advanced expansions. $parameter is a simpler syntax for just referring to a variable’s value but does not support advanced expansions.

Parameter : Variables, Positional parameters, Special parameters

1
2
3
4
5
6
7

# user-defined variable name should be all lower case, and no spaces around = sign!
student="John"

# to reference a parameter, also called parameter expansion
echo Hello ${student}
# or
echo Hello $student
  • Parameter expansion tricks

Bash Reference Manual - Shell Parameter Expansion

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# to upper case
$ echo $USER
$ echo ${USER^}
$ echo ${USER^^}

# to lower case
$ name=ZiYaD
$ echo $name
$ echo ${name,}
$ echo ${name,,}

# the length of a parameter, the number of charactors of the variable
$ echo ${#name}

# to slice a parameter, echo ${parameter:offset:length}
$ numbers=0123456789
$ echo ${numbers:1:5}
$ echo ${numbers:3:} # BAD! Don't do this!
$ echo ${numbers:3}
# careful with the mandatory whitespace before the - sign
$ echo ${numbers: -3:2}

Shell Variables

Bash Reference Manual - Shell Variables

PATH HOME USER HOSTNAME HOSTTYPE PS1 prompt string 1 PWD OLDPWD

1

$ export PATH=""$PATH:$HOME/.bin"

Command Substitution

A shell feature that allows you to grab the output of a command and do stuff with it. Syntax: $(command)

1
2
3

$ vartime=$(date +%H:%m:%S)
$ echo Hello $USER, the time right now is $vartime
$ echo Hello $USER, the time right now is $(date +%H:%m:%S)

Arithmetic Expansion

Syntax: $((expression)), only handle the whole number.

1
2
3
4
5
6
7

$ x=4
$ y=2

# the standard syntax
$ echo $(( $x + $y ))
# the shortcut
$ echo $(( x + y ))
  • Use the bc command to handle the decimal number. Syntax: echo "expression" | bc

basic calculator An arbitrary precision calculator language

1
2
3
4
5
6
7
8
9

# to use the interactive interface, exit by typing "quit"
$ bc

# to perform decimal calculations
# use the scale variable to set the number of decimal places to be included in the output
$ echo "scale=2; 5/2" | bc

# for exponentiation
$ echo "4^2" | bc

Tilde (~) Expansion

Bash Reference Manual - Tilde Expansion

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# it refer to the current user's home directory
# and expands to the current value of the HOME shell variable
$ echo ~

# the path to root's home directory
$ echo ~root

# expands to the current value of the PWD shell variable
$ echo ~+

# expands to the current value of the OLDPWD shell variable
$ echo ~-

Brace Expansion

a way to generate sequences of text that follow a particular pattern

  • String lists can contain any set of individual characters or words.

It’s important to note that there cannot be any unquoted spaces before or after the commas in the braces when making a string list.

1
2
3

$ echo {a,19,z,barry,42}
$ echo {jan,feb,mar,apr,may,jun}
$ echo {jan, feb,mar,apr,may,jun}
  • Range lists are useful for rapidly expanding out sequences of characters that follow a particular order
1
2
3
4
5
6

$ echo {0..9}
$ echo {9..0}
# control the step/incrementation
$ echo {0..1000..100}
# add leading zeros
$ echo month{01..12}/file{01..31}.txt

How Bash Processes Command Lines

Quoting

Quoting is used to remove special meaning from special characters so that they can be interpreted literally.

Bash’s three quoting methods;

  1. backslash (\) Removes special meaning from next character
1
2

$ echo john & jane
$ echo john \& jane
  1. Single Quotes (’’) Removes special meaning from all characters inside

The only rule that you need to remember with single quotes is that they cannot contain another single quote, even if that extra single quote is preceded by a backslash.

1
2
3
4
5
6
7
8

$ filepath=C:\Users\ziyad\Documents
$ echo $filepath

$ filepath=C:\\Users\\ziyad\\Documents
$ echo $filepath

$ filepath='C:\Users\ziyad\Documents'
$ echo $filepath
  1. Double Quotes ("") Removes special meaning from all except dollar signs ($) and backticks (`)
1
2

$ filepath="C:\Users\\$USER\Documents"
$ echo $filepath

the 5-step process bash uses to interpret a command line

Step 1: Tokenization - Identify Words & Operators

The shell identifies unquoted metacharacters and uses them to divide up the command line into tokens. It then characterizes the tokens into words and operators.

  • Token: “a sequence of characters that is considered as a single unit by the shell”

  • Metacharacters: | & ; () <> space/tab/newline ![[metachars.png]]

    • Word: “a token that does not contain an unquoted metacharacter”
    • Operator: “a token that contains at least one unquoted metacharacter”
      • Control operators are used to control how a command line is processed ![[control-chars.png]]
      • Redirection tell the shell to do certain redirections of the data streams that are connected to a command. ![[redirection-chars.png]]

  • Tokenization example ![[s1-tokennisation.png]]

Step 2: Command Identification

  • Simple commands A bunch of individual words, and each is terminated by a control operator. The first word of a single command is interpreted as the command name, and the following words are interpreted as arguments to that command.

  • Compound commands Provide bash with its programming constructs.

Step 3: Expansions

4 Stages of Expansions, and the expansions in earlier stage are performed first. A consequence of this is that expansions that happen in later stages can’t be used in the expansions that occur in earlier stages.

1
2
3
4

$ x=10
$ echo {1..$x}

$ echo {1..10}
  • Stage 1: Brace Expansion
  • Stage 2
    • Parameter Expansion (*)
    • Arithmetic Expansion (*)
    • Command Expansion (*)
    • Tilde Expansion
1
2

$ name=file
$ echo $name{1..3}.txt
  • Stage 3: Word Splitting (*)

A process the shell performs to split the result of some unquoted expansions into separate words. The characters used to split words are governed by the IFS (Internal Field Separator) variable.

1
2
3
4
5
6
7
8

# default value is space, tab and newline
$ echo "${IFS@Q}"

$ numbers="1 2 3 4 5"
$ touch $numbers

$ numbers="1,2,3,4,5"
$ touch $number

If you want the output of a Parameter/Arithmetic expansion or Command substitution to be considered as a single word: warp that expansion in double quotes!

  • Stage 4: Globbing (Filename Expansion) - only performed on words

    Bash Reference Manual - Pattern Matching

    • * Matches any string, including the null string.
    • ? Matches any single character.
    • […] Matches any one of the enclosed characters. A pair of characters separated by a hyphen denotes a range expression.

Used as a shortcut for listing the files that a command should operate on.

Step 4: Quote Removal

During quote removal, the shell removes all unquoted backslashes, single quote characters, and double quote characters that did NOT result from a shell expansion.

1
2
3
4
5

$ echo $HOME

$ echo \$HOME

$ echo '\$HOME'

Step 5: Redirection

&> The operator does redirect standard output and error to the same place.

  • Stream 0 - Standard Input (stdin)
  • Stream 1 - Standard Output (stdout)
  • Stream 2 - Standard Error (stderr)

Requesting User Input

Positional Parameters

1
2
3
4
5
6
7
8
9

# Parameter Expansion
$ cat positional.sh
#!/bin/bash 
echo "My nanme is $1"
echo "My home directory is $2"
echo "The 10th argument is ${10}"
echo "The 11th argument is $11"

$ ./positional.sh john $HOME 3 4 5 6 7 8 9 10 11

Special Parameters - unmodifiable

Bash Reference Manual - Special Parameters

  • $0 stores the script’s name

  • $# stores the number of command line arguments provides to the script

  • $@ expands to each positional parameters as its own word with subsequent word splitting

  • "$@" expands to each positional parameter as its own word without subsequent word splitting

  • $* exactly the same as $@

  • "$*" expands to all positional parameter as one word separated by the first letter of the IFS variable without subsequent word splitting

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# $@ : $1 $2 $3 ... $N
$ cat ./special.sh
#!/bin/bash
touch $@

$ ./special.sh "daily feedback" "monthly report"

# "$@" : "$1" "$2" "$3" ... "$N"
$ cat ./special.sh
#!/bin/bash
touch "$@"

$ ./special.sh "daily feedback" "monthly report"

The read and select Commands

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

$ echo $REPLY
$ read
hello
$ echo $REPLY

$ read input1 input2
hello goodbye
$ echo $input1
$ echo $input2

# -t timeout with seconds
# -p to prompt
# -s to mask the input
$ read -t 5 -p "Input your age: " age
$ echo $age

$ help read

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

#!/bin/bash

# PS3 variable controls the promopt string of select command
PS3="What is the day of the week?: " 
select day in mon tue wed thu fri sat sun;
do
	echo "The day of the week is $day"
	break
done

$ help select

Logic

List : when you put one or more commands on a given line

List Operators

  • & to run commands asynchronously (i.e. “in the background”)
  • ; to run commands sequentially
  • && to run “and” lists of commands
  • || to run “or” lists of commands
1
2

# a ternary operator
$ commandA && commandB || commandC

Test Commands and Conditional Operators

In bash, we can use the variable $? to check the exit status of the last command.

  • integer test operators
  • string test operators
  • file test operators
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

$ [[ 2 -eq 23 ]]; echo $?

$ a=hello
$ b=goodbye
$ [[ $a = $b ]] ; echo $?
$ [[ -z $c ]] ; echo $?

# exist
$ [[ -e today.txt ]] ; echo $?
$ touch today.txt
$ [[ -e today.txt ]] ; echo $?
# regular file
$ [[ -f today.txt ]] ; echo $?
# directory
$ [[ -d today.txt ]] ; echo $?
# a file exists and has execution permissions, basiclly design to check for scripts
$ [[ -d today.txt ]] ; echo $?

If Statement

1
2
3
4
5
6
7
8
9

#!/bin/bash

if [[ 2 -eq 1 ]]; then
	echo test passed
elif [[ 1 -eq 1 ]]; then
	echo second test passed
else
	echo test failed
fi

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

#!/bin/bash

a=$(cat file1.txt)
b=$(cat file2.txt)
c=$(cat file3.txt)

if [[ $a = $b ]] && [[ $a = $c ]]; then
	rm file2.txt file3.txt
else
	echo "Files do not match"
fi

1
2
3
4
5
6
7
8

#!/bin/bash

if [[ ! -d report ]]; then
	mkdir report
fi

# another trick, the 2nd command will only if the 1st test fails
[[ -d report ]] || mkdir report

Case Statement

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

#!/bin/bash

read -p "Please enter a number: " number
case "$number" in
	# globbing pattern
	[0-9]   )   echo echo "Digit";;
	[0-9][0-9]   )   echo "Two Digit";;
	[0-9][0-9][0-9]   )   echo "Three Digit";;

	# the catch-all default case
	*   )   echo "More than Three Digit";;
esac

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

#!/bin/bash

PS3="Please select a city: "
select city in London "Los Angeles" Moscow "New York" Paris;
do
	case "$city" in
		"Los Angeles"|"New York"   )   country="USA" ;;
		Moscow   )   country="Russia" ;;
		Paris   )   country="France" ;;
		London   )   country="UK" ;;
	esac
	echo "$city is in $country"
	break
done

Processing Options & Reading Files

While Loop and getopts Command

1
2
3
4
5
6
7

!/bin/bash

read -p "Input a number: " num
while [[ num -gt 5 ]]; do
	echo "The number is: " $num
	(( num-- ))
done 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

# ./temp_conv.sh
#!/bin/bash

# : to give arguments to the options
while getopts "f:c:" opt; do
    case "$opt" in
        c ) result=$(echo "scale=2; ($OPTARG * (9 / 5)) + 32" | bc) ;;
        f ) result=$(echo "scale=2; ($OPTARG - 32) * (5/9)" | bc) ;;
        \? ) ;;
    esac
done
echo $result

$ ./temp_conv.sh -f 32
$ ./temp_conv.sh -c 0

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# ./timer.sh
#!/bin/bash

total_seconds=0
while getopts "m:s:" opt; do
    case "$opt" in
        m ) total_seconds=$(( $total_seconds + $OPTARG * 60 )) ;;
        s ) total_seconds=$(( $total_seconds + $OPTARG )) ;;
        \? ) ;;
    esac
done

while [[ $total_seconds -gt 0 ]]; do
    echo "$total_seconds seconds left."
    sleep 1s
    (( total_seconds-- ))
done

echo "Time's Up!"

Iterating over Files with read-while Loops

  • Use read-while loops to iterate over the contents of files
1
2
3
4
5

#!/bin/bash

while read line; do
	echo "$line"
done < "$1"

1
2
3
4
5

#!/bin/bash

while read line; do
	echo "$line"
done < <(ls $HOME)

Arrays and For Loops

Indexed Arrays

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

$ numbers=(1 2 3 4 5)

$ echo $numbers
$ echo ${numbers[2]}
$ echo ${numbers[@]}
$ echo ${numbers[@]:1:2}  # parameter expansion tricks
$ echo ${#numbers[@]}   # parameter expansion tricks

# add value to an array
$ numbers+=(6)
$ echo ${numbers[@]}

# delete an element
$ unset numbers[2]
$ echo ${numbers[@]}
# to see the index numbers of elements
# the index is not re-adjust automatically
$ echo ${!numbers[@]}

# update an element
$ numbers[0]=a
$ echo ${numbers[@]}
  • The readarray Command

The readarray command converts whatever it reads on its standard input stream into an array per line

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

$ cat days.txt
Monday
Tuesday
Wednesday

$ readarray days < days.txt
$ echo ${day[@]}
Monday Tuesday Wednesday
$ echo ${day[@]@Q}
$'Monday\n' $'Tuesday\n' $'Wednesday\n'

# -t remove trailing newlines by default (and only store the raw data)
$ readarray -t days < days.txt
$ echo ${day[@]@Q}
'Monday' 'Tuesday' 'Wednesday'

1
2

# use with process substitution <()
$ readarray files < <(realpath *)

For Loops

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

#!/bin/bash

readarray -t files < files.txt

for file in "${files[@]}"; do
    if [ -f "$file" ]; then
        echo "$file already exists"
    else
        touch "$file"
        echo "$file was created"
    fi
done

1
2
3
4
5
6
7
8

#!/bin/bash

readarray -t urls < urls.txt

for url in "${urls[@]}"; do
    webname=$(echo "$url" | cut -d "." -f 2)
    curl --head "$url" >> ""$webname.txt"
done

Debugging

ShellCheck - finds bugs in your shell scripts

Error Message Structure

1
2

$ ls 1234
ls: cannot access '1234': No such file or directory

How to Find Help

  • Internal commands are commands that are built into the bash
    • help
  • External commands are commands that are external to bash How to Read Command Synopsis?
    • man (short for manual)
    • info to get more detailed information
1
2
3
4
5
6
7

$ type -a cd
cd is a shell builtin

$ type -a ls
ls is aliased to `ls --color=auto'
ls is /usr/bin/ls
ls is /bin/ls

1
2
3

$ help cd
$ help -d cd
$ help -s cd

1
2
3
4
5
6

# -k : keyword : search the short description
$ man -k compress
$ man -k "list directory contents"

# -K : search the entire man page
$ man -K partition

1
2

$ info ls
$ info

Scheduling and Automation

The at Command

  • atd the at daemon
  • Limitations:
    • only execute job of the PC is on at the time
    • no way to set up recurring jobs
1
2
3
4

$ sevice atd status
$ sudo service atd start
$ sudo service atd stop
$ sudo service atd restart

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

$ at 09:30am
$ at -l
$ at -r <jobid>

$ at 10:05am -f <script>
$ at 9am Monday -f <script>
$ at 9am 12/23/2022 -f <script>
$ at 9am 23.12.2022 -f <script>
$ at now + 5 minutes
$ at now + 2 days

Using cron to schedule and automate bash scripts

Only execute job of the PC is on at the time

1
2
3
4
5
6
7
8
9

$ service cron status

# user specific crontab
# this command will restart the service automatically
$ crontab -e

# system wide crontab
$ sudo vi /etc/crontab
$ sudo service cron restart
  • cron directories and run-parts command

folders on the system where can place scripts to run at a particular frequency.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

$ ls /etc | grep "cron.*y"
cron.daily
cron.hourly
cron.monthly
cron.weekly

$ mkdir ~/cron.daily.2am
$ sudo vi /etc/crontab
# m   h   dom   mon   dow   command
00   02   *   *   *   run-parts --report ~/cron.daily.2am

anacron service

can recovery missed jobs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

$ service anacron status

# only system wide configuration file
$ sudo vi /etc/anacrontab
# period dealy job-identifier command
7   10   backup.weekly   /home/bsun/.bin/back_script.sh
@monthly   15   cron.monthly   run-parts --report /etc/cron.monthly

# log per job identifier
$ ls /var/spool/anacron

Working with Remote Servers

How to execute scripts on a remote server (ssh : secure shell)

1
2
3
4

# use -p to specify the port
$ ssh bsun@pek-kong-06

$ ssh pek-kong-06 -lbsun

How to send and receive scripts on a remote server (scp)

1
2

# scp source target
$ scp /home/bsun/ip_script.sh bsun@pek-kong-06:~
comments powered by Disqus