I find it most useful when I approach any learning with a clear goal (or at least a specific enough task) to accomplish.

Here’s a list of simple Bash scripts that I think could be a useful learning exercise:

• what’s your name? – asks for your name and then shows a greeting
• hello, world! (that also shows hostname and list server IP addresses)
• write a system info script – that shows you a hostname, disks usage, network interfaces and maybe system load
• directory info – script that counts disks space taken up by a directory and shows number of files and number of directories in it
• users info – show number of users on the system, their full names and home directories (all taken from /etc/passwd file)
• list virtual hosts on your webserver – I actually have this as a login greeting on my webservers – small script that highlights which websites (domains) your server has in web server (Apache or Nginx) configuration.

Do you have any more examples of what you’d like to do in a Linux shell? If not, I’ll just start with the examples above. The plan is to replace each example name in the list above with a link to the Unix Tutorial post.

Although I’ve already shown you how to sum numbers up in bash, I only covered the bash way of doing it. I really like scripting with bash, but when it comes to calculations, there’s quite a few important features missing from bash, and fixed point (thanks for the correction, Azrael Tod!) calculations is one of them. Fortunately, bc command comes as a standard in most Unix distros, and can be used for quite complex calculations.

Basic calculations with bc

bc is a very simple command. It takes standard input as an expression and then evaluates this, performing all the necessary calculations and showing you the result. Thus, to quickly sum numbers up or get a result of some other calculation, simply echo the expression and then pipe it out to the bc command:

ubuntu$ echo "1+2" | bc
3

Now, in scripts your calculations with bc are done quite similarly to what we did in bash. Here’s an example:

ubuntu$ NUMBER1=1
ubuntu$ NUMBER2=2
ubuntu$ SUM=$(echo "$NUMBER1+$NUMBER2"| bc)
ubuntu$ echo $SUM
3

I told you these calculations would be basic, right? Now onto the more interesting stuff!

Fixed point calculations with bc

Most people learn about bash math limitations when they attempt to do a simple calculation but can’t get the current answer with fixed point values. By default, all the operations happen with integers, and that’s what you would get:

ubuntu$ echo "1/2" | bc
0

Now, if you expect 0.5 to be the result of dividing 1 by 2, you need to explain it to bc, because by default it doesn’t show you any fractional part of the number.

The way you do this is quite simple: all you have to do is specify the number of digits you’d like to see  after the radix point of your result. For example, if I set this number to 5, I’ll get bc to output the result of my calculation with 5 digits after the radix point. The special keyword to convey this intention to the bc command is called scale. Just specify the scale value and separate it from the rest of your expression by the semicolon sign:

ubuntu$ echo "scale=5; 1/2" | bc
.50000

Here’s another example:

ubuntu$ echo "scale=5; 0.16*10.79" | bc
1.7264

Hope this answers your question! bc command is very powerful, so I’ll definitely have to revisit it again in the future. For now though, enjoy the fixed point calculations and be sure to ask questions if you think I can help!

See also:

• Summing numbers up in Unix scripts
• Basic math in Unix scripting
• Another way of doing calculations in scripts
  

If you’re ever thought of summing up more than two numbers in shell script, perhaps this basic post will be a good start for your Unix scripting experiments.

Basic construction for summing up in shell scripts

In my Basic arithmetic operations in Unix shell post last year, I’ve shown you how to sum up two numbers:

SUM=$(($NUMBER1 + $NUMBER2)

using the same approach, it’s possible to calculate sums of as many numbers as you like, if you use one of the loops available in your shell.

Before we get started, let’s create a simple file with numbers we’ll work with:

ubuntu$ for i in 1 2 3 4 5 6 7 8 9; do echo $i >> /tmp/nums; done;
ubuntu$ cat /tmp/nums
1
2
3
4
5
6
7
8
9

Using a while loop to sum numbers up in Unix

Here’s an example of how you can use a while loop in Unix shell for summing numbers up. Save it as a /tmp/sum.sh script, and don’t forget to do chmod a+x /tmp/sum.sh so that you can run it!

#!/bin/sh
#
SUM=0
#
while read NUM
do
        echo "SUM: $SUM";
        SUM=$(($SUM + $NUM));
        echo "+ $NUM: $SUM";
done < /tmp/nums

Here’s how the output will look when you run it:

ubuntu$ /tmp/sum.sh
SUM: 0
+ 1: 1
SUM: 1
+ 2: 3
SUM: 3
+ 3: 6
SUM: 6
+ 4: 10
SUM: 10
+ 5: 15
SUM: 15
+ 6: 21
SUM: 21
+ 7: 28
SUM: 28
+ 8: 36
SUM: 36
+ 9: 45

Of course, your data will most probably will be in a less readable form, so you’ll have to do a bit of parsing before you get to sum the numbers up, but the loop will be organized in the same way.

That’s it for today, enjoy and feel free to ask questions!

See also:

• Basic arithmetics in scripts
• Variables in Unix shells
• Unix glossary
  

When I was writing a post about using date command to confirm date and time in your Unix scripts, I made a note in my future posts list to cover the date calculations – finding out the date of yesterday or tomorrow, and so on. Today I’ll show you the simplest way to calculate this.

GNU date command advantage

GNU version of the date command, although supporting a common syntax, has one great option: it allows you to specify the desired date with a simple string before reporting it back. What this means is that by default this specified date is assumed to be “now”, but you can use other keywords to shift the result of the date command and thus show what date it was yesterday or a week ago:

Here’s a default date output for the current date and time:

ubuntu$ date
Fri Sep 19 08:06:41 CDT 2008

Now, the parameter for specifying desired date is -d or –date, and if you use it with the “now” or “today” parameter, you’ll get similar output:

ubuntu$ date -d now
Fri Sep 19 08:06:44 CDT 2008
ubuntu$ date -d today
Fri Sep 19 08:06:50 CDT 2008

Showing tomorrow’s date

Similarly, you can get tomorrow’s date:

ubuntu$ date -d tomorrow
Sat Sep 20 08:02:12 CDT 2008

Obviously, if you feel like specifying a format for the date, you can do it:

ubuntu$ date -d tomorrow "+%b %d, %Y"
Sep 20, 2008

Find out yesterday’s date

Again, there’s no rocket science involved in showing yesterday’s date neither:

ubuntu$ date -d yesterday "+%b %d, %Y"
Sep 18, 2008

Show a date few days away

If you’re interested in a certain date a few days or even weeks away, here’s how you can do it:

Example 1: a date 5 days ago

ubuntu$ date -d "-5 days"
Sun Sep 14 08:45:57 CDT 2008

Example 2: a day 2 weeks from now

ubuntu$ date -d "2 weeks"
Fri Oct  3 08:45:08 CDT 2008

Example 3: a day two weeks ago from now

ubuntu$ date -d "-2 weeks"
Fri Sep  5 08:45:11 CDT 2008

Extreme example: a day 50 years ago

If you’re really curious about dates in Unix, you can even make GNU date go back a few years:

ubuntu$ date -d "-50 years"
Fri Sep 19 08:47:51 CDT 1958

That’s it for today, hope you like this little discovery – having mostly worked with Solaris systems most of my career, I didn’t know my Ubuntu had this functionality bonus. Really useful!

See also:

• Time and date in Unix scripts
• Variables in Unix shell scripts
  
• Updating shell variables in Unix
  

Today I’d like to expand a bit more on the basic calculations in Unix scripts.

Use Parenthesis to simplify math expressions

In Basic Arithmetic Operations post I’ve shown you how expression evaluation can be used to calculate simple math expressions in your Unix shell:

ubuntu$ START=1
ubuntu$ FINISH=10
ubuntu$ ELAPSED=$(($FINISH - $START))
ubuntu$ echo $ELAPSED
9

Although this approach looks clean enough, there’s a way to simplify it even further if you put everything in parenthesis. In other words, the same result (ELAPSED receiving a correct value of FINISH value minus START value) can be achieved this way:

ubuntu$ ((ELAPSED=FINISH-START))
ubuntu$ echo $ELAPSED
9

It’s a matter of preference, and I used to always do such calculations the former way shown, but now that one of the readers of this blog pointed out the latter way of using it, I think I might change my habits – I actually like this way much better.

See also:

• Time and date in Unix scripts
• Using variables in scripts
  
• Basic math in Unix scripts

If you’re just getting started with your Unix scripting or new to Unix shell, today’s little tip will make your life much easier: I’m going to show you how to update your shell variables.

Updating shell variable in Unix

If you have a text variable in your script and would like to append some text to it or somehow process this value, it’s perfectly normal to reuse the current variable value using a syntax like this:

ubuntu$ echo $VAR1
hello
ubuntu$ VAR1=$VAR1" world!"
ubuntu$ echo $VAR1
hello world!

You see? It’s this easy!

Common cases of reusing variable values in Unix shells

Most frequently, I use the technique described above to update some of my environment variables. I’m sure you’ll find them useful too.

Updating the PATH variable

PATH is the user environment variable responsible for the directories where Unix shell looks for executable commands every time you type something. Quite often you get a “file not found” error not because there isn’t such a command installed in your OS, but simply because your PATH variable has not a corrept path to a directory with that command.

Here’s an example from a standard user on one of my Red Hat Enterprise Linux systems.

I like the runlevel command, it’s quite simple and can be useful in scripts. When I attempt to run it, here’s what happens:

redhat$ runlevel
bash: runlevel: command not found

Like I said earlier, it’s most likely because my PATH variable doesn’t have the /sbin directory which is where this command resides. Let’s confirm this:

redhat$ echo $PATH
/usr/lib/qt-3.3/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin

Now it’s time to update the PATH and append the /sbin path to it:

redhat$ PATH=$PATH:/sbin

runlevel command will be found now and here’s a proof:

redhat$ runlevel
N 5

Updating the MANPATH variable

Another good example is the MANPATH variable, which contains the list of directories with manpages which man command uses.

If some command doesn’t have its man page in neither of the directories specified by MANPATH, you’ll get an error.

Here’s an example from one of the Solaris systems, I’m looking for a Veritas vxprint command man page:

solaris$ man vxprint
No manual entry for vxprint.
solaris$ echo $MANPATH
/usr/man:/opt/sfw/man:/usr/perl5/man

Let’s add the /opt/VRTS/man to the MANPATH variable:

bash-2.03# MANPATH=$MANPATH:/opt/VRTS/man
solaris$ echo $MANPATH
/usr/man:/opt/sfw/man:/usr/perl5/man:/opt/VRTS/man

It’s bound to work now:

solaris$ man vxprint
Reformatting page.  Please Wait... done
...

That’s all I wanted to share with you today. Hope you liked the tip, and as always – feel free to ask any questions!

See also

• Environment variables in Unix
• Using variables in shell scripts
• Unix scripts: basic arithmetic operations
  
• Unix commands
• Unix Basic Commands

I’m finally back from my holidays and thrilled to be sharing next of my Unix tips with you!

Today I’d like to talk about parsing text files in Unix shell scripts. This is one of the really popular areas of scripting, and there’s a few quite typical limitations which everyone comes across.

Reading text files in Unix shell

If we agree that by “reading a text file” we assume a procedure of going through all the lines found in a clear text file with a view to somehow process the data, then cat command would be the simplest demonstration of such procedure:

redhat$ cat /etc/redhat-release
Red Hat Enterprise Linux Client release 5 (Tikanga)

As you can see, there’s only one line in the /etc/redhat-release file, and we see what this line is.

But if you for whatever reason wanted to read this file from a script and assign the whole release information line to a Unix variable, using cat output would not work as expected:

bash-3.1$ for i in `cat /etc/redhat-release`; do echo $i; done;
RedHat
Enterprise
Linux
Client
release
5
(Tikanga)

Instead of reading a line of text from the file, our one-liner splits the line and outputs every word on a separate line of the output. This happens because of the shell syntax parsing – Unix shells assume space to be a delimiter of various elements in a list, so when you do a for loop, Unix shell interpreter treats each line with spaces as a list of elements, splits it and returns elements one by one.

How to read text files line by line

Here’s what I decided: if I can’t make Unix shell ignore the spaces between words of each line of text, I’ll disguise these spaces. Since my solution was getting pretty bulky for a one-liner, I’ve made it into a script. Here it is:

bash-3.1$ cat /tmp/cat.sh
#!/bin/sh
FILE=$1
UNIQUE='-={GR}=-'
#
if [ -z "$FILE" ]; then
        exit;
fi;
#
for LINE in `sed "s/ /$UNIQUE/g" $FILE`; do
        LINE=`echo $LINE | sed "s/$UNIQUE/ /g"`;
        echo $LINE;
done;

As you can see, I’ve introduced an idea of a UNIQUE variable, something containing a unique combination of characters which I can use to replace spaces in the original string. This variable needs to be a unique combination in a context of your text files, because later we turn the string back into its original version, replacing all the instances of $UNIQUE text with plain spaces.

Since most of the needs of mine required such functionality for a relatively small text files, this rather expensive (in terms of CPU cycles) approach proved to be quite usable and pretty fast.

Update: please see comments to this post for a much better approach to the same problem. Thanks again, Nails!

Here’s how my script would work on the already known /etc/redhat-release file:

bash-3.1$ /tmp/cat.sh /etc/redhat-release
Red Hat Enterprise Linux Client release 5 (Tikanga)

Exactly what I wanted! Hopefully this little trick will save some of your time as well. Let me know if you like it or know an even better one yourself!

Related books

If you want to learn more, here’s a great book:

See also:

• Using variables in scripts
• Unix scripting: time and date

As I was writing the Time and date in Unix scripts post last week, I’ve realized that there’s one really useful feature of Unix shell scripting which I haven’t covered on my pages here: it’s the double bracket parenthesis which allows you to evaluate arithmetic expressions.

Basic arithmetic calculations in Unix scripts

In most scenarios, your script will need only basic arithmetic operations: summing or subtracting numbers, multiplying or dividing them. Using only these operations, you’ll be able to implement all sorts of counters and calculate percentages or any other progress indicators you may need.

So far, I’ve only shown you this way of evaluating arithmetic expressions in Unix shell (yes, it’s not a script – you can just type commands one after another right in your shell prompt):

ubuntu$ START=1
ubuntu$ FINISH=5
ubuntu$ ELAPSED=`expr $FINISH - $START`
ubuntu$ echo $ELAPSED
4

But an even better way to evaluate arithmetic expressions is to use the built-in functionality of double brackets. Just continue typing in your shell:

ubuntu$ ELAPSED2=$(($FINISH - $START))
ubuntu$ echo $ELAPSED2
4

You see, everything that is inside double brackets gets evaluated and then this calculated value is assigned to the variable. I’m using a different variable – ELAPSED2 – to clearly show that this calculation has nothing to do with the original ELAPSED variable.

Assignment operator

If you wonder about the theory behind such a way of evaluating expressions, I should talk a bit about using variables in Unix shells.

As you probably remember, when you’re assigning your variable a new value, you’re simply specifying the name of the variable to the left of the expression, then use the = sign (it’s not just a symbol but an assignment operator by the way), and to the right of this operator goes the expression which represents the new value:

ubuntu$ MYVAR=123

Variable substitution operator

As you also remember, when you want to access the value of the variable, for example to print it out, you use the variable name with a leading $ sign. That’s because $, once again, isn’t just a fancy symbol, but in fact a special character in Bash – it’s a variable substitution operator, which means your Unix script substitutes the variable name you’re specifying in the script with the value of the variable:

ubuntu$ echo $MYVAR
3

Using the substitution operator ($) before the name of a variable is crucial. If you omit the $ sign, your shell will think you have simply passed it a text string, and will not recognize it to be a name of the variable:

ubuntu$ echo MYVAR
MYVAR

Expression evaluation explained

Coming back to arithmetic evaluations, you can probably see now that a line similar to this:

ubuntu$ ELAPSED=$(($FINISH - $START))

uses the variable substitution operator ($) and it’s therefore only logical that the further specified expression gets evaluated and the ELAPSED variable gets assigned with the resulting value.

Once again, using $ sign is crucial as without it your shell will not understand you:

ubuntu$ ELAPSED=(($FINISH - $START))
sh: syntax error near unexpected token `('

So, what do you think? I hope this makes sense to you, and will be sure to use this technique from now on – once you get used to it, it’s really easy to write and read shell scripts based on this functionality.

Let me know if there’s anything else you’d like to hear an explanation of! I’ll do my best! 🙂

See also:

• Basics of BASH scripting
• Using variables in Unix scripts

If you have followed this blog for a while, you should remember how to use variables in Unix shell scripts.

Going further, I’d like to show you some basics of working with time and date in your scripts – generating all sorts of timestamps and timing some parts of your script for reporting purposes.

The history of (Unix) time

In case you didn’t know, time in Unix starts with a Unix epoch, sometimes also referred to as POSIX epoch. The Unix epoch is the time 00:00:00 UTC on January 1, 1970.

Most of Unix-like system today store and calculate this time according to Unix epoch, which means they all have an internal counter which counts the number of seconds elapsed since midnight of January 1, 1970.

As you can imagine, this is a pretty big number (it passed 1,000,000,000 seconds back in 2001, just so that you know), and reporting in directly would not be very useful to real humans. Because of this, most of Unix time and date reporting commands and functions do the conversion from Unix time into something more meaningful to the end user.

While for the vast majority of time and date related tasks this conversion is good, a task of timing certain events can sometimes benefit from using raw second counters and simply subtracting them when needed.

Getting Unix time and date

In Unix shells, the easiest way to get a current time and date is to use the date command:

ubuntu$ date
Tue Jun 10 10:46:07 IST 2008

date is a very smart command, and apart from this default behavior it supports template system for printing the current time and date – so you can use it to report only specific part of the time and date like the current hour or the day of a week or the year.

Read the man page for date (type man date in your shell prompt) to learn all the details, but this is how you use date output templates:

ubuntu$ date "+%b %d, %Y"
Jun 10, 2008

In this example, the %b parameter in this template represents the short name of the current month, %d is the day of the month, and %Y is the four-digit representation of the current year.

Timing parts of your Unix script

If you’re after timing some parts of your script, you’ll need to use two variables: one for saving the time before the start of an observed part of the script, and another one for saving the time after the same piece of code.

These two variables will be used to subtract the time and report the elapsed time in seconds, so in order to do this we’ll need the date command to report time in seconds. That’s why I’ve given you an introduction to Unix time earlier: date reports the number of seconds since the Unix epoch:

ubuntu$ date +%s
1213091896

If you run it just a few seconds later, you’ll see a different number:

ubuntu$ date +%s
1213091922

That’s why, if such values are saved and then subtracted, we’ll get the elapsed time in seconds.

Here’s a simple script showing how this is done:

#!/bin/bash
#
START=`date +%s`
echo "Script start time (Unix epoch): $START"
#
echo "- sleeping for 3 seconds..."
sleep 3
echo "- sleeping for 2 seconds more..."
sleep 2
#
FINISH=`date +%s`
echo "Script finish time (Unix epoch): $FINISH"
#
ELAPSED=`expr $FINISH - $START`
echo "Elapsed time: $ELAPSED"

And if you run it, you will see this output:

ubuntu$ /tmp/time-example.sh
Script start time (Unix epoch): 1213092131
- sleeping for 3 seconds...
- sleeping for 2 seconds more...
Script finish time (Unix epoch): 1213092131
Elapsed time: 5

That’s all I wanted to show you today. In future posts, I’ll show you a few more things you can do regarding timing and timestamps in Unix. Until then – good luck with your scripting, and feel free to ask if you need any more help!

See Also

• Unix epoch
• Epoch time
• Converting time with Perl

Any Unix shell script longer than a line will most likely involve using variables. Variables are used to store temporary values to simply using them in various Unix commands of your script. The beauty of using variables is that they can be evaluated and set ones, but then reused as many times as you like without your shell interpreter having to re-evaluate them again.

Defining a variable in Unix shell

To specify a value for a variable, you need to decide on the variable name – can be any word or combination of English alphabet symbols and digits, and specify the value.

In Bourne shell (sh), Bourne Again Shell (bash) and Korn Shell (ksh), here’s how you would define a new variable and assign it a value:

CONFIG_FILE="/etc/myfile"

In C-Shell (csh), it’s done like this:

setenv CONFIG_FILE "/etc/myfile"

Basic variables usage

To access the value of a variable, you need to use the same variable name, but with a dollar sign in front of it, like this:

$CONFIG_FILE

Important: to set a new value to the variable, you use just the variable name like CONFIG_FILE, but to access this value later you need to use the $CONFIG_FILE form.

The most basic way to use variables is to assign them constant values at the beginning of your script. This is a good way to define locations of standard files or directories your script will work with, etc:

#!/bin/sh
#
CONFIG_FILE=/etc/myfile
MY_DIR=/etc
echo $CONFIG_FILE

Using output of Unix commands to set variables

One of the best things about shell scripting is that it’s very easy to use any Unix command to generate the output and use it to set the variable.

In this example, I’m running a date command and saving its output as values for my variables:

ubuntu$ cat /tmp/1.sh

#!/bin/sh
#
STARTED=`date`
sleep 5
FINISHED=`date`
#
echo "Script start time: $STARTED"
echo "Script finish time: $FINISHED"

If I run this simple script, I see the following:

ubuntu$ /tmp/1.sh
Script start time: Wed May 7 04:56:51 CDT 2008
Script finish time: Wed May 7 04:56:56 CDT 2008
The same approach can be used for practically any scenario.

Here’s an example of using uname command to extract some useful information about our system:

#!/bin/sh
#
STARTED=`date`
NODE=`uname -n`
OS=`uname -o`
CPU=`uname -p`
FINISHED=`date`
#
echo "Nodename: $NODE"
echo "OS type: $OS"
echo "Processor: $CPU"
echo "Script start time: $STARTED"
echo "Script finish time: $FINISHED"

And this is how it works:

ubuntu$ /tmp/1.sh
Nodename: ubuntu
OS type: GNU/Linux
Processor: unknown
Script start time: Wed May  7 05:05:31 CDT 2008
Script finish time: Wed May  7 05:05:31 CDT 2008

That’s it for today! Let me know what next topic about Unix shell scripting you’d like to see covered, and I’ll do my best to explain it in the coming posts.

Related books

If you want to learn more, here’s a great book: