Bash Scripts – Examples

I find it most useful when I approach any learning with a clear goal (or at least a specific enough task) to accomplish.

Here’s a list of simple Bash scripts that I think could be a useful learning exercise:

  • what’s your name? – asks for your name and then shows a greeting
  • hello, world! (that also shows hostname and list server IP addresses)
  • write a system info script – that shows you a hostname, disks usage, network interfaces and maybe system load
  • directory info – script that counts disks space taken up by a directory and shows number of files and number of directories in it
  • users info – show number of users on the system, their full names and home directories (all taken from /etc/passwd file)
  • list virtual hosts on your webserver – I actually have this as a login greeting on my webservers – small script that highlights which websites (domains) your server has in web server (Apache or Nginx) configuration.

Do you have any more examples of what you’d like to do in a Linux shell? If not, I’ll just start with the examples above. The plan is to replace each example name in the list above with a link to the Unix Tutorial post.




Using Dropbox with Unix

Although last week saw some pretty exciting developments in the cloud storage (Google Drive announcement and SkyDrive free 25Gb space), the truth is that Dropbox is still the king of the cloud storage hill – it’s hands down the easiest to use and integrate.

I’ve been a Dropbox user for a few years now, but have started using it actively only in the last 12 months or so. It’s been an invaluable tool for me thanks to its integration with 1Password, the password tool of my choice. Dropbox also helps with lots of day-to-day tasks and thats why I decided it’s time to share some of the tips.

Having used Dropbox extensively on Windows systems (XP on laptop and Win7 on desktops), I’ve recently moved on to using Dropbox with my Mac OSX desktop and Linux hosting.

So here are the top tips for using Dropbox with Unix – each one does wonders for me and so I hope you like them as well.

Important: If you’re not a Dropbox user yet, please use this link to sign up – it means I’ll get a small bonus (extra 500MB to my free account) for referring you.

Storing all the common apps and tools in Dropbox

Dropbox is really smart when it comes to uploading your files into the cloud storage and making it universally accessible across all the devices that you choose to pair with your Dropbox account.

One thing I particularly like using Dropbox for is storing the latest (or sometimes not the latest but verified to be fully working) versions of apps and tools I find handy to have on my desktops. In addition to having installers for all your favourite tools avaialble on each workstation, Dropbox account is also handy for simply storing all the necessary software in one location. When traveling, for example, I can open my Dropbox account and safely download the exact version of a particular tool that I need. It saved me a lot of time because I don’t have to go to each website and search for that download link.

Syncronizing scripts and config files between hosting systems

I have a dedicated server and use it for running a number of Ubuntu VMs. I’ve created a separate Dropbox account for my hosting needs, and this means that I now have 2.5Gb of space available for my VMs to exchange files or store immediate backups. Because Dropbox takes care of synchronizing all the content (and it has a LAN sync feature meaning VMs transfer files directly to each other instead of uploading back to the Dropbox site), it’s super easy and super fast to have a particular script updated and deployed to multiple systems.

I’m not quite there yet with actually running stuff like important automation or whole websites straight from Dropbox directory, but I use it for deploying scripts and configs all the time – once I get something working properly on one VM, I can then hope from one system to another and run the same set of commands against the files which are synchronized by Dropbox.

Transferring files to and from my hosting

This is a very recent addition to the things I do with Dropbox, but it’s an incredibly useful one. Having setup a separate Dropbox account for hosting, I shared one of the folders with my personal Dropbox account, and this means that transferring any files to and from my hosting had gotten to be this much easier. By putting a file into a local directory on my desktop, I have it accessible accross all the VMs on my hosting within seconds.

Likewise, if I’m reading logs or working on updating a particular config file, I can always copy it into Dropbox directory and have it synced back to my desktop.

Prior to this setup I had to rely on scp (passwordless logins using passphrase), and although it was pretty convenient to use, Dropbox approach is much more robust. Because files appear to be local, you get to work with them and manage directories as you like. You don’t have to remember the directory tree structure or follow any naming conventions – your files are the same across all the systesm and you don’t have to remeber to always sync.

Keeping backups of DBs or websites in Dropbox

Since majority of my websites are publicly available blogs, I don’t consider most of the backups to be a sensitive information. To be clear, I don’t store my passwords (wp-config.php file or htpasswd files) in Dropbox account, but everything else gets copied into it as a first level backup. I also have been doing automatic backups to Amazon’s S3 storage for about 5 years now, this means I can recover from most disasters quickly enough.

The reason Dropbox wins is because I don’t have to pay for each minor transfer or for storing an extra gigabyte or two – and yes, every little helps even though Amazon’s services are quite affordable. Another major reason I started doing backups to Dropbox is because it’s a local directory – I don’t have to use any extra tools to access all the backups in a simple directories/files structure. With Amazon’s S3 it’s also possible but setup is not as trivial.

Using Dropbox for controlling Unix systems remotely

With a few minutes and a really simple script, it’s possible to setup your own mission control for all the VMs in your hosting.

For example, if you create a cronjob which looks for a particular file, you can control which DB server your systems will connect to or which directory you’ll get the latest important log file copied into.

I’m also playing with services management based on the Dropbox account. If there’s a file present, I keep a particular service running. As soon as the file is gone, my cronjob gracefully stops the service. A slightly more sophisticated approach involves storing services names and system names associations in a Dropbox synchronized file – this allows for more flexibility as I can specify which service I want to be running on which nodes.

Sure enough, this isn’t the most straightforward way to manage your system, but such an approach can be used on the go from your iPhone. For example, I can restart a webserver by just touching a file from my iPhone, while previously I would have to find the nearest computer I trust, download SSH client, connect to the box and only then fix the problem.

Have I convinced you enough? Did you like any tips, or do you have some more perhaps? Let me know in the comments section!

pS: if you don’t have a Dropbox account or perhaps if I persuaded you to create a separate one for your hosting – please use this link so that I get some extra Dropbox space for referring. Thanks!

 




Unix scripts: how to sum numbers up

If you’re ever thought of summing up more than two numbers in shell script, perhaps this basic post will be a good start for your Unix scripting experiments.

Basic construction for summing up in shell scripts

In my Basic arithmetic operations in Unix shell post last year, I’ve shown you how to sum up two numbers:

SUM=$(($NUMBER1 + $NUMBER2)

using the same approach, it’s possible to calculate sums of as many numbers as you like, if you use one of the loops available in your shell.

Before we get started, let’s create a simple file with numbers we’ll work with:

ubuntu$ for i in 1 2 3 4 5 6 7 8 9; do echo $i >> /tmp/nums; done;
ubuntu$ cat /tmp/nums
1
2
3
4
5
6
7
8
9

Using a while loop to sum numbers up in Unix

Here’s an example of how you can use a while loop in Unix shell for summing numbers up. Save it as a /tmp/sum.sh script, and don’t forget to do chmod a+x /tmp/sum.sh so that you can run it!

#!/bin/sh
#
SUM=0
#
while read NUM
do
        echo "SUM: $SUM";
        SUM=$(($SUM + $NUM));
        echo "+ $NUM: $SUM";
done < /tmp/nums

Here’s how the output will look when you run it:

ubuntu$ /tmp/sum.sh
SUM: 0
+ 1: 1
SUM: 1
+ 2: 3
SUM: 3
+ 3: 6
SUM: 6
+ 4: 10
SUM: 10
+ 5: 15
SUM: 15
+ 6: 21
SUM: 21
+ 7: 28
SUM: 28
+ 8: 36
SUM: 36
+ 9: 45

Of course, your data will most probably will be in a less readable form, so you’ll have to do a bit of parsing before you get to sum the numbers up, but the loop will be organized in the same way.

That’s it for today, enjoy and feel free to ask questions!

See also:




How To Change Ownership of Files and Directories in Unix

I’ve just been asked a question about changing the ownership of files from one Unix user to another, and thought it probably makes sense to have a quick post on it.

File ownership in Unix

Just to give you a quick reminder, I’d like to confirm that every single file in Unix belongs to some user and some group. There simply isn’t a way to create a file without assigning ownership. I’ve briefly touched the topic of confirming file ownership in Unix before, so today I will simply build on that and show you how to change ownership of files.

Here’s a setup for today: I have created a temporary directory with a few files and made myself the owner of all the files:

ubuntu$ ls -al /home/greys/example/
total 12
drwxr-xr-x  3 greys admin 4096 Feb  9 03:55 .
drwxr-xr-x 13 greys greys 4096 Feb  9 03:54 ..
drwxr-xr-x  2 greys admin 4096 Feb  9 03:55 dir1
-rw-r--r--  1 greys admin    0 Feb  9 03:54 file1
-rw-r--r--  1 greys admin    0 Feb  9 03:55 file2

As you can see from this listing, the owner (third field in each line) is my username – greys. The next field is a Unix group of each file’s owner – admin in my example.

Changing owner of a file in Unix

Changing file ownership means only updating the association between a Unix user and a file, and nothing else. When you’re changing the owner of a file, no data contained in a file is changed.

To change the owner of a file, you need to use the chown command (easy enough to remember: CHange OWNer – chown), with the following syntax:

ubuntu$ chown nobody file1

In this command, nobody is the username of the new owner for a list of files. In my example, the only file we’d like to change ownership for is file1.

It is important to realize that you can only change file ownership as a super-user (root). Any regular Unix user cannot change the ownership of any file, and I’d like to explain why.

Indeed, some people are surprised: if I’m the owner of a given file, why can’t I change the ownership for it? That’s because transferring the ownership will mean some other Unix user will become the owner of the file(s) in question. So changing ownership is like making a decision not only for yourself, but for the new owner of the files.This is only something a super-user – special administrative account in Unix – can do.

The same logic applies to other people not being able to become owners of your files, even if they’re willing to assume the new responsibilities of owning files. They cannot revoke your ownership, because each Unix user is only allowed to make decisions and take actions on his/her own behalf.

That’s why you will probably see an error like this if you attempt to change ownership of a file as your own regular Unix user:

ubuntu$ id
uid=1000(greys) gid=113(admin) groups=33(www-data),113(admin)
ubuntu$ chown nobody file1
chown: changing ownership of `file1': Operation not permitted

But if we become root:

ubuntu$ sudo -i
[sudo] password for greys:
ubuntu#

… we’ll have no problem changing owners for any files:

ubuntu# cd /home/greys/example
ubuntu# chown nobody file1
ubuntu# ls -l file1
-rw-r--r-- 1 nobody admin 0 Feb  9 03:54 file1

Changing owner for multiple files

If you’re going to change owner of a few files, this can easily be done using either a full list of files or a mask.

First, here’s an example of updating ownership for a specified list of files (and as you can see, directories as well):

ubuntu# chown nobody file2 dir1
ubuntu# ls -al
total 12
drwxr-xr-x  3 greys  admin 4096 Feb  9 03:55 .
drwxr-xr-x 13 greys  greys 4096 Feb  9 03:54 ..
drwxr-xr-x  2 nobody admin 4096 Feb  9 03:55 dir1
-rw-r--r--  1 nobody admin    0 Feb  9 03:54 file1
-rw-r--r--  1 nobody admin    0 Feb  9 03:55 file2

IMPORTANT: here’s one thing which is often forgotten: when you’re changing an owner of a directory, this DOES NOT automatically change owner of all the files which already exist in this directory. So, if we check the file3 in dir1 after the example above, we can see that even though dir1 now belongs to user nobody, file3 in it still belongs to me:

ubuntu# ls -l dir1/file3
-rw-r--r-- 1 greys admin 0 Feb  9 03:55 dir1/file3

If your intention is to change ownership of all the files and directories of a certain location in your filesystem, you need to use a -R option of the chown command, which means recursive ownership change:

ubuntu# chown -R nobody dir1
ubuntu# ls -l dir1/file3
-rw-r--r-- 1 nobody admin 0 Feb  9 03:55 dir1/file3

And just to further demonstrate this, I’m going to change owner of all the files and directories in /home/greys/example directory back to my own username, greys:

ubuntu# chown -R greys /home/greys/example/
ubuntu# ls -l /home/greys/example/
total 4
drwxr-xr-x 2 greys admin 4096 Feb  9 03:55 dir1
-rw-r--r-- 1 greys admin    0 Feb  9 03:54 file1
-rw-r--r-- 1 greys admin    0 Feb  9 03:55 file2

Changing group ownership for a file

Similar to the chown command, there’s a command specifically helping you with changing not the owner (user) of a file.

IMPORANT: unlike chown command, chgrp can be used by non-privileged (regular) users of a system. So you don’t have to be root if you want to change a group ownership for some of your files, provided that you’re changing the ownership to a group you’re a member of.

For example, I’m a member of quite a few groups on one of my Ubuntu servers:

ubuntu$ id greys
uid=1000(greys) gid=1000(greys) groups=1000(greys),4(adm),20(dialout),24(cdrom),46(plugdev),114(lpadmin),115(sambashare),116(admin)

Now, if I create a new file, it will by default belong to my primary group (called greys, just like my username):

ubuntu$ touch file
ubuntu$ ls -al file
-rw-r--r-- 1 greys greys 0 2012-09-20 10:48 file

I can now change group ownership of this file, in this case to a group admin, which I’m also part of.

ubuntu$ chgrp admin file

and this is just to confirm that the change actualyl happened:

ubuntu$ ls -al file
-rw-r--r-- 1 greys admin 0 2012-09-20 10:48 file

That’s it for today, good luck with changing file owners on your Unix system!

Recommended books:

See also:




List Groups A User Belongs To In Unix

Still seeing this request quite frequently in my website logs, I thought I’d expand the topic a bit more, given the fact that I’ve somehow left out the most obvious way to confirm the group membership for a Unix user.

Group membership in Unix

Every Unix-like OS is bound to have a group command, which is aimed to help you confirm Unix group membership for any user known to your system.

The easiest it to run it and specify a username in command line. The result is a list of Unix groups:

ubuntu$ groups greys
greys : admin www-data

If you’re looking at confirming the membership for a few users, you can specify usernames in the same command line:

ubuntu$ groups greys root
greys : admin www-data
root : root

There you go – hopefully this satisfies your interest. Good luck with Unix experiments!

See also:




Unix scripts: basic arithmetic operations

As I was writing the Time and date in Unix scripts post last week, I’ve realized that there’s one really useful feature of Unix shell scripting which I haven’t covered on my pages here: it’s the double bracket parenthesis which allows you to evaluate arithmetic expressions.

Basic arithmetic calculations in Unix scripts

In most scenarios, your script will need only basic arithmetic operations: summing or subtracting numbers, multiplying or dividing them. Using only these operations, you’ll be able to implement all sorts of counters and calculate percentages or any other progress indicators you may need.

So far, I’ve only shown you this way of evaluating arithmetic expressions in Unix shell (yes, it’s not a script – you can just type commands one after another right in your shell prompt):

ubuntu$ START=1
ubuntu$ FINISH=5
ubuntu$ ELAPSED=`expr $FINISH - $START`
ubuntu$ echo $ELAPSED
4

But an even better way to evaluate arithmetic expressions is to use the built-in functionality of double brackets. Just continue typing in your shell:

ubuntu$ ELAPSED2=$(($FINISH - $START))
ubuntu$ echo $ELAPSED2
4

You see, everything that is inside double brackets gets evaluated and then this calculated value is assigned to the variable. I’m using a different variable – ELAPSED2 – to clearly show that this calculation has nothing to do with the original ELAPSED variable.

Assignment operator

If you wonder about the theory behind such a way of evaluating expressions, I should talk a bit about using variables in Unix shells.

As you probably remember, when you’re assigning your variable a new value, you’re simply specifying the name of the variable to the left of the expression, then use the = sign (it’s not just a symbol but an assignment operator by the way), and to the right of this operator goes the expression which represents the new value:

ubuntu$ MYVAR=123

Variable substitution operator

As you also remember, when you want to access the value of the variable, for example to print it out, you use the variable name with a leading $ sign. That’s because $, once again, isn’t just a fancy symbol, but in fact a special character in Bash – it’s a variable substitution operator, which means your Unix script substitutes the variable name you’re specifying in the script with the value of the variable:

ubuntu$ echo $MYVAR
3

Using the substitution operator ($) before the name of a variable is crucial. If you omit the $ sign, your shell will think you have simply passed it a text string, and will not recognize it to be a name of the variable:

ubuntu$ echo MYVAR
MYVAR

Expression evaluation explained

Coming back to arithmetic evaluations, you can probably see now that a line similar to this:

ubuntu$ ELAPSED=$(($FINISH - $START))

uses the variable substitution operator ($) and it’s therefore only logical that the further specified expression gets evaluated and the ELAPSED variable gets assigned with the resulting value.

Once again, using $ sign is crucial as without it your shell will not understand you:

ubuntu$ ELAPSED=(($FINISH - $START))
sh: syntax error near unexpected token `('

So, what do you think? I hope this makes sense to you, and will be sure to use this technique from now on – once you get used to it, it’s really easy to write and read shell scripts based on this functionality.

Let me know if there’s anything else you’d like to hear an explanation of! I’ll do my best! 🙂

See also:




Unix Scripting: Time and Date

If you have followed this blog for a while, you should remember how to use variables in Unix shell scripts.

Going further, I’d like to show you some basics of working with time and date in your scripts – generating all sorts of timestamps and timing some parts of your script for reporting purposes.

The history of (Unix) time

In case you didn’t know, time in Unix starts with a Unix epoch, sometimes also referred to as POSIX epoch. The Unix epoch is the time 00:00:00 UTC on January 1, 1970.

Most of Unix-like system today store and calculate this time according to Unix epoch, which means they all have an internal counter which counts the number of seconds elapsed since midnight of January 1, 1970.

As you can imagine, this is a pretty big number (it passed 1,000,000,000 seconds back in 2001, just so that you know), and reporting in directly would not be very useful to real humans. Because of this, most of Unix time and date reporting commands and functions do the conversion from Unix time into something more meaningful to the end user.

While for the vast majority of time and date related tasks this conversion is good, a task of timing certain events can sometimes benefit from using raw second counters and simply subtracting them when needed.

Getting Unix time and date

In Unix shells, the easiest way to get a current time and date is to use the date command:

ubuntu$ date
Tue Jun 10 10:46:07 IST 2008

date is a very smart command, and apart from this default behavior it supports template system for printing the current time and date – so you can use it to report only specific part of the time and date like the current hour or the day of a week or the year.

Read the man page for date (type man date in your shell prompt) to learn all the details, but this is how you use date output templates:

ubuntu$ date "+%b %d, %Y"
Jun 10, 2008

In this example, the %b parameter in this template represents the short name of the current month, %d is the day of the month, and %Y is the four-digit representation of the current year.

Timing parts of your Unix script

If you’re after timing some parts of your script, you’ll need to use two variables: one for saving the time before the start of an observed part of the script, and another one for saving the time after the same piece of code.

These two variables will be used to subtract the time and report the elapsed time in seconds, so in order to do this we’ll need the date command to report time in seconds. That’s why I’ve given you an introduction to Unix time earlier: date reports the number of seconds since the Unix epoch:

ubuntu$ date +%s
1213091896

If you run it just a few seconds later, you’ll see a different number:

ubuntu$ date +%s
1213091922

That’s why, if such values are saved and then subtracted, we’ll get the elapsed time in seconds.

Here’s a simple script showing how this is done:

#!/bin/bash
#
START=`date +%s`
echo "Script start time (Unix epoch): $START"
#
echo "- sleeping for 3 seconds..."
sleep 3
echo "- sleeping for 2 seconds more..."
sleep 2
#
FINISH=`date +%s`
echo "Script finish time (Unix epoch): $FINISH"
#
ELAPSED=`expr $FINISH - $START`
echo "Elapsed time: $ELAPSED"

And if you run it, you will see this output:

ubuntu$ /tmp/time-example.sh
Script start time (Unix epoch): 1213092131
- sleeping for 3 seconds...
- sleeping for 2 seconds more...
Script finish time (Unix epoch): 1213092131
Elapsed time: 5

That’s all I wanted to show you today. In future posts, I’ll show you a few more things you can do regarding timing and timestamps in Unix. Until then – good luck with your scripting, and feel free to ask if you need any more help!

See Also