How To Monitor Linux Memory Usage with Watch Command

Hi all, today I’m going to teach you not one, but two really cool things in one post! First, I’ll introduce you to advanced memory usage stats available on Linux systems through /proc/meminfo file, and then I’ll explain the basics of using the watch command.

Memory usage with /proc/meminfo

As you know, quite a few Unix-like systems use the so-called pseudo file systems like /proc. It’s not a real filesystem, but just a convenient representation of processes managed by your Unix OS. In Linux systems, this directory also contains quite a few files allowing you to access various information about your system. /proc/meminfo is one of such files, it gives you access to most of the memory usage stats.

To get a snapshot of the current state of memory usage on your Linux system, simply cat the /proc/meminfo file:

ubuntu$ cat /proc/meminfo
MemTotal:       523008 kB
MemFree:         35336 kB
Buffers:         85560 kB
Cached:         137220 kB
SwapCached:      24480 kB
Active:         327420 kB
Inactive:        91308 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       523008 kB
LowFree:         35336 kB
SwapTotal:     1048568 kB
SwapFree:       998960 kB
Dirty:             504 kB
Writeback:           0 kB
Mapped:         212232 kB
Slab:            39140 kB
CommitLimit:   1310072 kB
Committed_AS:   655992 kB
PageTables:       4748 kB
VmallocTotal: 34359738367 kB
VmallocUsed:       628 kB
VmallocChunk: 34359737739 kB

This probably gives you more information about memory usage that you’ll ever want to know, but there’s quite a few really useful stats there like the MemFree or SwapFree ones, they’re useful for making sure your OS environment is healthy enough in terms of having plenty of free memory for efficient operation.

Using watch command to track progress

watch command is a really neat tool which does a simple but incredibly useful thing: it repeatedly runs a given command line and shows you the output. So, you’re effectively monitoring a progress of some process by watching relevant files.

The default interval is 2, which gives enough dynamics for most of the needs.

Here’s how you use this command:

ubuntu$ watch cat /proc/meminfo

So it’s the same command we used in previous example, cat /proc/meminfo, but this time we’re asking the watch command to re-run the command every 2 seconds and show us the output.

The result of running a watch command is going to be a constantly refreshed console showing something like this:

Every 2.0s: cat /proc/meminfo       Fri Feb 13 03:51:01 2009

MemTotal:       523008 kB
MemFree:         46396 kB
Buffers:         82636 kB
Cached:         131044 kB
SwapCached:      24480 kB
Active:         308512 kB
Inactive:        99372 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       523008 kB
LowFree:         46396 kB
SwapTotal:     1048568 kB
SwapFree:       998960 kB
Dirty:             832 kB
Writeback:           0 kB
Mapped:         211076 kB
Slab:            39132 kB
CommitLimit:   1310072 kB
Committed_AS:   654860 kB
PageTables:       4856 kB
VmallocTotal: 34359738367 kB
VmallocUsed:       628 kB
VmallocChunk: 34359737739 kB

This output gets refreshed every 2 seconds, so the numbers shown are constantly updated.

That’s it for today! There are limitless possibilities for monitoring various processes using watch command and I’ll be sure to cover them in the future, but for now – have a great weekend and hope Friday 13th turns out great!

See also:



How To Change Ownership of Files and Directories in Unix

I’ve just been asked a question about changing the ownership of files from one Unix user to another, and thought it probably makes sense to have a quick post on it.

File ownership in Unix

Just to give you a quick reminder, I’d like to confirm that every single file in Unix belongs to some user and some group. There simply isn’t a way to create a file without assigning ownership. I’ve briefly touched the topic of confirming file ownership in Unix before, so today I will simply build on that and show you how to change ownership of files.

Here’s a setup for today: I have created a temporary directory with a few files and made myself the owner of all the files:

ubuntu$ ls -al /home/greys/example/
total 12
drwxr-xr-x  3 greys admin 4096 Feb  9 03:55 .
drwxr-xr-x 13 greys greys 4096 Feb  9 03:54 ..
drwxr-xr-x  2 greys admin 4096 Feb  9 03:55 dir1
-rw-r--r--  1 greys admin    0 Feb  9 03:54 file1
-rw-r--r--  1 greys admin    0 Feb  9 03:55 file2

As you can see from this listing, the owner (third field in each line) is my username – greys. The next field is a Unix group of each file’s owner – admin in my example.

Changing owner of a file in Unix

Changing file ownership means only updating the association between a Unix user and a file, and nothing else. When you’re changing the owner of a file, no data contained in a file is changed.

To change the owner of a file, you need to use the chown command (easy enough to remember: CHange OWNer – chown), with the following syntax:

ubuntu$ chown nobody file1

In this command, nobody is the username of the new owner for a list of files. In my example, the only file we’d like to change ownership for is file1.

It is important to realize that you can only change file ownership as a super-user (root). Any regular Unix user cannot change the ownership of any file, and I’d like to explain why.

Indeed, some people are surprised: if I’m the owner of a given file, why can’t I change the ownership for it? That’s because transferring the ownership will mean some other Unix user will become the owner of the file(s) in question. So changing ownership is like making a decision not only for yourself, but for the new owner of the files.This is only something a super-user – special administrative account in Unix – can do.

The same logic applies to other people not being able to become owners of your files, even if they’re willing to assume the new responsibilities of owning files. They cannot revoke your ownership, because each Unix user is only allowed to make decisions and take actions on his/her own behalf.

That’s why you will probably see an error like this if you attempt to change ownership of a file as your own regular Unix user:

ubuntu$ id
uid=1000(greys) gid=113(admin) groups=33(www-data),113(admin)
ubuntu$ chown nobody file1
chown: changing ownership of `file1': Operation not permitted

But if we become root:

ubuntu$ sudo -i
[sudo] password for greys:
ubuntu#

… we’ll have no problem changing owners for any files:

ubuntu# cd /home/greys/example
ubuntu# chown nobody file1
ubuntu# ls -l file1
-rw-r--r-- 1 nobody admin 0 Feb  9 03:54 file1

Changing owner for multiple files

If you’re going to change owner of a few files, this can easily be done using either a full list of files or a mask.

First, here’s an example of updating ownership for a specified list of files (and as you can see, directories as well):

ubuntu# chown nobody file2 dir1
ubuntu# ls -al
total 12
drwxr-xr-x  3 greys  admin 4096 Feb  9 03:55 .
drwxr-xr-x 13 greys  greys 4096 Feb  9 03:54 ..
drwxr-xr-x  2 nobody admin 4096 Feb  9 03:55 dir1
-rw-r--r--  1 nobody admin    0 Feb  9 03:54 file1
-rw-r--r--  1 nobody admin    0 Feb  9 03:55 file2

IMPORTANT: here’s one thing which is often forgotten: when you’re changing an owner of a directory, this DOES NOT automatically change owner of all the files which already exist in this directory. So, if we check the file3 in dir1 after the example above, we can see that even though dir1 now belongs to user nobody, file3 in it still belongs to me:

ubuntu# ls -l dir1/file3
-rw-r--r-- 1 greys admin 0 Feb  9 03:55 dir1/file3

If your intention is to change ownership of all the files and directories of a certain location in your filesystem, you need to use a -R option of the chown command, which means recursive ownership change:

ubuntu# chown -R nobody dir1
ubuntu# ls -l dir1/file3
-rw-r--r-- 1 nobody admin 0 Feb  9 03:55 dir1/file3

And just to further demonstrate this, I’m going to change owner of all the files and directories in /home/greys/example directory back to my own username, greys:

ubuntu# chown -R greys /home/greys/example/
ubuntu# ls -l /home/greys/example/
total 4
drwxr-xr-x 2 greys admin 4096 Feb  9 03:55 dir1
-rw-r--r-- 1 greys admin    0 Feb  9 03:54 file1
-rw-r--r-- 1 greys admin    0 Feb  9 03:55 file2

Changing group ownership for a file

Similar to the chown command, there’s a command specifically helping you with changing not the owner (user) of a file.

IMPORANT: unlike chown command, chgrp can be used by non-privileged (regular) users of a system. So you don’t have to be root if you want to change a group ownership for some of your files, provided that you’re changing the ownership to a group you’re a member of.

For example, I’m a member of quite a few groups on one of my Ubuntu servers:

ubuntu$ id greys
uid=1000(greys) gid=1000(greys) groups=1000(greys),4(adm),20(dialout),24(cdrom),46(plugdev),114(lpadmin),115(sambashare),116(admin)

Now, if I create a new file, it will by default belong to my primary group (called greys, just like my username):

ubuntu$ touch file
ubuntu$ ls -al file
-rw-r--r-- 1 greys greys 0 2012-09-20 10:48 file

I can now change group ownership of this file, in this case to a group admin, which I’m also part of.

ubuntu$ chgrp admin file

and this is just to confirm that the change actualyl happened:

ubuntu$ ls -al file
-rw-r--r-- 1 greys admin 0 2012-09-20 10:48 file

That’s it for today, good luck with changing file owners on your Unix system!

Recommended books:

See also:



How To Update atime and mtime for a File in Unix

If you remember, all files and directories in Unix filesystems have three timestamps associated with them – atime, ctime and mtime. Since questions about modifying access time (atime) and modification time (mtime) are quite frequent in my website logs, I thought I’d explain how it is done.

How to view atime, ctime and mtime

Before we go any further, I’d like to remind you that using stat command is probably the easiest way to look at all the three timestamps associated with each file:

ubuntu$ stat ./try
  File: `./try'
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: 801h/2049d      Inode: 655596      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/   greys)   Gid: (  113/   admin)
Access: 2008-11-17 05:01:16.000000000 -0600
Modify: 2008-11-17 05:01:16.000000000 -0600
Change: 2008-11-17 05:01:16.000000000 -0600

Even though ls command can be used to view the same times, we will depend on the stat command for today’s post simply because it shows all the times together – it’s great for explanations.

Modifying atime and mtime

There’s a very simple way to update either atime or mtime for a given file, or even both at the same time: you should use the touch command.

Here’s how it can be used to update the atime:

ubuntu$ touch -at 0711171533 ./try

The -a in the command line parameters refers to atime, while -t and the following sequence are nothing but a timestamp we want assigned to the file. In my example, 0711171533 means this:

  • 07 – year of 2007
  • 11 – November
  • 17 – 17th
  • 1533 – time of the day, 15:33 Now, if we run stat command again, you can see how the access time field got updated:
ubuntu$ stat ./try
  File: `./try'
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: 801h/2049d      Inode: 655596      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/   greys)   Gid: (  113/   admin)
Access: 2007-11-17 15:33:00.000000000 -0600
Modify: 2008-11-17 05:01:16.000000000 -0600
Change: 2008-11-17 05:01:48.000000000 -0600

Similarly,  we can set the mtime, in my particular example it’s the future – a day exactly one year from now.

-m is the command line option to specify that mtime is our main focus:

ubuntu$ touch -mt 0911171533 ./try
ubuntu$ stat ./try
  File: `./try'
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: 801h/2049d      Inode: 655596      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/   greys)   Gid: (  113/   admin)
Access: 2007-11-17 15:33:00.000000000 -0600
Modify: 2009-11-17 15:33:00.000000000 -0600
Change: 2008-11-17 05:05:41.000000000 -0600

Changing atime and mtime to the current Unix time

It’s probably useful to know that the default behavior of the

touch command is to update both access time and modification time of a file, changing them to the current time on your system. Here’s what will happen if I run touch against the same file we used in all the examples:

ubuntu$ touch ./try
ubuntu$ stat ./try
  File: `./try'
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: 801h/2049d      Inode: 655596      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/   greys)   Gid: (  113/   admin)
Access: 2008-11-17 05:09:33.000000000 -0600
Modify: 2008-11-17 05:09:33.000000000 -0600
Change: 2008-11-17 05:09:33.000000000 -0600

As you can see, all three fields have been reset to the new (current time) value. That’s it for today, I hope this solved another one of your Unix mysteries!

See also



How To Remove Files and Directories with Special Characters

Today is going to be a practical tip. If you’re managing many Unix systems, sooner or later you come across files with special characters – they can’t be deleted with rm command using standard approach and require a bit of trickery to be successfully removed.

Examples of files with special characters

Any language apart from English will probably have special characters in the alphabet, but for the purpose of today’s exercise I’ll give you more standard examples: files starting with dash (-) and hash (#) characters:

ubuntu$ ls -al
-rw-r--r-- 1 greys admin    0 Sep 25 05:50 #try
-rw-r--r-- 1 greys admin    0 Sep 25 05:48 -try

Now, if you try to access these files or remove them, you will get errors:

ubuntu$ cat -try
cat: invalid option -- r
Try `cat --help' for more information.
ubuntu$ rm -try
rm: invalid option -- t
Try `rm ./-try' to remove the file `-try'.
Try `rm --help' for more information.

These errors happen because commands treat file names as command line options because they start with dash (-).

With filenames starting with hash (#), you’ll get a different kind of error: your Unix shell will treat the rest of a filename (and anything that might follow it) as a comment because hashes are used to do it in shell scripts. That’s why your cat command will not show any error but will not finish until you Ctrl+C it:

ubuntu$ cat #try

… and if you try removing such a file, you’ll get a complaint from the rm command about missing command line parameters – because of the hash (#) sign, rm command receives no text as a parameter:

ubuntu$ rm #try
rm: missing operand
Try `rm --help' for more information.

How to remove a file when filename starts with dash (-)

First I’m gonna show you how to make your Unix shell interpret any filename directly instead of trying to analyze it as a set of command line options.

To make command ignore the leading dash (-) in a filename, use the — command line option:

ubuntu$ rm -- -try

As you can see, our file is gone:

ubuntu$ ls -al
-rw-r--r-- 1 greys admin    0 Sep 25 05:50 #try

Using backslash to escape special characters in a filename

Another option we have is to use a backslash (\), which will make shell interpreter ignore the special functionality of a character which immediately follows it. To escape the hash (#) our second file has, we should therefore do the following:

ubuntu$ rm \#try

Interesting to know: bash shell has an auto-completion functionality built in. When you type a filename, just press Tab key to make it auto-complete the name for you. Speaking of special characters in particular, quite a few of them are recognized by auto-completion and get escaped automatically.

So, if you start typing:

ubuntu $ rm #t

… and then press Tab, bash will not only auto-complete the name, but escape the leading hash (#):

ubuntu $ rm \#try

There’s a few more tricks you can use for escaping special characters, but they’re worth a separate post, so stay tuned! Until then, enjoy getting rid of annoying files with special characters in filenames!

See also:



Easy date calculations in Unix scripts with GNU date

When I was writing a post about using date command to confirm date and time in your Unix scripts, I made a note in my future posts list to cover the date calculations – finding out the date of yesterday or tomorrow, and so on. Today I’ll show you the simplest way to calculate this.

GNU date command advantage

GNU version of the date command, although supporting a common syntax, has one great option: it allows you to specify the desired date with a simple string before reporting it back. What this means is that by default this specified date is assumed to be “now”, but you can use other keywords to shift the result of the date command and thus show what date it was yesterday or a week ago:

Here’s a default date output for the current date and time:

ubuntu$ date
Fri Sep 19 08:06:41 CDT 2008

Now, the parameter for specifying desired date is -d or –date, and if you use it with the “now” or “today” parameter, you’ll get similar output:

ubuntu$ date -d now
Fri Sep 19 08:06:44 CDT 2008
ubuntu$ date -d today
Fri Sep 19 08:06:50 CDT 2008

Showing tomorrow’s date

Similarly, you can get tomorrow’s date:

ubuntu$ date -d tomorrow
Sat Sep 20 08:02:12 CDT 2008

Obviously, if you feel like specifying a format for the date, you can do it:

ubuntu$ date -d tomorrow "+%b %d, %Y"
Sep 20, 2008

Find out yesterday’s date

Again, there’s no rocket science involved in showing yesterday’s date neither:

ubuntu$ date -d yesterday "+%b %d, %Y"
Sep 18, 2008

Show a date few days away

If you’re interested in a certain date a few days or even weeks away, here’s how you can do it:

Example 1: a date 5 days ago

ubuntu$ date -d "-5 days"
Sun Sep 14 08:45:57 CDT 2008

Example 2: a day 2 weeks from now

ubuntu$ date -d "2 weeks"
Fri Oct  3 08:45:08 CDT 2008

Example 3: a day two weeks ago from now

ubuntu$ date -d "-2 weeks"
Fri Sep  5 08:45:11 CDT 2008

Extreme example: a day 50 years ago

If you’re really curious about dates in Unix, you can even make GNU date go back a few years:

ubuntu$ date -d "-50 years"
Fri Sep 19 08:47:51 CDT 1958

That’s it for today, hope you like this little discovery – having mostly worked with Solaris systems most of my career, I didn’t know my Ubuntu had this functionality bonus. Really useful!

See also:



Another way to use math expressions in shell scripts

Today I’d like to expand a bit more on the basic calculations in Unix scripts.

Use Parenthesis to simplify math expressions

In Basic Arithmetic Operations post I’ve shown you how expression evaluation can be used to calculate simple math expressions in your Unix shell:

ubuntu$ START=1
ubuntu$ FINISH=10
ubuntu$ ELAPSED=$(($FINISH - $START))
ubuntu$ echo $ELAPSED
9

Although this approach looks clean enough, there’s a way to simplify it even further if you put everything in parenthesis. In other words, the same result (ELAPSED receiving a correct value of FINISH value minus START value) can be achieved this way:

ubuntu$ ((ELAPSED=FINISH-START))
ubuntu$ echo $ELAPSED
9

It’s a matter of preference, and I used to always do such calculations the former way shown, but now that one of the readers of this blog pointed out the latter way of using it, I think I might change my habits – I actually like this way much better.

See also:



Updating Values of Your Shell Variables in Unix

If you’re just getting started with your Unix scripting or new to Unix shell, today’s little tip will make your life much easier: I’m going to show you how to update your shell variables.

Updating shell variable in Unix

If you have a text variable in your script and would like to append some text to it or somehow process this value, it’s perfectly normal to reuse the current variable value using a syntax like this:

ubuntu$ echo $VAR1
hello
ubuntu$ VAR1=$VAR1" world!"
ubuntu$ echo $VAR1
hello world!

You see? It’s this easy!

Common cases of reusing variable values in Unix shells

Most frequently, I use the technique described above to update some of my environment variables. I’m sure you’ll find them useful too.

Updating the PATH variable

PATH is the user environment variable responsible for the directories where Unix shell looks for executable commands every time you type something. Quite often you get a “file not found” error not because there isn’t such a command installed in your OS, but simply because your PATH variable has not a corrept path to a directory with that command.

Here’s an example from a standard user on one of my Red Hat Enterprise Linux systems.

I like the runlevel command, it’s quite simple and can be useful in scripts. When I attempt to run it, here’s what happens:

redhat$ runlevel
bash: runlevel: command not found

Like I said earlier, it’s most likely because my PATH variable doesn’t have the /sbin directory which is where this command resides. Let’s confirm this:

redhat$ echo $PATH
/usr/lib/qt-3.3/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin

Now it’s time to update the PATH and append the /sbin path to it:

redhat$ PATH=$PATH:/sbin

runlevel command will be found now and here’s a proof:

redhat$ runlevel
N 5

Updating the MANPATH variable

Another good example is the MANPATH variable, which contains the list of directories with manpages which man command uses.

If some command doesn’t have its man page in neither of the directories specified by MANPATH, you’ll get an error.

Here’s an example from one of the Solaris systems, I’m looking for a Veritas vxprint command man page:

solaris$ man vxprint
No manual entry for vxprint.
solaris$ echo $MANPATH
/usr/man:/opt/sfw/man:/usr/perl5/man

Let’s add the /opt/VRTS/man to the MANPATH variable:

bash-2.03# MANPATH=$MANPATH:/opt/VRTS/man
solaris$ echo $MANPATH
/usr/man:/opt/sfw/man:/usr/perl5/man:/opt/VRTS/man

It’s bound to work now:

solaris$ man vxprint
Reformatting page.  Please Wait... done
...

That’s all I wanted to share with you today. Hope you liked the tip, and as always – feel free to ask any questions!

See also



System Resources Usage When Running Unix commands

In my system administration job, I come across this particular requirement quite often: run a command, then confirm how long it took to complete the job. You guessed it right – there’s a very easy way to do it.

Using the time command

What you need to do is to use the time command, which takes as a parameter the full command line you need to run. Upon completion of the command, time will report the system resources usage during the execution. All the reported numbers are presented in a simple table.

Here’s an example: running du command to confirm the size of a directory while timing the effort and reporting the stats:

ubuntu# time du -sk /var
4228720 /var

real    0m17.747s
user    0m0.010s
sys     0m0.080s

As you can see, first the du command output is shown, and then the table with real, user and system times reported.

See also:



List Groups A User Belongs To In Unix

Still seeing this request quite frequently in my website logs, I thought I’d expand the topic a bit more, given the fact that I’ve somehow left out the most obvious way to confirm the group membership for a Unix user.

Group membership in Unix

Every Unix-like OS is bound to have a group command, which is aimed to help you confirm Unix group membership for any user known to your system.

The easiest it to run it and specify a username in command line. The result is a list of Unix groups:

ubuntu$ groups greys
greys : admin www-data

If you’re looking at confirming the membership for a few users, you can specify usernames in the same command line:

ubuntu$ groups greys root
greys : admin www-data
root : root

There you go – hopefully this satisfies your interest. Good luck with Unix experiments!

See also:



How To Determine Physical Memory Size in Linux

If you’re logged in at some remote Linux system and need to quickly confirm the amount of available memory, there’s a few commands you will find quite useful.

free – free and used memory stats

free command is the most obvious choice for a first command when it comes to your RAM.

Simply run it without any parameters, and it will show you something like this:

ubuntu# free
             total       used       free     shared    buffers     cached
Mem:       4051792    4024960      26832          0      63768    3131532
-/+ buffers/cache:     829660    3222132
Swap:      4096492      43212    4053280

For this exercise, you’re only interested in the “total” column of the first line. 4051792 confirms that my home PC seems to have around 4Gb of memory available for Ubuntu to use.

Using dmesg to check memory size as recognized by Linux kernel

dmesg command shows you the last status messages reported by your OS kernel, and since every boot procedure includes scanning the hardware and confirming the devices and resources recognized by the kernel, you can see some basic information by using dmesg.

For our purpose, we need to filter out the memory stats:

ubuntu# dmesg | grep Memory
[   18.617904] Memory: 4043492k/5242880k available (2489k kernel code, 150360k reserved, 1318k data, 320k init)

Once again, the overall amount of memory confirms that 4Gb of RAM were still found during the last time my PC booted up.

Using /proc/meminfo to confirm the RAM size

/proc/meminfo is one of the special files managed by Linux kernel. It’s a clear text presentation of the most vital memory stats of your system (this means you can do something like cat /proc/meminfo to see all the parameters)

This is what you need to do to get the total size of your physical memory:

ubuntu# grep MemTotal /proc/meminfo
MemTotal:      4051792 kB

That’s it for today, enjoy!