Using Dropbox with Unix

Although last week saw some pretty exciting developments in the cloud storage (Google Drive announcement and SkyDrive free 25Gb space), the truth is that Dropbox is still the king of the cloud storage hill – it’s hands down the easiest to use and integrate.

I’ve been a Dropbox user for a few years now, but have started using it actively only in the last 12 months or so. It’s been an invaluable tool for me thanks to its integration with 1Password, the password tool of my choice. Dropbox also helps with lots of day-to-day tasks and thats why I decided it’s time to share some of the tips.

Having used Dropbox extensively on Windows systems (XP on laptop and Win7 on desktops), I’ve recently moved on to using Dropbox with my Mac OSX desktop and Linux hosting.

So here are the top tips for using Dropbox with Unix – each one does wonders for me and so I hope you like them as well.

Important: If you’re not a Dropbox user yet, please use this link to sign up – it means I’ll get a small bonus (extra 500MB to my free account) for referring you.

Storing all the common apps and tools in Dropbox

Dropbox is really smart when it comes to uploading your files into the cloud storage and making it universally accessible across all the devices that you choose to pair with your Dropbox account.

One thing I particularly like using Dropbox for is storing the latest (or sometimes not the latest but verified to be fully working) versions of apps and tools I find handy to have on my desktops. In addition to having installers for all your favourite tools avaialble on each workstation, Dropbox account is also handy for simply storing all the necessary software in one location. When traveling, for example, I can open my Dropbox account and safely download the exact version of a particular tool that I need. It saved me a lot of time because I don’t have to go to each website and search for that download link.

Syncronizing scripts and config files between hosting systems

I have a dedicated server and use it for running a number of Ubuntu VMs. I’ve created a separate Dropbox account for my hosting needs, and this means that I now have 2.5Gb of space available for my VMs to exchange files or store immediate backups. Because Dropbox takes care of synchronizing all the content (and it has a LAN sync feature meaning VMs transfer files directly to each other instead of uploading back to the Dropbox site), it’s super easy and super fast to have a particular script updated and deployed to multiple systems.

I’m not quite there yet with actually running stuff like important automation or whole websites straight from Dropbox directory, but I use it for deploying scripts and configs all the time – once I get something working properly on one VM, I can then hope from one system to another and run the same set of commands against the files which are synchronized by Dropbox.

Transferring files to and from my hosting

This is a very recent addition to the things I do with Dropbox, but it’s an incredibly useful one. Having setup a separate Dropbox account for hosting, I shared one of the folders with my personal Dropbox account, and this means that transferring any files to and from my hosting had gotten to be this much easier. By putting a file into a local directory on my desktop, I have it accessible accross all the VMs on my hosting within seconds.

Likewise, if I’m reading logs or working on updating a particular config file, I can always copy it into Dropbox directory and have it synced back to my desktop.

Prior to this setup I had to rely on scp (passwordless logins using passphrase), and although it was pretty convenient to use, Dropbox approach is much more robust. Because files appear to be local, you get to work with them and manage directories as you like. You don’t have to remember the directory tree structure or follow any naming conventions – your files are the same across all the systesm and you don’t have to remeber to always sync.

Keeping backups of DBs or websites in Dropbox

Since majority of my websites are publicly available blogs, I don’t consider most of the backups to be a sensitive information. To be clear, I don’t store my passwords (wp-config.php file or htpasswd files) in Dropbox account, but everything else gets copied into it as a first level backup. I also have been doing automatic backups to Amazon’s S3 storage for about 5 years now, this means I can recover from most disasters quickly enough.

The reason Dropbox wins is because I don’t have to pay for each minor transfer or for storing an extra gigabyte or two – and yes, every little helps even though Amazon’s services are quite affordable. Another major reason I started doing backups to Dropbox is because it’s a local directory – I don’t have to use any extra tools to access all the backups in a simple directories/files structure. With Amazon’s S3 it’s also possible but setup is not as trivial.

Using Dropbox for controlling Unix systems remotely

With a few minutes and a really simple script, it’s possible to setup your own mission control for all the VMs in your hosting.

For example, if you create a cronjob which looks for a particular file, you can control which DB server your systems will connect to or which directory you’ll get the latest important log file copied into.

I’m also playing with services management based on the Dropbox account. If there’s a file present, I keep a particular service running. As soon as the file is gone, my cronjob gracefully stops the service. A slightly more sophisticated approach involves storing services names and system names associations in a Dropbox synchronized file – this allows for more flexibility as I can specify which service I want to be running on which nodes.

Sure enough, this isn’t the most straightforward way to manage your system, but such an approach can be used on the go from your iPhone. For example, I can restart a webserver by just touching a file from my iPhone, while previously I would have to find the nearest computer I trust, download SSH client, connect to the box and only then fix the problem.

Have I convinced you enough? Did you like any tips, or do you have some more perhaps? Let me know in the comments section!

pS: if you don’t have a Dropbox account or perhaps if I persuaded you to create a separate one for your hosting – please use this link so that I get some extra Dropbox space for referring. Thanks!

 



Mounting NFS shares on Mac OS X

I’ve recently decided to give Mac OS X a try. For the past week or so I’ve been spending a good few hours a day working in Snow Leopard installed on a MacBook Pro borrowed from a friend.

While Mac OS is unlike any Unix-like operating system I’ve managed so far, there are certainly some of similarities. I can honestly say that I’m enjoying the Mac Book Pro so far, and hope to discover most of the differences compared to my previous Unix-like desktop which is Ubuntu 9.10.

Mounting NFS on MAC OS X

One thing which I noticed immediately was that out of the box it was impossible to mount any NFS shares from my Ubuntu NAS server. Any attempt to mount a remote filesystem would give me an error like this:

mbp:~ root# mount nasbox:/try /mnt
mount_nfs: /mnt: Operation not permitted

This error was happening for a relatively simple NFS share on the Ubuntu box:

nasbox# cat /etc/exports
/try            (rw)

… so I started looking around and realised that the reason for this strange problem is quite simple.

Mac OS X uses non-standard port for outgoing NFS connections

That’s right! Apparently, Mac OS X uses a non-privileged port (2049) for TCP and UDP connections serving the NFS transport. What this means is that most likely attempts to mount remote filesystems will fail, because most NFS servers don’t really like connections from insecure ports.

There are two ways to approach this problem:

  1. Fix it on the client side (probably makes more sense)
  2. Fix it on the NFS server side (if you manage both systems)

Using reserved NFS port number on Mac OS X

There’s a mount option supported by the mount_nfs command, which allows you to force the NFS client connections to originate from a privileged port. This will magically make your attempts to mount remote filesystems successful. The option is called resvport.

First we double-check that default mounts still don’t work:

mbp:~ root# mount nasbox:/try /mnt
mount_nfs: /mnt: Operation not permitted

… and now let’s use the resvport option:

mbp:~ root# mount -o resvport nasbox:/try /mnt

… and make sure we’re actually looking at a mounted filesystem:

mbp:~ root# df -h /mnt
Filesystem   Size   Used  Avail Capacity  Mounted on
nasbox:/try    61Gi   56Gi  2.6Gi    96%    /mnt

Allowing connections from non-privileged ports on NFS server

Like I said, if you manage both the Mac OS based client and the NFS server, perhaps it makes more sense to relax the default NFS server security and allow the connections from non-privileged ports.

Just to remind you about the validity of such a decision, the option to allow non-privileged connections is called insecure.

Here’s how you use it:

nasbox# cat /etc/exports
/try            (rw,insecure)

After making this change to the /etc/exports file, you’ll have to restart your NFS server. On my Ubuntu NAS box, it’s done like this:

nasbox# /etc/init.d/nfs-kernel-server restart
* Stopping NFS kernel daemon
...done.
* Unexporting directories for NFS kernel daemon...
...done.
* Exporting directories for NFS kernel daemon...
...done.
* Starting NFS kernel daemon
...done.

We know are ready to attempt the default mount of the same filesystem on the Mac OS X client:

mbp:~ root# mount nasbox:/try /mnt

That’s it! I won’t promise any Mac OS posts just yet, but if there is enough interest – I’d love to do the research and to post all the discoveries on the Unix Tutorial pages.

See Also



Fixed calculations in Unix scripts

Although I’ve already shown you how to sum numbers up in bash, I only covered the bash way of doing it. I really like scripting with bash, but when it comes to calculations, there’s quite a few important features missing from bash, and fixed point (thanks for the correction, Azrael Tod!) calculations is one of them. Fortunately, bc command comes as a standard in most Unix distros, and can be used for quite complex calculations.

Basic calculations with bc

bc is a very simple command. It takes standard input as an expression and then evaluates this, performing all the necessary calculations and showing you the result. Thus, to quickly sum numbers up or get a result of some other calculation, simply echo the expression and then pipe it out to the bc command:

ubuntu$ echo "1+2" | bc
3

Now, in scripts your calculations with bc are done quite similarly to what we did in bash. Here’s an example:

ubuntu$ NUMBER1=1
ubuntu$ NUMBER2=2
ubuntu$ SUM=$(echo "$NUMBER1+$NUMBER2"| bc)
ubuntu$ echo $SUM
3

I told you these calculations would be basic, right? Now onto the more interesting stuff!

Fixed point calculations with bc

Most people learn about bash math limitations when they attempt to do a simple calculation but can’t get the current answer with fixed point values. By default, all the operations happen with integers, and that’s what you would get:

ubuntu$ echo "1/2" | bc
0

Now, if you expect 0.5 to be the result of dividing 1 by 2, you need to explain it to bc, because by default it doesn’t show you any fractional part of the number.

The way you do this is quite simple: all you have to do is specify the number of digits you’d like to see  after the radix point of your result. For example, if I set this number to 5, I’ll get bc to output the result of my calculation with 5 digits after the radix point. The special keyword to convey this intention to the bc command is called scale. Just specify the scale value and separate it from the rest of your expression by the semicolon sign:

ubuntu$ echo "scale=5; 1/2" | bc
.50000

Here’s another example:

ubuntu$ echo "scale=5; 0.16*10.79" | bc
1.7264

Hope this answers your question! bc command is very powerful, so I’ll definitely have to revisit it again in the future. For now though, enjoy the fixed point calculations and be sure to ask questions if you think I can help!

See also:



Command Aliases in Unix shells

One of the really useful features almost every Unix shell has is support for command aliases – a way to run a command or a series of Unix commands using a shorter name you get associated with such commands.

An example of a command alias in Unix shell

Here’s one of the most useful aliases I have for Solaris systems:

solaris$ alias ls='/usr/local/gnu/bin/ls --color -F'

What is allows me to do is to simply type “ls” instead of the really long command line it refers to: /usr/local/gnu/bin/ls –color -F.

You see, the ls command which is shipped with Solaris, doesn’t have many options of the more up-to-date GNU ls command, and working with many Linux systems I quite like some of them like color highlighting of different directory objects – files, directories and executables.

Try typing the longer command a few times in a row, and compare it to the “ls” to get the idea of how much of a productivity gain one command alias can be!

Why you should use command aliases

Like many other things in Unix, aliases are a way to become more productive. The general rule of thumb is this: if you have to run some command something more than once every day – consider creating an alias for it. These are just a few cases where it makes sense to employ them:

  • if you repeatedly check whether some files exist or get updated
  • if you’re monitoring a certain aspect of your OS and you get the values using grep command
  • if you’re connecting to the same hosts using rsh or ssh

All of these and many more examples are greatly simplified if you alias them to some shorter commands.

Creating new aliases in bash

Setting up a new alias is quite easy, the syntax for alias command is very straightforward. Let’s say I want to automate the confirmation of swap usage based on a free command in Linux:

redhat$ free
total       used       free     shared    buffers     cached
Mem:       2075156     945712    1129444          0     177292     503416
-/+ buffers/cache:     265004    1810152
Swap:      2040244          0    2040244

The result I’m after is this command:

redhat$ free | grep Swap
Swap:      2040244          0    2040244

And here’s how I can create an alias called “swp” which refers to this series of commands:

redhat$ alias swp='free | grep Swap'

Once you execute this command, you can start using swp as a Unix command:

redhat$ swp
Swap:      2040244          0    2040244

Important: such a creation of new aliases is going to be only active for your current Unix shell and sub-shells you may spawn. To make your alias permanent, you’ll have to update one of your initialization scripts like. For Linux and bash, you should add the same alias command to your .bashrc file.

Removing aliases in Unix

In very much the same way, you can use the unalias command to get rid of a certain alias. The nature of this command is such that you’ll most likely use it when creating and debugging new aliases. It’s unlikely that you’ll need to use it in your initialization scripts.

Following the example above, here’s how to get rid of the swp alias and verify that it’s gone:

redhat$ unalias swp
redhat$ swp
bash: swp: command not found

How to list your current aliases

If you run the alias command without any parameters, you’ll be shown a full list of aliases currently configured for your user account, here’s an example from one of my systems:

l.      ls -d .* --color=tty
ll      ls -l --color=tty
ls      ls --color=tty
vi      vim

That’s it for today! Stay tuned for a follow-up post which will share some of the examples for command aliases in Unix. If you have some – please leave a comment so that I can share it with others!

See also:



Unix scripts: how to sum numbers up

If you’re ever thought of summing up more than two numbers in shell script, perhaps this basic post will be a good start for your Unix scripting experiments.

Basic construction for summing up in shell scripts

In my Basic arithmetic operations in Unix shell post last year, I’ve shown you how to sum up two numbers:

SUM=$(($NUMBER1 + $NUMBER2)

using the same approach, it’s possible to calculate sums of as many numbers as you like, if you use one of the loops available in your shell.

Before we get started, let’s create a simple file with numbers we’ll work with:

ubuntu$ for i in 1 2 3 4 5 6 7 8 9; do echo $i >> /tmp/nums; done;
ubuntu$ cat /tmp/nums
1
2
3
4
5
6
7
8
9

Using a while loop to sum numbers up in Unix

Here’s an example of how you can use a while loop in Unix shell for summing numbers up. Save it as a /tmp/sum.sh script, and don’t forget to do chmod a+x /tmp/sum.sh so that you can run it!

#!/bin/sh
#
SUM=0
#
while read NUM
do
        echo "SUM: $SUM";
        SUM=$(($SUM + $NUM));
        echo "+ $NUM: $SUM";
done < /tmp/nums

Here’s how the output will look when you run it:

ubuntu$ /tmp/sum.sh
SUM: 0
+ 1: 1
SUM: 1
+ 2: 3
SUM: 3
+ 3: 6
SUM: 6
+ 4: 10
SUM: 10
+ 5: 15
SUM: 15
+ 6: 21
SUM: 21
+ 7: 28
SUM: 28
+ 8: 36
SUM: 36
+ 9: 45

Of course, your data will most probably will be in a less readable form, so you’ll have to do a bit of parsing before you get to sum the numbers up, but the loop will be organized in the same way.

That’s it for today, enjoy and feel free to ask questions!

See also:



Converting date and time to Unix epoch in Perl

Today I was working on a script, and one of the subroutines needed simple seconds-based arithmetics with time. As you probably remember from  my date and time in Unix scripts article, the easiest way to approach this task is to deal with the raw representation of date and time in Unix – the Unix epoch times. This post will show you how to convert standard dates into Unix epoch times in Perl.

Why would you want to convert timestamps to Unix epoch time?

Unix epoch time is an ever-growing counter which increments every second and shows the number of seconds elapsed since 00:00:00 UTC on January 1, 1970. Such numbers are pretty useless in their raw form for the tasks of confirming current time and date, but they’re perfect for measuring between two points in time.

Current epoch time in Perl

First things first. If you want to confirm the current Unix epoch time in Perl, here’s how you do it: just use the time() subroutine.

#!/bin/perl
print "Current (epoch) time: " . time() . "\n";

Converting regular date and time into epoch time

It’s equally easy to convert time into Unix epoch representation, for this you should use timegm() or localtime() subroutine from the Time::Local module.

timegm() has the following syntax:

$current = timegm($sec,$min,$hours,$day,$month,$year);

Important: While most of the parameters are self-explanatory, it is probably worth reminding that $year should be specified as a number like 109 and not 2009.
Also, the $month is not the number of a month, but a number of months since January. So, 0 will mean January itself, 1 will mean February, and 11 means December.

Here’s a fully working example, showing the countdown in seconds until midnight on the New Year’s Eve 2009:

#!/usr/bin/perl
#
use Time::Local;
#
$sec=59;
$min=59;
$hours=23;
$day=31;
$month=11;
$year=109;
#
$current = time();
print "Current (epoch) time: $current\n";
#
$newyear = timegm($sec,$min,$hours,$day,$month,$year);
print "New Year's Eve 2009: $newyear\n";
#
print "Only " .($newyear-$current) . " seconds to go\n";

When you save this script as epoch.pl and make the file executable, here’s what you’ll get if you run it:

# ./epoch.pl
Current (epoch) time: 1234911622
New Year's Eve 2009: 1262303999
Only 27392377 seconds to go

That’s it for today! Hope this little post will save you time when tacking next scripting challenge with Perl.

See also



How To Change Ownership of Files and Directories in Unix

I’ve just been asked a question about changing the ownership of files from one Unix user to another, and thought it probably makes sense to have a quick post on it.

File ownership in Unix

Just to give you a quick reminder, I’d like to confirm that every single file in Unix belongs to some user and some group. There simply isn’t a way to create a file without assigning ownership. I’ve briefly touched the topic of confirming file ownership in Unix before, so today I will simply build on that and show you how to change ownership of files.

Here’s a setup for today: I have created a temporary directory with a few files and made myself the owner of all the files:

ubuntu$ ls -al /home/greys/example/
total 12
drwxr-xr-x  3 greys admin 4096 Feb  9 03:55 .
drwxr-xr-x 13 greys greys 4096 Feb  9 03:54 ..
drwxr-xr-x  2 greys admin 4096 Feb  9 03:55 dir1
-rw-r--r--  1 greys admin    0 Feb  9 03:54 file1
-rw-r--r--  1 greys admin    0 Feb  9 03:55 file2

As you can see from this listing, the owner (third field in each line) is my username – greys. The next field is a Unix group of each file’s owner – admin in my example.

Changing owner of a file in Unix

Changing file ownership means only updating the association between a Unix user and a file, and nothing else. When you’re changing the owner of a file, no data contained in a file is changed.

To change the owner of a file, you need to use the chown command (easy enough to remember: CHange OWNer – chown), with the following syntax:

ubuntu$ chown nobody file1

In this command, nobody is the username of the new owner for a list of files. In my example, the only file we’d like to change ownership for is file1.

It is important to realize that you can only change file ownership as a super-user (root). Any regular Unix user cannot change the ownership of any file, and I’d like to explain why.

Indeed, some people are surprised: if I’m the owner of a given file, why can’t I change the ownership for it? That’s because transferring the ownership will mean some other Unix user will become the owner of the file(s) in question. So changing ownership is like making a decision not only for yourself, but for the new owner of the files.This is only something a super-user – special administrative account in Unix – can do.

The same logic applies to other people not being able to become owners of your files, even if they’re willing to assume the new responsibilities of owning files. They cannot revoke your ownership, because each Unix user is only allowed to make decisions and take actions on his/her own behalf.

That’s why you will probably see an error like this if you attempt to change ownership of a file as your own regular Unix user:

ubuntu$ id
uid=1000(greys) gid=113(admin) groups=33(www-data),113(admin)
ubuntu$ chown nobody file1
chown: changing ownership of `file1': Operation not permitted

But if we become root:

ubuntu$ sudo -i
[sudo] password for greys:
ubuntu#

… we’ll have no problem changing owners for any files:

ubuntu# cd /home/greys/example
ubuntu# chown nobody file1
ubuntu# ls -l file1
-rw-r--r-- 1 nobody admin 0 Feb  9 03:54 file1

Changing owner for multiple files

If you’re going to change owner of a few files, this can easily be done using either a full list of files or a mask.

First, here’s an example of updating ownership for a specified list of files (and as you can see, directories as well):

ubuntu# chown nobody file2 dir1
ubuntu# ls -al
total 12
drwxr-xr-x  3 greys  admin 4096 Feb  9 03:55 .
drwxr-xr-x 13 greys  greys 4096 Feb  9 03:54 ..
drwxr-xr-x  2 nobody admin 4096 Feb  9 03:55 dir1
-rw-r--r--  1 nobody admin    0 Feb  9 03:54 file1
-rw-r--r--  1 nobody admin    0 Feb  9 03:55 file2

IMPORTANT: here’s one thing which is often forgotten: when you’re changing an owner of a directory, this DOES NOT automatically change owner of all the files which already exist in this directory. So, if we check the file3 in dir1 after the example above, we can see that even though dir1 now belongs to user nobody, file3 in it still belongs to me:

ubuntu# ls -l dir1/file3
-rw-r--r-- 1 greys admin 0 Feb  9 03:55 dir1/file3

If your intention is to change ownership of all the files and directories of a certain location in your filesystem, you need to use a -R option of the chown command, which means recursive ownership change:

ubuntu# chown -R nobody dir1
ubuntu# ls -l dir1/file3
-rw-r--r-- 1 nobody admin 0 Feb  9 03:55 dir1/file3

And just to further demonstrate this, I’m going to change owner of all the files and directories in /home/greys/example directory back to my own username, greys:

ubuntu# chown -R greys /home/greys/example/
ubuntu# ls -l /home/greys/example/
total 4
drwxr-xr-x 2 greys admin 4096 Feb  9 03:55 dir1
-rw-r--r-- 1 greys admin    0 Feb  9 03:54 file1
-rw-r--r-- 1 greys admin    0 Feb  9 03:55 file2

Changing group ownership for a file

Similar to the chown command, there’s a command specifically helping you with changing not the owner (user) of a file.

IMPORANT: unlike chown command, chgrp can be used by non-privileged (regular) users of a system. So you don’t have to be root if you want to change a group ownership for some of your files, provided that you’re changing the ownership to a group you’re a member of.

For example, I’m a member of quite a few groups on one of my Ubuntu servers:

ubuntu$ id greys
uid=1000(greys) gid=1000(greys) groups=1000(greys),4(adm),20(dialout),24(cdrom),46(plugdev),114(lpadmin),115(sambashare),116(admin)

Now, if I create a new file, it will by default belong to my primary group (called greys, just like my username):

ubuntu$ touch file
ubuntu$ ls -al file
-rw-r--r-- 1 greys greys 0 2012-09-20 10:48 file

I can now change group ownership of this file, in this case to a group admin, which I’m also part of.

ubuntu$ chgrp admin file

and this is just to confirm that the change actualyl happened:

ubuntu$ ls -al file
-rw-r--r-- 1 greys admin 0 2012-09-20 10:48 file

That’s it for today, good luck with changing file owners on your Unix system!

Recommended books:

See also:



Perl Scripting: Check If a File Exists

It’s not often that I write about Perl Scripting on Unix Tutorial, but that’s just because I don’t script in Perl this much on a regular basis. Today, however, I’d like to share one of the building blocks – a really basic piece of functionality which you’ll find really useful in almost any Perl script.

How to check if a file exists in Perl

First of all, we need to create one file for our experiments:

ubuntu$ touch /tmp/file.try

Now, create a new file with our script: /tmp/file-try.pl (don’t forget to chmod a+x /tmp/file-try.pl afterwards to make it executable):

#!/usr/bin/perl
#
$FILE1="/tmp/file.try";
$FILE2="/tmp/file2.try";
#
if (-e $FILE1) {
        print "File $FILE1 exists!\n";} else {
        print "File $FILE1 is not found!\n";
}
if (-e $FILE2) {
        print "File $FILE2 exists!\n";
} else {
        print "File $FILE2 is not found!\n";
}

Now, as you can see from the example, the actual condition checking for the existence of a file is this:

if (-e $FILE)

where $FILE is the string containing the filename.

In my example, there’s two filenames ($FILE1 and $FILE2) and two conditions, so that you can see for yourself how one of them confirms the existence of a file and another one proves that there’s no such file found.

To see the script work, simply run it without any parameters:

ubuntu$ /tmp/file-try.pl
File /tmp/file.try exists!
File /tmp/file2.try is not found!

That’s all for today! Enjoy your Perl scripting!

See also:



What to do if numeric id is shown instead of Unix username

As you know, every file in your Unix OS belongs to some user and some group. It is very easy to confirm the ownership of any file because user id and group id which own the file are always linked to the file. However, sometimes you can’t tell which user owns the file, and today I’m going to explain why. It’s a rather lengthy post and a complicated matter, so please leave questions or comments to help me polish this article off.

Files and directories ownership in Unix

If you look at any file using ls command, you will see an output like the one shown below – it reveals file access permissions, user and group id of the owner, the modification timestamp and the file name itself:

ubuntu$ ls -l /tmp/myfile
-rw-r--r-- 1 greys admin 0 Jan  6 03:51 /tmp/myfile

In this example, the /tmp/myfile file belongs to me, hence the username is greys. It also belongs to my default (primary) Unix group – admin.

Similarly, ownership of any file or any directory can be confirmed for every object in available filesystems. Here’s just a few more examples, these are the standard system files belonging to root:

ubuntu$ ls -ald /etc /etc/passwd
drwxr-xr-x 91 root root 4096 Jan  6 03:51 /etc
-rw-r--r--  1 root root 1481 Jan  6 03:51 /etc/passwd

Why numeric IDs are sometimes shown instead of username or groupname

Sometimes though, you will look at a file and instead of the username you will see a numeric ID:

ubuntu# ls -al /tmp/file
-rw-r--r-- 1 1006 root 0 Jan  6 03:51 /tmp/file

The reason numberic ID (1006 in the example above) is shown instead of a username is because your system doesn’t recognize this ID – it can’t be associated to any username known to your Unix OS.

There are a few possible scenarios for this to happen, but most likely the user has been removed since the file was created. Naturally, deleting any User doesn’t automatically mean removing every single file belonging to such a user, that’s why the files stay but can no longer be associated with the existing user. All they have to show is the user ID which once owned the file.

How to find the missing username using user id

Unfortunately, there are no easy ways to recover the username (or any other user-specific information) based on a misterious user ID some of your files might have. There are a few things can try though.

  1. Try other Unix systems in your environment

    It can be the case that Unix account was a local one automatically created by your system administrators. There’s still a chance the same uid exists on other systems. Log into a few of them and verify if they have a user with the same user id (read this post for more information: How to Find Out user id):

    ubuntu$ getent passwd 1006
newowner:x:1006:1006::/home/newowner:/bin/sh

    The same tip applies in case of more mature environments where Unix systems don’t have local users, but instead rely on NIS or LDAP for accessing user accounts information.

    If your system for whatever reason can’t access the centralized storage for users, you will experience the same symptoms – most of files belonging to users will appear to have numeric IDs instead of usernames. Most likely though, you’ll have more important problems in this case – like not being able to log in as anything else but root (which is an administrative account always created locally on each system).

  2. Look at home directories and their owners

    When a new user is added to Unix system, it usually gets a home directory assigned to it. Creating a home directory is not a default behaviour at times, but it’s a good practice and so there’s a very high probability that the user you’re looking for had a home directory. Removing home directories isn’t usually done at the same time when a user is removed, so there’s also a good chance that even though the user isn’t found anymore, the home directory is still there.

    What you should be looking for is a home directory which belongs to the same user id which some of the unidentified files of yours belong to.

    Simply do an ls command under /home directory and see if any of the directories there appear to have numeric IDs instead of usernames:

    ubuntu# ls -ald /home/*
drwxr-xr-x  2 ftp   nogroup  4096 May 22  2007 /home/ftp
drwxr-xr-x 11 greys greys    4096 Dec 13 19:56 /home/greys
drwxr-xr-x  2 1006  admin    4096 Jan  6 04:23 /home/mike

    As you can see, sometimes you might get lucky – the directory is there, and since most home directories usually have the same name as the username which owns them, you can deduct that the username of the user id 1006 was “mike“. You can now recreate Mike’s account and it will be immediately reflected for all the files owned by user id 1006:

    ubuntu# ls -ald /home/mike
drwxr-xr-x 2 1006 admin 4096 Jan  6 04:23 /home/mike
root@simplyunix:~# useradd -u 1006 mike
root@simplyunix:~# ls -ald /home/mike /tmp/file
drwxr-xr-x 2 mike admin 4096 Jan  6 04:23 /home/mike
-rw-r--r-- 1 mike root 0 Jan  6 03:51 /tmp/file
  3. Look at other users known to your systemSometimes users are created in batches, and you can guess who the user was by looking which users were created before and after. All you have to do is to use the same getent passwd approach for user ids which are smaller or larger than the one you want to identify.

    Another way to user other users’ information to your advantage is to verify which groups they belong to and to then query the groups to see if they have any members not currently known to your system. This will only work for NIS/LDAP groups, not local ones. What could happen is that even though a user was removed, the username is still listed in a few NIS groups.

That’s it for today. Hope this post helps you in your investigations, and stay tuned for more!

See also:



Easy date calculations in Unix scripts with GNU date

When I was writing a post about using date command to confirm date and time in your Unix scripts, I made a note in my future posts list to cover the date calculations – finding out the date of yesterday or tomorrow, and so on. Today I’ll show you the simplest way to calculate this.

GNU date command advantage

GNU version of the date command, although supporting a common syntax, has one great option: it allows you to specify the desired date with a simple string before reporting it back. What this means is that by default this specified date is assumed to be “now”, but you can use other keywords to shift the result of the date command and thus show what date it was yesterday or a week ago:

Here’s a default date output for the current date and time:

ubuntu$ date
Fri Sep 19 08:06:41 CDT 2008

Now, the parameter for specifying desired date is -d or –date, and if you use it with the “now” or “today” parameter, you’ll get similar output:

ubuntu$ date -d now
Fri Sep 19 08:06:44 CDT 2008
ubuntu$ date -d today
Fri Sep 19 08:06:50 CDT 2008

Showing tomorrow’s date

Similarly, you can get tomorrow’s date:

ubuntu$ date -d tomorrow
Sat Sep 20 08:02:12 CDT 2008

Obviously, if you feel like specifying a format for the date, you can do it:

ubuntu$ date -d tomorrow "+%b %d, %Y"
Sep 20, 2008

Find out yesterday’s date

Again, there’s no rocket science involved in showing yesterday’s date neither:

ubuntu$ date -d yesterday "+%b %d, %Y"
Sep 18, 2008

Show a date few days away

If you’re interested in a certain date a few days or even weeks away, here’s how you can do it:

Example 1: a date 5 days ago

ubuntu$ date -d "-5 days"
Sun Sep 14 08:45:57 CDT 2008

Example 2: a day 2 weeks from now

ubuntu$ date -d "2 weeks"
Fri Oct  3 08:45:08 CDT 2008

Example 3: a day two weeks ago from now

ubuntu$ date -d "-2 weeks"
Fri Sep  5 08:45:11 CDT 2008

Extreme example: a day 50 years ago

If you’re really curious about dates in Unix, you can even make GNU date go back a few years:

ubuntu$ date -d "-50 years"
Fri Sep 19 08:47:51 CDT 1958

That’s it for today, hope you like this little discovery – having mostly worked with Solaris systems most of my career, I didn’t know my Ubuntu had this functionality bonus. Really useful!

See also: