Archive for Linux

Shell tricks: shorthands

Even with tab completion, typing long commands is tedious. But, there’s something even worst: typing the same long commands again, and again, and again… So how do you solve that? It’s simple: you shorten them. Surprising, uh? Okay enough theory, let me show you some examples.

Here’s a tedious command of Type-A:

% sudo aptitude install zsh

Look at it carefully since you will need to hunt these long commands down until none remains. Now, let me explain how you execute a such command. Open up your personal shell initialization file (e.g. ~/.bashrc for Bash, ~/.zshrc for Zsh, etc). Then, add the following:

alias spkgi="sudo aptitude install"

Reload your shell and finally, enjoy:

% spkgi zsh

Now I can introduce, as you can deduce, other shorten commands that you can produce and reproduce:

# Package Management
alias pkg="aptitude"
alias spkg="sudo aptitude"
alias spkgi="sudo aptitude install"
alias spkgu="sudo aptitude safe-upgrade"
alias spkgr="sudo aptitude remove"
alias spkgd="sudo apt-get build-dep"

# Miscellaneous Helpers
alias nc="rlwrap nc"
alias e=$EDITOR
alias se=sudoedit
alias reload="source ~/.zshrc"
alias g=egrep

Next after Type-A tedious commands, we have the Type-S ones. To execute these, you will you need some sort of special shell support. So, here’s some examples of the Type-S monstrosity:

% find Lib/ -name '*.c' -print0 | xargs -0 grep ^PyErr
% find -name '*.html' -print0 | xargs -0 rename 's/\.html$/.var/'
% find -name '*.patch' -print0 | xargs -0 -I {} cp {} patches/

I hope you start to see some patterns (if you don’t, then try harder). The first one could (and should) be rewritten as:

% rgrep --include='*.c' ^PyErr Lib/

But that isn’t short enough for me, so I have a short helper:

    shift 2
    grep -Er --include=$filepat $pat ${@:-.}
# In Zsh, 'noglob' turns off globing.
# (e.g, "noglob echo *" outputs "*")
alias rg='noglob rg'

It is lovely to use:

% rg *.c ^PyErr Lib/
% rg *.c PyErr_Restore . -C 10 | less
% rg *.[ch] stringlib
% rg *.c ^[a-zA-Z]*_dealloc Modules/ Objects/

The second example is quite similar to the previous one. However, the find/rename combination is much less common (at least for me) than the find/grep one. This one needs to be broken in pieces. One obvious thing to factor out is the find -name with an alias:

alias fname="noglob find -name"

Using this alias, you can rewrite the second example as:

% fname *.html -print0 | xargs -0 rename 's/\.html$/.var/'

It’s better, but it’s not short enough yet. The ugly part of this command is the -print0 | xargs -0. I hate to type that. Wouldn’t it be nice if we could define an alias for it? How about:

alias each="-print0 | xargs -0"

Unfortunately, that doesn’t work since aliases are only expanded if they are in the command position. Luckly, Zsh has that neat feature called global aliases, which does exactly what we want.

alias -g each="-print0 | xargs -0"

With this feature of Zsh, the second example become:

% fname *.html each rename 's/\.html$/.var/'

Now, we can also attack the third one:

% fname *.patch each -I {} cp {} patches/

It is possible to shorten a bit by defining another alias combining each and -I {}, but that won’t make a big difference.

Finally, there are the Type-R tedious commands. These are hard to avoid, unless you’re careful. Here’s again some ridiculous examples to help you recognize these redundant commands:

% gcc -o stackgrow stackgrow.c
% pkg show emacs-snapshot-bin-common emacs-snapshot-common emacs-snapshot-gtk emacs-snapshot
% cat ../lispref.patch ../lwlib.patch ../etc.patch | patch -p1

To reduce these, you don’t need change your shell configuration; you change your habits instead. Using alternations (which are non-standard, but supported by most shells), you can rewrite the two first example as:

% gcc -o stackgrow{,.c}
% pkg show emacs-snapshot{{-bin,}-common,-gtk,}

Now, you are surely asking yourself: “what is different about the third one?” Well, think about it. Got it? No? Ah, come on, it is easy. Here’s a hint:

% echo 'cat ../{lispref,lwlib,etc}.patch | patch -p1' | wc -c
% echo 'cat ../lispref.patch ../lwlib.patch ../etc.patch | patch -p1' | wc -c

You like my hint, don’t you? Here’s the answer:

% echo 'cat ../li\t ../lw\t ../et\t | patch -p1' | wc -c

Tab completion doesn’t work well with prefix alternations. Even if the command using alternation is shorter, it still doesn’t beat good old tab completion.

And that’s all folks. I surely have plenty of other tricks to show, but that will be for the other posts of this short series.

Welcome to Mr. Crystal Ball

As some of you may already know, I am a die-hard fan of productive editing. That is probably because I don’t find myself very fast, on a keyboard. So, I am always trying to find ways to improve my editing speed. And when I ain’t surfing on the web, I am either typing stuff in my shell or my editor. So today, I would like to share a few tricks I uses in my default shell, Z Shell.

The shell history can be a powerful tool. If you find yourself typing commands again, and again, and again, you probably can use it at your advantage. You probably already know about Ctrl+R, which is bound to history-incremental-search-backward command in most shells. Personally, I don’t find it very useful since it tries to find a match everywhere, but it’s better than cycling through the history with the Up/Down keys. In fact, anything is better than the Up/Down keys. So, why not rebind them to something more useful, like history-search-backward? Well, that is easy. With Zsh, you need to add these two line to your .zshrc:

bindkey '\e[A' history-search-backward
bindkey '\e[B' history-search-forward

In fact, if you’re using Emacs key-bindings, you don’t even need to do anything, because Meta+P and Meta+N are already bound to these two functions. Incidentally, Steven Harms is advocating to enable this feature by default in Ubuntu, for Bash’s users. Personally, I am not sure if it’s really necessary to make it a default. I am not a fan of modifications in .inputrc, either. But, I will leave that discussion for another blog post.

Now, that we have functional Up/Down arrow keys, can we do more? Yes, we can! Let me introduce one of my favorite features of Zsh, preemptive auto-completion. If you’re tired typing TAB a zillion times a day, you will love this one. This feature implements predictive typing using history search and auto-completion. Again, to enable it, just copy these lines to your configuration file:

autoload predict-on
zle -N predict-on
zle -N predict-off
bindkey '^Z'   predict-on
bindkey '^X^Z' predict-off
zstyle ':predict' verbose true

Here, note that predict-on and predict-off, are bounded to Ctrl+Z and Ctrl+X Z respectively. That means you can turn it on/off, whenever you need to. You will find useful to turn it off when you edit the middle of a command, since it can confuse the prediction. But other than that, it’s great.

Sometime, the shell editor is not enough for me — I need something more powerful when I edit long commands. So, I use another cool built in function of Zsh, called edit-command-line. With this feature, I can edit the current command with an external editor, defined by the environment variable $EDITOR. To enable it, just copy-and-paste this:

autoload edit-command-line
zle -N edit-command-line
bindkey '^Xe' edit-command-line

So, when I think the command will be long, like a for-loop. I just press Ctrl+X e, which launches, on my system, emacsclient. I am always running Emacs with its server, therefore the shell command is instantaneously loaded into a Emacs buffer. Then when I am done, I close the Emacs session with Ctrl+x # and the command appears in my shell. It is just sweet.

Even if you’re a master with your editor, nothing beats a short alias, or a shell script. I keep a full directory of useful scripts, to automate my daily tasks. At first, writing scripts feels a bit awkward. If you’re like me, you will always worry that your scripts might go terribly wrong, and eat your data. That’s totally normal, but don’t be a fool. Automating your tasks, even the most trivial ones, will save some of your precious time. Unlike scripts, which can really do some heavy automation, aliases are just a shell convenience, like auto-completion. Personally, I am not a big fan of fancy aliases. (I tend to use functions for the more fancy things.) Anyway, here some of my favourite aliases:

# Set up aliases
alias c=clear
alias d='dirs -v'
alias e=$EDITOR
alias grep=egrep
alias h=history
alias j=jobs
alias po=popd
alias pu=pushd
alias ss='screen -Rx'

# Global aliases -- These do not have to be
# at the beginning of the command line.
alias -g M='|more'
alias -g L='|less'
alias -g H='|head'
alias -g T='|tail'

# Go to parent directories without `cd'
setopt autocd
alias -g ...='../..'
alias -g ....='../../..'
alias -g .....='../../../..'

I certainly have a ton of shell tricks, but I will keep them for my other blog posts. So, that’s all folks!

Understanding Linux File Permissions

File permissions are probably one of the biggest difference between Windows and Unix-style operating systems. They make Linux much more secure when they are well used. However, they can also cause nightmare to the casual Linux administrator.

The first thing you need to know is that a Linux system has two way of classifying users. There is, of course, the user name, but there is also groups. Groups are, strictly speaking, only a way to share permissions between the users. For example, all the member of the admin group on your system is able to use the command sudo. As you probably know, sudo allows you to run a command as another user (by default, the root user).

Let me introduce you to your command-line friends that will help you to manage the permissions of your system.

  • adduser: This command let you add new user on your system. It can also add a user into a group.
  • addgroup: Its name says it all. This command let you add new group on your system.
  • chmod: I believe this is the most widely known Unix command. It is even a verb in the world of server-side technology, like PHP. This command let you alter the permissions of a file. It is a swiss-army knife. Learn it, and use it well.
  • chown: Also a very important command, chown can change the user and group ownership of a file.
  • chgrp: This is chown‘s little brother. Unlike chown, this command can only change the group ownership of a file.
  • groups: Somehow less important but still useful, groups shows you the groups you are a member of.
  • whoami: Don’t know why, but I love the name of this command. Anyway, this command tells you who you are.
  • who: This command shows you who is login on your system. I never use it, since I find w more useful for my usage.
  • w: And here our last little friend, the w command. It displays a list of the logged users like who, but also display their attached process and the uptime of the machine you’re on.

Obviously if you want to learn to use those commands well, you will need to do some homework and read their respective manual pages (with man <command>).

So, how permissions work? First, we need an example:

alex@helios /etc % ls -l
total 1548
-rw-r--r--  1 root   root      2584 2006-11-29 08:40 adduser.conf
drwxr-xr-x  4 root   root      4096 2006-12-13 10:46 apt
drwxr-xr-x  2 root   root      4096 2006-12-17 00:15 cron.d
drwxr-sr-t  5 cupsys lp        4096 2006-11-29 08:51 cups
-rw-r--r--  1 root   root       817 2006-11-29 08:39 fstab
-rw-r--r--  1 root   root       806 2006-12-17 00:15 group
-rw-r--r--  1 root   root      1430 2006-12-17 00:15 passwd
lrwxrwxrwx  1 root   root        13 2006-11-29 08:40 motd -> /var/run/motd
drwxr-xr-x  2 root   root      4096 2006-12-22 23:36 rc0.d
drwxr-xr-x  2 root   root      4096 2006-12-19 12:06 rc1.d
drwxr-xr-x  2 root   root      4096 2006-12-19 12:06 rc2.d
drwxr-xr-x  2 root   root      4096 2006-12-19 12:06 rc3.d
drwxr-xr-x  2 root   root      4096 2006-12-19 12:06 rc4.d
drwxr-xr-x  2 root   root      4096 2006-12-19 12:06 rc5.d
drwxr-xr-x  2 root   root      4096 2006-12-22 23:36 rc6.d
-rwxr-xr-x  1 root   root       306 2006-11-29 08:40 rc.local
-rwxr-xr-x  1 root   root       306 2006-11-29 08:40 rc.local
-rw-r-----  1 root   shadow     873 2006-12-17 00:15 shadow
-rw-r--r--  1 root   root       214 2006-12-02 13:27 shell
-r--r-----  1 root   root       403 2006-11-29 09:10 sudoers

Only the first, third and fourth column are interesting for us, right now. The first column gives us information about the file permissions. The third is the owner of the file and the fourth is the group.

So, what all this mess means? File permissions are like little switches you turn on and off. There is three types of permission: read, write, and execute. There’s also three types of ownership: owner (or user), group, and other. So, 3 times 3 equals 9 switches you can control.

That is exactly what we see in the first column. The first element of this column is the type of the file. A - means it’s a normal file; a d is for a directory and l is for a link pointing to a file. There is several other types of file, but they are much less useful to know for the casual Linux system administrator.

You probably figured that the rest are the permissions. Here a legend of the symbol I will use for the rest of this post:

u - owner
g - group
o - others

r - read
w - write
x - execute

t - file type

As you will see, there is nothing complicated about the first column in the output of the ls -l command. It’s a simple representation of the switches I mentioned earlier. So, let’s decrypt it:


That’s it. Just read it out loud: type, owner, group and others. So, if you see something like -rwxr-xr-x, you can read it as: “a normal file which the owner has the read, write and execute permission and which its group and others has the read and execute permission.” That is extremely verbose, but correct.

You can change the permissions with the chmod command:

alex@helios ~ % ls -l file
-rw-r--r-- 1 alex alex 0 2007-01-01 23:58 file
alex@helios ~ % chmod og+rw file
alex@helios ~ % ls -l file
-rw-rw-rw- 1 alex alex 0 2007-01-01 23:58 file

I won’t go in details here, because it’s quite simple to understand. If you want to know more, The info page of chmod is a great source of information (info coreutils ls).

If you already knew what are permissions, you are probably 1) rolling on the floor laughing, how I gone into the great details of that simple thing, or 2) grumbling that you want a refund because I wasted your bandwidth. So, hold on here the more advanced stuff.

You probably saw numerical (or should I say octal) permissions, like 777. But, do you actually know how to read them? For example, what 645 means? Hopefully, you aren’t trying to remember all of them. I going to give a trick.

As you probably know, each digit represents the permissions of one type of ownership (owner, group and other). One thing you need to know is they are not decimal digits; they are octal digits. So, something like 855 is not a valid permission.

Now, here one interesting property of octal digit: you can write them all as three bits (binary digits) number. Here the full list:

Octal   Binary
0       000
1       001
2       010
3       011
4       100
5       101
6       110
7       111

As you may know, bits are like switches you flip on and off. Sound familiar? Right, they are exactly like permissions. Now imagine that instead of letters, the permissions in the ls -l were shown as binary numbers:

alex@helios ~ % ls -l file
-110100100 1 alex alex 0 2007-01-01 23:58 file

110100100 is a perfectly legit binary number and in octal it is 644. So, what happens if we chmod our file to 644? You certainly deduced it:

alex@helios ~ % chmod 644 file
-rw-r--r-- 1 alex alex 0 2007-01-01 23:58 file

Pretty nice, eh? You been working with the binary system without knowing it. Back to our problem, you need to change the permissions of a file to 645. So, how do you calculate what it means? That is simple, now that you know it’s just a binary number:

Binary  Octal  English
100     4      read
010     2      write
001     1      execute


owner: 6 = 4+2 = read+write
group: 4 = 4   = read
other: 5 = 4+1 = read+execute

So, let’s check if we were right:

alex@helios ~ % chmod 645 file
alex@helios ~ % ls -l file
-rw-r--r-x 1 alex alex 0 2007-01-01 23:58 file

I bet you didn’t know it was that simple. Now, you can show to your geek friends how good you are by calculating any octal permissions in your head.

There is also some special permissions you can use too, like setuid, setgid, and sticky. I won’t cover them here, because they are pretty useless to the casual Linux system administrator.

I hope you enjoyed this introduction, because that’s all folks!