A Slightly Skeptical View on Unix Shells and Open Source Shells Webliography
Softpanorama
(slightly skeptical)
Open Source Software Educational Society
May the
source be with you,
but remember the KISS principle ;-)
Shellorama
A Slightly Skeptical View on Unix Shells and Open Source Shells
Webliography
(A Slightly Skeptical View on role of Shell Scripting in
Unix administrators)
News
Introduction
Best Shell
Books
Recommended
Links
Papers,
ebooks tutorials
Bourne
Shell and portability
Man pages
Reference
Scripts
Collections
Bash
ksh93
AWK
Sed
Restricted shell
C-shell and tcsh
zsh
TCL
Expect
Language
Debugging
Advanced
filesystem navigation
Command completion
Pipes
Vi editing mode
Pushd and popd
Aliases
Functions
Regular Expressions
Shell
Prompts
Shell Dotfiles
Command history reuse in shell
Tips
Monitoring scripts
Pretty Printing
Unix shells
history
Humor
Etc
"The lyfe so short, the craft so long to lerne,''
Chaucer c. 1340–1400
(borrowed from the bash FAQ)
This collection of links is oriented on students (initially
it was provided as a reference material to my shell programming university course)
and is designed to emphasize usage of advanced shell constructs and pipes
in shell programming (as well as ksh93 that has the best support for those constructs).
An introductory paper
Slightly
Skeptical View on Shell discusses the shell as a scripting language and as one
of the earliest examples of very high level languages. This might also be
useful for system administrators who constitute the major part of shell users and
lion part of shell programmers.
This page is the main page to a set of sub-pages devoted to shell
that collectively are known as Shellorama. The most important are:
Shell
Papers,E-books and Tutorials
Language
page - describes some of the exotic shell contracts and provides links to web
resources about them. Bash 3.x added several useful extensions. Among
them (Bash Reference
Manual ):
Arithmetic expansion allows the evaluation of an arithmetic expression
and the substitution of the result. The format for arithmetic expansion
is:
$(( expression ))
Older (( ... )) construct borrowed from ksh93 is also supported.
Process substitution. It takes the form of <(list)
or >(list)
=~ operator. An additional
binary operator, ‘=~’, is available,
with the same precedence as ‘==’
and ‘!=’. When it is used, the string
to the right of the operator is considered an extended regular expression
and matched accordingly (as in regex3)). The return value is 0 if
the string matches the pattern, and 1 otherwise. If the regular expression
is syntactically incorrect, the conditional expression's return value is
2. If the shell option nocasematch (see the description of
shopt in Bash Builtins) is enabled,
the match is performed without regard to the case of alphabetic characters.
Substrings matched by parenthesized subexpressions within the regular expression
are saved in the array variable BASH_REMATCH. The element of
BASH_REMATCH with index 0 is the portion of the string matching
the entire regular expression. The element of BASH_REMATCH
with index n is the portion of the string matching the nth
parenthesized subexpression.
C-style for loop. Bash implements the for ((
expr1 ; expr2 ;
expr3 )) loop, similar to the C language (see
Looping Constructs).
Brace expansion -- a mechanism for genration of a sequnce of
similar stings (see also
Filename Expansion), but the file names generated need not exist. Patterns
to be brace expanded take the form of an optional preamble, followed
by either a series of comma-separated strings or a sequnce expression between
a pair of braces, followed by an optional postscript. Brace expansions
may be nested. For example,
bash$ echo a{d,c,b}e
ade ace abe
Tilde notation for content of DIRSTACK array which is rather
convenient for implementing "directory favorites" concept with "push/pop/dirs"
troika.
~N -- The string that would be displayed
by ‘dirs +N’
~+N -- The string that would be displayed
by ‘dirs +N’
~-N -- The string that would
be displayed by ‘dirs -N’
Debugging
which among other things contains some links to bash debugger. Until bash
3.0 none of the shells has had a built-in debugger, which IMHO was the major
weakness of the existing implementations. Actually it was more a weakness,
it was a blunder and it is strange that such talented persons as Bill Joy and
David Korn did not realize this.
Command
history reuse which is devoted to one of the most important command line
features
Dotfiles
Command completion
pushd/pops/dirs
troika
Advanced Unix filesystem
navigation
Vi editing mode
Effective Use
of Shell history
I strongly recommend getting a so-called
orthodox file manager (OFM). This
tool can immensely simplify Unix filesystem navigation and file operations (Midnight
Commander while defective in handling command line can be tried first
as this is an active project and it provides fpt and sftp virtual filesystem in
remote hosts)
Actually filesystem navigation in shell is an area of great concern
as there are several serious problems with the current tools for Unix filesystem
navigation. I would say that usage of cd command (the
most common method) is conceptually broken and deprives people from the full understanding
of Unix filesystem; I doubt that it can be fixed within the shell paradigm
(C-shell made an attempt to compensate for this deficiency by introducing history
and popd/pushd/dirs troika, but this
proved to be neither necessary nor sufficient for compensating problems with the
in-depth understanding of the classical Unix hierarchical filesystem inherent
in purely command line navigation ;-). Paradoxically sysadmins
who use OFMs usually have much better understanding of the power and flexibility
of the Unix filesystem then people who use command line. All-in-all
usage of OFM is system administration represents Eastern European school of administration
and it might be a better way to administer system that a typical "North American
Way".
The second indispensable tool for shell programmer is
Expect. This is a
very flexible application that can be used for automation of interactive sessions
as well as automation of testing of applications.
Usually people who know shell and awk and/or
Perl well are usually considered to be advanced Unix system
administrators (this is another way to say the system administrators who does not
know shall/awk/Perl troika well are essentially a various flavors of entry-level
system administrators no matter how many years of experience they have). I would
argue that no system administrator can consider himself to be a senior Unix system
administrator without in-depth knowledge of both one of the OFMs and Expect.
No system administrator can consider himself to
be a senior Unix system administrator without in-depth knowledge of
both one of the OFMs and Expect.
An OFM tends to educate the user about the Unix filesystem in some subtle, but
definitely psychologically superior way. Widespread use of OFMs in Europe,
especially in Germany and Eastern Europe, tend to produce specialists with substantially
greater skills at handling Unix (and Windows) file systems than users that only
have experience with a more primitive command line based navigational tools.
And yes, cd navigation is conceptually broken. This is not a bizarre
opinion of the author, this is a fact: when you do not even suspect
that a particular part of the tree exists something is conceptually broken.
People using command line know only fragments of the file system structure like
blinds know only the parts of the elephant. Current Unix file system with,
say, 13K directories for a regular Solaris installation, are just unsuitable
for the "cd way of navigation"; 1K directories was probably OK. But when there are
over 10K of directories you need something else. Here quantity turns into quality.
That's my point.
The page provides rather long quotes as web pages as web pages are
notoriously unreliable medium and can disappear without trace. That makes
this page somewhat difficult to browse, but it's not designed for browsing; it's
designed as a supplementary material to the university shell course and for self-education.
Note:
A highly recommended shell site is
SHELLdorado
by Heiner Steven.
This is a really excellent site with the
good
coding practice section,
some interesting example scripts and
tips
and tricks
A complementary page with
Best Shell Books Reviews
is also available. Although the best book selection is to a certain extent individual,
the selection of a bad book is not: so this page might at least help you to avoid
most common bad books (often the book recommended by a particular university are
either weak or boring or both;
Unix Shell by Example
is one such example ;-). Still the shell literature is substantial (over a hundred
of books) and that mean that you can find a suitable textbook. Please
be aware of the fact that that few authors of shell programming books have a broad
understanding of Unix necessary for writing a comprehensive shell book.
IMHO the first edition of O'Reilly
Learning Korn Shell
is probably one of the best and contains nice set of
examples
(the second edition is more up to date but generally is weaker). Also the first
edition has advantage of being available in HTML form too (O'Reilly Unix CD). It
does not cover ksh93 but it presents ksh in a unique way that no other book does.
Some useful examples can also be found in
UNIX Power Tools Book( see
Archive of all shell scripts (684 KB); the book is available in HTML from one
of O'Reilly CD bookshelf
collections).
Still one needs to understand that Unix shells are pretty archaic languages which
were designed with compatibility with dinosaur shells in mind (and Borne is a dinosaur
shell by any definition). Designers even such strong designers as David Korn were
hampered by compatibility problems from the very beginning (in a way it is amazing
how much ingenuity they demonstrate in enhancing Borne shell; I am really
amazed how David Korn managed to extend borne shell into something much more usable
and much loser to "normal" scripting language. In this sense ksh93 stands like a
real pinnacle of shell compatibility and the the testament of the art of shell language
extension).That means that outside of interactive usage and small one page scripts
they generally outlived their usefulness. That's why for more or less complex tasks
Perl is usually used (and should be used) instead of shells. While shells continued
to improve since the original C-shell and Korn shell, the shell syntax is frozen
in space and time and now looks completely archaic. There are a large number
of problems with this syntax as it does not cleanly separate lexical analysis from
syntax analysis. Bash 3.2 actually made some progress of overcoming most archaic
features of old shells but still it has it own share of warts (for example last
stage of the pipe does not run in on the same level as encompassing the pipe script)
Some syntax features in shell are idiosyncratic as Steve Bourne played with Algol
68 before starting work on the shell. In a way, he proved to be the most influential
bad language designer, the designer who has the most lasting influence on Unix environment
(that does not exonerate the subsequent designers which probably can take a more
aggressive stance on the elimination of initial shell design blunders by marking
them as "legacy").
For example there is very little logic in how different types of blocks
are delimitated in shell scripts. Conditional statements end with (broken) classic
Algor-68 the reverse keyword syntax: 'if condition; then echo yes; else
echo no; fi', but loops are structured like perverted version of PL/1 (loop
prefix do; ... done;) , individual case branches blocks ends with
';;' . Functions have C-style bracketing "{", "}". M. D. McIlroy as Steve
Borne manager should be ashamed. After all at this time the level of compiler construction
knowledge was pretty sufficient to avoid such blunders (David
Gries book was published in 1971) and Bell Labs staff were not a bunch of enthusiasts
;-).
Also the original Bourne shell was a almost pure macro language. It performed variable
substitution, tokenization and other operations on one line at a time without understanding
the underlying syntax. This results in many unexpected side effects: Consider a
simple commandrm $file
If variable $file is accidentally contains
space that will lead to treating it as two separate augments to the rm command with
possible nasty side effects. To fix this, the user has to make sure every
use of a variable in enclosed in quotes, like in rm "$file".
Variable assignments in Bourne shell are whitespace sensitive. 'foo=bar'
is an assignment, but 'foo = bar' is not. It is a function call with
"= "and
"bar" as two arguments. This is another
strange idiosyncrasy.
There is also an overlap between aliases and functions. Aliases are positional
macros that are recognized only as the first word of the command like in classic
alias ll='ls -l'. Because of this, aliases have several limitations:
You can only redirect input/output to the last command in the alias.
You can only specify arguments to the last command in the alias.
Alias definitions are a single text string, this means complex functions
are nearly impossible to create.
Functions are not positional and can in most cases can emulate aliases functionality:
ll() { ls -l $*; }
The curly brackets are some sort of pseudo-commands, so skipping the semicolon in
the example above results in a syntax error. As there is no clean separation between
lexical analysis and syntax analysis removing the whitespace between the opening
bracket and 'ls' will also result in a syntax error.Since the use of variables
as commands is allowed, it is impossible to reliably check the syntax of a script
as substitution can accidentally result in key word as in example that I found in
the paper about fish (not that I like or recommend fish):
if true; then if [ $RANDOM -lt 1024 ]; then END=fi; else END=true; fi; $END
Both bash and zsh try to determine if the command in the current buffer is finished
when the user presses the return key, but because of issues like this, they will
sometimes fail.
Dr. Nikolai Bezroukov
Old News ;-)
2009
2008
2007
2006
2005
2004
2003
2002
freshmeat.net Project details for Advanced Bash Scripting Guide 5.3
Recreational scripts were added, such as an almost full-featured
Perquacky clone script, a script that does the "Petals Around the Rose"
puzzle, and a crossword puzzle solver script. There is also the usual batch
of bugfixes and other new material.
[Apr 3, 2008]
LinuxDevCenter.com -- Excerpt from Linux Cookbook, Part 1
[Mar 31, 2008]
Bash Shell Programming
in Linux by P. Lutus
Simple but nice intro with good examples
[Mar 30, 2008]
Linux, Unix, -etc Unix Shell
Scripts
These are simple stand-alone scripts. You are unlikely to find anything very
impressive here if you already hack your own scripts, but a novice might find
some of the ideas new.
Handy
"Desktop" Thingies
Dates and
Times
Text Utilities
File Utilities
Simple
Databases
Administration
Networking
Special
Stuff for Unix Haters
E-mail
Miscellaneous
Handy Things
[Oct 31, 2007]
freshmeat.net Project details for Bash Debugger
Version 3.1-0.09
The Bash Debugger (bashdb) is a debugger for Bash scripts. The debugger command
interface is modeled on the gdb command interface. Front-ends supporting bashdb
include GNU-Emacs and ddd. In the past, the project has been used as a springboard
for other experimental features such as a timestamped history file (now in Bash
versions after 3.0).
Release focus: Minor bugfixes
Changes:
This release contains bugfixes accumulated over the year and works on bash version
3.2 as well as version 3.1.
UNIX System Administration
Tools
chrec
Records system changes to flat text log files. (Bourne shell)
View the
README
Download
version 2.2 - gzipped tarball, 5 KB
Last update: June 2007
[Oct 27, 2007]
System Administration Toolkit Standardizing your UNIX command-line tools
Freelance Writer, Consultant
22 Aug 2006
Examine methods for standardizing your interface to simplify movements between
different UNIX® systems. If you manage multiple UNIX systems, particularly
in a heterogeneous environment, then the hardest task can be switching between
the different environments and performing the different tasks while having
to consider all of the differences between the systems. This article does
not cover specific differences, but you'll look at ways that can provide
compatible layers, or wrappers, to support a consistent environment.
[Oct 3, 2007]
freshmeat.net Project details for rr
rr is a basic command-line utility designed to retain/recall file and directory
paths. This is done by treating the filename itself as a unique key to be referenced
for future rr program calls. The purpose of this is to assist the user in shorthand
typing and/or not having to remember arbitrary full paths. So, for example,
"/etc/httpd/conf/httpd.conf" can be referenced as "// httpd.conf" in daily operations.
Release focus: Initial freshmeat announcement
[Sep 27, 2007]
freshmeat.net Project details for Closebracket
This is essentially reinvention and re-implementation of OFM context sensitive
linage of extension to commands (via ext file) the additional twist that if there
is no extension file type (for example as discovered by file) is used.
Closebracket lets you define multiple shell actions in a single command to
speed up the typing of the most repetitive shell commands. It includes ']' and
'][' commands, which are located near the "Enter" key and are easy to type quickly.
They invoke primary and secondary actions respectively.
Linux tip Controlling the duration of scheduled jobs
Say you need to debug a pesky problem by running some traces for 30 minutes
at midnight, or you would just like to use your Linux system as an alarm clock.
This tip helps you stop jobs, such as those started with the cron
and at capabilities, after the jobs have run for a certain time,
or when some other criteria are met.
... ... ...
Listing 2 shows an enhanced runclock2.sh script that captures
some information about the process ids of the shell and the xclock
processes, along with the output of the script and the output of the ps
command, showing the process status for xclock, after the shell
completes.
Listing 2. Gathering diagnostic information runclock2.sh
[ian@attic4 ~]$ cat runclock2.sh
#!/bin/bash
runtime=${1:-10m}
mypid=$$
# Run xclock in background
xclock&
clockpid=$!
echo "My PID=$mypid. Clock's PID=$clockpid"
ps -f $clockpid
#Sleep for the specified time.
sleep $runtime
echo "All done"
[ian@attic4 ~]$ ./runclock2.sh 10s
My PID=8619. Clock's PID=8620
UID PID PPID C STIME TTY STAT TIME CMD
ian 8620 8619 0 19:57 pts/1 S+ 0:00 xclock
All done
[ian@attic4 ~]$ ps -f 8620
UID PID PPID C STIME TTY STAT TIME CMD
ian 8620 1 0 19:57 pts/1 S 0:00 xclock
Notice that the parent process id (PPID) in the first output of ps
is 8619, which is the process id (PID) of the script. Once the script terminates,
the clock process becomes an orphan and is assigned to be a child of the
init process—process 1. The child does not terminate immediately
when its parent terminates, although it will terminate when you log out of the
system.
Terminating a child process
The solution to the problem of non-terminating child processes is to explicitly
terminate them using the kill command. This sends a signal
to a process, and that usually terminates the process. Later in this tip you
see how a process can trap signals and not terminate, but here you use the interrupt
signal (SIGINT) to terminate the clock.
To see a list of signals available on your system, use the kill
command with the -l option, as shown in Listing 3. Note that some
signals are common to all Linux systems, but some may be specific to the particular
machine architecture. Some, such as floating point exceptions (SIGFPE) or segment
violation (SIGSEGV), are generated by the system, while others such as interrupt
(SIGINT), user signals (SIGUSR1 or SIGUSR2), or unconditional terminate (SIGKILL),
can be sent by applications.
[Aug 12, 2007]
BigAdmin Submitted Tech Tip Automatically Saving Script Output Using the nohup Utility
by István Bátori
Can simplify logging for shell scripts
... Scripts can have their own logging functions, but the output also contains
important information. Saving the output is typically solved by output redirection,
for example:
nohup mynohupscript.sh >/var/log/myscript.log 2>&1
How do you feel about typing such a long command? Isn't it boring? It's a
lot to type. And it's confusing to read the long, uninteresting ballast in an
admin document, especially if the script has many arguments and options and
the log file contains the start date.
Copy the
nohup.txt file, changing the ".txt" suffix to ".sh", and save into the
/opt/bin directory. Then give it execute rights.
cp nohup.sh /opt/bin
chmod 755 /opt/bin/nohup.sh
chown root:admin /opt/bin/nohup.sh
Usage
There are two different ways to use the wrapper script:
1. Start the wrapper script directly from the command line:
/opt/bin/nohup.sh <command> [<args...>]
For example:
/opt/bin/nohup.sh foo.sh
2. Put the following into a script as the first line:
#!/usr/bin/ksh /opt/bin/nohup.sh
For example:
cat >/tmp/nohup_test.sh <<EOF
#!/usr/bin/ksh /opt/bin/nohup.sh
# example script to demonstrate nohup.sh usage
default_cycles=3
default_sleeping=1
cycles=${1:-$default_cycles}
sleeping=${2:-$default_sleeping}
i=0
while [[ $i -lt $cycles ]]
do
echo "$(date): $i"
sleep $sleeping
i=$((i+1))
done
EOF
Executing nohup_test.sh automatically calls
nohup.sh, which starts the script with the
nohup utility and redirects the script output into
/var/tmp/nohup_test_<date>.log.
Configuration
Both usage methods have the same functionality. The functionality can be
configured by shell variables, as follows:
nohup.sh starts the script with the interpreter
specified in the $NOHUP_SHELL variable. If the
variable is not set, the default interpreter is started according to the
script file's extension:
A .sh script is started with
/usr/bin/ksh.
A .pl script is started with
/usr/bin/perl.
A .rb script is started with
/usr/bin/ruby.
A .py script is started with
/usr/bin/python.
Output is redirected into the directory specified by
$NOHUP_LOGDIR. If $NOHUP_LOGDIR
is not set, the /var/tmp directory is used.
The log file name is the script name postfixed with the date, for example,
foo_2007_03_23_11_11_22.log.
Set NOHUP_MONITORING=yes to automatically
start the monitoring of the command output with tail
-f just after starting the script in the background. You can stop
this monitoring process without stopping the nohup
command.
[Aug 10, 2007]
Speaking UNIX, Part 6 Automate, automate, automate!
Level: Intermediate
Martin Streicher ( |
|
