Softpanorama

May the source be with you, but remember the KISS principle ;-)
Contents Bulletin Scripting in shell and Perl Network troubleshooting History Humor

Nikolai Bezroukov. Portraits of Open Source Pioneers

For readers with high sensitivity to grammar errors access to this page is not recommended :-)


Bourne Shells Family

Copyright 1996-2005, Dr. Nikolai Bezroukov. This is a copyrighted unpublished work. All rights reserved.

"The lyfe so short, the craft so long to lerne,''
Chaucer c. 1340–1400
(borrowed from the bash FAQ)

Few people understand that classic Bourne-style shells are really powerful (albeit slightly outdated) scripting language. And despite their age in some respects they are still unsurpassed as very high level languages.  It is actually shells that make the power of Unix kernel available to ordinary people :-).

Korn shell was the oldest of a series of  successful scripting languages produced by Unix  (ksh was probably the first true "glue" language, awk added some new data processing capabilities that were later integrated by Perl into the Bourne-Cshell-Kornshell framework). Because it was invented for a very flexible Unix environment, Unix shells incorporate some non-trivial capabilities and are underutilized by most users.  The fact that Bourne/Korn shell survived for so many years despite the availability of alternatives suggests that it probably represented the first "good enough" implementation of scripting language.

Moreover "Shell+C" combination was the first practical components-based programming  methodology: an important idea here is to use rather small C programs and preexisting Unix tools as a sort of Lego to create a solution of the problem with the shell serving as a glue. That approach can be very successful even today (in the days of monstrous middleware and Java misunderprogramming ;-) and  it is definitely remains one of the best approaches in creating prototypes for complex systems.  In a sense Unix promotes LEGO approach to building prototypes and a special ("unixoidal") type of thinking about architecture in terms available in Unix (including such powerful protocols as pipes and sockets).  In case this way of thinking and componentization compete with architectural thinking of talented people who did not have exposure to Unix the difference in favor of Unix style thinking can be pretty dramatic.  This was actually the case even when on the opposite side was Donald Knuth.

It's very important to realize that shell in general and Korn shell in particular to a certain extent is a quintessence of Unix programming methodology. As such Korn shell is pretty close to the crown jewel of Unix as a programming environment and it does deserve in-depth study.

I would like to stress again that both Bourne-shell and Korn-shell were not "instant" final products. They were incremental improvements in a long chain of development (Bourne shell is based on Mashey shell, which in turn was an extension of the original Thompson shell, which in turn was a derivative of Multics shell ;-). 

As John Mashey, the author of Mashey shell (the precursor of the Bourne shell and the successor of the original Thompson shell, see History), mentioned (personal correspondence), people in Bell Labs instantly realized that shell represents a new a highly productive programming environment: 

People all over BTL wrote shell scripts, but anything big was just too painful in the original [Thompson] shell, so we added simple variables, reasonable control structures, the variable path search which meant that individuals and groups could establish their own libraries of scripts not in /bin or /usr/bin], and pre-initialized variables for home directory. As a result, in BTL Piscataway, there was an explosion in the use of shell as a scripting language, and there were large number of stored scripts. There was some argument with Steve later, as PY had accumulated so much of this, that the slow speed of the original Bourne shell implementation would have been very painful.

I would like to stress that at this time Bell Labs were populated by very talented people who became pioneers in several areas of OS design. Still the invention of scripting is one of their top achievements. Among other achievements that directly influenced shell programming and shell language capabilities were pipes, regular expressions and the hierarchical filesystem. BTW the level of integration of those facilities into shell (especially named pipes and regular expressions) are still unsurpassed (Perl has a better regular expressions integration, but its pipes support simply sucks). Unfortunately those capabilities are under-appreciated and under-used by most users and system administrators.  This page is the result of my desire somewhat improve this situation ;-).

It truly unfortunate that many capabilities of shell that substantially increase its usefulness are a kind of esoteric art. Even good .profile and .kshrc files are a such a rare thing that can be used as a good interview question for advanced Unix sysadmin positions ;-(. As David Korn aptly said (bold italic is mine  - BNN):

There are many people who use UNIX or Linux who IMHO do not understand UNIX. UNIX is not just an operating system, it is a way of doing things, and the shell plays a key role by providing the glue that makes it work. The UNIX methodology relies heavily on reuse of a set of tools rather than on building monolithic applications. Even perl programmers often miss the point, writing the heart and soul of the application as perl script without making use of the UNIX toolkit.

It's probably true that Perl is better for any sizable programming project. And IMHO it is one of the few scripting languages that to a large extent retained the spirit of Born-Korn-Cshell troika. But as I already mentioned above,  the problem is that Perl pipe support is weak, and that makes many tasks that involve their usage inconvenient.  Actually, this shortcoming makes Perl much less expressive then ksh93 and bash for the large class that can be solved using pipes.  Due to this shortcoming often Perl programmers are implicitly forced to reinvent the bicycle (often in a form of loops) instead of writing a pipe. But other than that, Perl it's a better, higher level language with much better debugging capabilities and with a good integration with Unix API.

Still you can and should use shell for programming in cases where simplicity and transparency is very important. For programming  among a half dozen of available shells I recommend ksh93. Bash is a better interactive shell, but  not for programming, especially outside Linux environment.  It also suffers from feature creep. While ksh93 has some interesting programming constructs and is more clean design (and especially code) then bash, no claim of exclusivity is made, especially as shells copy one another.

Bash 3.x  are now de-facto standard among open source shells. Actually bash was the best free shell until ksh93 became freely available. Now situation is not that clear and ksh93 has a definite edge if programmability or quality of implementation are important. Bash still holds his own as an interactive shell (Bash  3.x has a built-in interactive debugger that is more than a diagnostic tool,  it is a fine way to explore the very guts of bash), but you should consider using Orthodox File Manager like mc to use bash fully.

Generally, I would say that FSF dropped the ball and the product development was stalled.  The implementation became too baroque. The quality and elegance of ksh93 implementation is higher and its port to windows is very impressive (the best port of Unix shell ever).  The only important enhancement for the last decade was the built-in debugger added in version 3.0.  The latter is a very nice addition and I never understood why nobody added a simple debugger to the shell after, say, 1990, when PC became too powerful for personal computers anyway :-).

What is really important is that ksh93 has superior pipes support  both in comparison with other shells and Perl. From the programming language design point of view ksh93 has better integrated with piping constructs that other widespread shells (rc shell is the only more powerful in this respect).  BTW very few people know and understand that dtksh, the Desktop KornShell that comes standard with CDE on all commercial Unixes is an extension of ksh93 that contains a set of built-in commands to interface with Motif, but can be used as a regular shell as well. For example in Solaris 9 it can be found at /usr/dt/bin/dtksh.

dtksh, the Desktop KornShell that comes standard with CDE on all commercial Unixes is an extension of ksh93 that contains a set of built-in commands to interface with Motif, but can be used as a regular shell as well.  If can (and probably should) be used instead of Posix ksh.

The only other shell that has advanced piping facilities is rc-shell and that actually makes rc-shell sometimes superior to Perl in scripts that use heavy piping. See also Softpanorama Perl bibliography.

Here we will discuss using shell in Unix environment. But it can be used both in Windows environment too, although in the Windows environment it's much less powerful and stable. But my students did some amazing things with Uwin  -- ksh93 in Windows environment (preferable to Cygwin from the quality standpoint, but less popular).  BTW for NT and Win2K you can also try Jscript -- a very interesting and a pretty novel shell that shares with REXX a very important capability to be used both as shell and macrolanguage (the most attractive feature of OS/2 and Amiga).  Jscript which is a pretty conformant implementation of JavaScript that provides an unique advantage of using the same language for HTML scripting and shell programming.  For Ms Office JScript can be used instead of VBA. I do not recommend VBScript, but your mileage may vary, as most examples and books are devoted to this language.

Other than more powerful piping constructs, ksh93 has some other nice features:

Of course nothing is perfect and there are some minor things that are missing in ksh93:

Still I prefer ksh93 to bash for shell programming because of its better supports of pipes. A very cool feature of ksh93 is `|&` which is a way of getting a pipe to a background process (co-processes). You can also call it a bi-directional pipe. You can then use `read` and `print` to send and receive stuff from the  process started with `|&`. For example, if I need to do floating point math, I can start `bc` in the background with `|&`. Interesting. with zsh, |& is a pipe which takes both stdout and stderr, similar to >& and >>& for redirection. It's handy, especially when you need to use a program with a rather verbose --help option. Bash doesn't recognize |& at all :-(.

But the main shortcoming of bash it its treatment of the last stage of the pipe. Consider the following example derived from Slashdot discussion  of David Korn interview:

x="hello"
echo "goodbye" | read x
echo $x

In both bash and pdksh, running the script would incorrectly return "hello", because the "read x" would be run in a subshell with its own environment. The subshell would then close without affecting the instance of x in the main shell. That's suck. In ksh, however, putting 'read' at the end of a pipeline does not create a subshell, so running the script would return "goodbye" just as any programmer might expect. Scripts that rely on classic Unix shell idiom "echo $string | read x y junk" are convenient to parse space-separated lists, and of course these statements will never work in bash (you need to do a few backtick/sed operations). Actually, the best way to do this bash is the following kludge. You need to save off the current argument list ($*) if necessary, and then:

set -- $string
x=$1; shift
y=$1; shift
junk="$*"

As you can see the ksh implementation is much cleaner and elegant.

The example above is a small, but nice illustration of my main point: one of the most important concepts traditionally associated with shells is piping -- I urge you to learn this concept as deeply as possible -- this is actually a pretty powerful programming paradigm and many classic software products, including such complex as compilers,  can re restructured to a solution my several stages of pipes (coroutines were actually invented by a compiler writer). Unfortunately compiler construction it now out of fashion and seldom taught in nowadays university curriculums.

Again I feel that sub par support of piping is the major weakness of Perl that prevented it from overtaking the Unix scripting despite unique favorable historic conditions (Perl became the first Web scripting language).

One of the most important concepts traditionally associated with shells is piping;
it is actually a pretty powerful and largely underutilized programming paradigm

This page emphasizes ksh93 which I think is the most underappreciated shell in Unix world.  But I am not a shell fundamentalist and bash is pretty much OK as a login shell and it is somewhat more user friendly as a command line shell (please note that the recent versions of ksh93 support arrow-based navigation of history and TAB-based file completion).

For an long time, beginners were better off from starting with tcsh that is the most user friendly shell out of the box. But right now bash  (v 3.2 and later) is probably competitive to tcsh as for the user friendliness, so why bother. Most user friendly features introduced by csh like 'autolist' now have their counterparts in bash too and the latest versions of bash support up/down browsing of command history (the hallmark of csh and tcsh). Command completion mode is now also borrowed from tcsh -- all variants can be listed after an ambiguous completion by pressing tab (bash requires two tab's), e.g.:

 % ls /usr/l[tab] lib/ libexec/ local/.

In bash this can be set via readline. In /etc/inputrc or ~/.inputrc, put "set show-all-if-ambiguous on". This'll cause bash (which uses readline) to list all possibilities after a single tab. Check the readline man page for other nifty stuff, including the incremental reverse-search-history (default Ctrl-R).

Another important point. Shell is far from being perfect for navigation in a complex filesystem. It does have a rudimentary features like pushd/popd, but there are not enough. I've never see a Unix administrator who is able to complete in speed and productivity with sysadmin who is an expert user of a good orthodox file manager. There are several open source implementations of such manages.  I recommend to use Midnight Commander -- see my OFM page. It's more or less feature-complete and is OK interactive shell for Linux and other Unixes. A little bit heavy for remote usage, but with the current line speed still OK. Some versions are pretty buggy. From 2000 to 2002 Midnight Commander suffered from the desire to turn it into a graphical gnome file manager, but later Pavel Roshkin managed to straiten things up and remove most of the gnome-related code (he preserved the dependence on glib, though).  See my on-line book about Orthodox file managers.

Other useful links:  Editors LinksUnix/Linux links. Overview of books about shells is in Shell bibliography. Again, a highly recommended shell site is SHELLdoradoYou should probably visit it first. More or less useful dotfiles can be found at dotfiles.com home. Check it out.

Dr. Nikolai Bezroukov



Etc

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of environmental, political, human rights, economic, democracy, scientific, and social justice issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit exclusivly for research and educational purposes.   If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner. 

ABUSE: IPs or network segments from which we detect a stream of probes might be blocked for no less then 90 days. Multiple types of probes increase this period.  

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least


Copyright © 1996-2016 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License.

The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Created May 16, 1996; Last modified: September 28, 2017