Plines

Plines is both a framework for building applications in Bourne-compatible shell, and the name of the primitive object type that is manipulated by that framework in order to build up applications. It was created by MicahCowan, in order to provide a foundation for the Niwt project.

Plines's purpose is to exploit the power and flexibility of the Unix shell's pipelining ability - the power to split responsibilities into discrete programs, streaming data from one to another. A classic example of the power of Unix pipelines is this one, which has been around for decades:

The troff command would process an input format (still used today for Unix manpages) into commands for a printer or typesetting machine; the tbl, eqn, and pic commands processed files that had troff input, with sections of data in specialized languages that each of these separate commands new how to translate into troff's typesetting language in order to produce formatted tables, mathematical equations, or diagrams (respectively). You didn't need to shove all these different formatting languages into troff, and absolutely anyone could write their own additional parser, to support new features and translate them into troff's language, without having to modify any existing applications; just plug it into the pipeline and you're done.

The Problem

This delightful paradigm makes it a snap to cobble together a variety of little programs into a larger application. However, the more complex such an application gets, the less user-friendly it is.

Take, for example, the following pipeline that roughly corresponds to the one that is executed by Niwt in a basic invocation:

At this mild, but already somewhat awkward level of complexity, managing your nifty Unix command pipeline starts to become something of a chore. You may know what functionality you want to plug into this pipeline, but it's easy to become confused as to where to plug your new command in. Especially if you're not clear on the functionality and responsibilities of the other commands.

Even if you're completely familiar with every detail of your pipeline... it's become rather large. If you're editing it from the command-line, you can't simply retype it each time; you have to backup in your shell's history, move the cursor to the right point for insertion or adjustment, and type your changes there.

You can get around some of the problem by slapping the whole thing into a shell function or script, but then editing the pipeline means directly editing a file. It's no longer flexible, and you can't just take this pipeline and pass it around to friends and (especially) non-technical users, and expect them to comprehend it well enough to make whatever adjustments they may need. If all they want to do in niwt is change the name of the file that gets saved, they don't want to be forced to edit a pipeline to place some sed command invocation or what have you; they want to add a command option.

A Solution

Enter Plines.

Plines removes these issues by doing two things:

For instance, niwt's overall pipeline (the primary pipeline that is executed, is always called "it"), is composed of the following three sub-plines (which themselves may be decomposed into further plines):

(See NiwtInternals for more detailed descriptions of each of these.)

When you invoke the --head option of niwt, this option simply adds a filter into the requests-generate pline (actually, into an exec chain that resides in a sub-pline of requests-generate), which transforms the automatically-generated HTTP GET request into a HEAD request. The user doesn't need to know the details of the pipeline, or what sort of shell command is necessary to achieve the desired results; they just run niwt --head example.com and it does what needs to be done.

Scripted pline manipulation

In fact, the niwt application itself is just a very, very short shell script that sets up the PATH environment variable, and then runs the single shell command

The niwt-init.sh file contains shell instructions to configure the default plines, and define options that users can use to manipulate them (without knowing that they're manipulating anything!).

If you run the command niwt --show it, niwt will describe the contents of its "it" pline, breaking it apart recursively (example abbreviated):

A sample of the shell commands from niwt-init.sh that organized this structure:

When a plines_pline_define or _append (there's also a _prepend) is given a definition that starts with the tag "pline:", it indicates that that portion of the pipeline is defined by another sub-pline, which must be expanded before the full pipeline is executed by the shell. Without a tag, the definition is assumed to refer to actual shell code to be executed (for instance, see the initial definitions of the requests-generate and niwt-fetcher plines).

The code that defines niwt's --head option (found in niwt-opt.sh) is:

(For information on the meaning behind plines_chain_append, you'll need to read PlinesExecChain.)

Now, the cool thing is that niwt can read in a user config file (~/.niwtrc, or use the --config option just like niwt itself does) that adds user-defined options (see TryingOutNiwt for one example).

Command-line pline manipulation

You don't need to write shell code invocations to manipulate plines:

The --pline manipulation is available from both niwt and plines; check out the --help output form either command (or their manpages) for more information.

Plines (last edited 2011-02-14 01:12:09 by MicahCowan)