     UP(3)                                                                 UP(3)

     UP
          up - user mode pipeline library

     SYNOPSIS
          #include <up.h>

          UP * upAllocate();
          upAddDevice(UP *, ...);
          upSetupDone(UP *);
          UP_BOOL upReadWrite(UP, int timeout);
          upClose(UP *);

     DESCRIPTION
          The up library provides a user mode pipeline capability.

          A user mode pipeline is similar in concept to a normal UNIX pipeline
          in which various filter programs are used to transform a stream of
          data.  But in this case, the pipeline is run in a single process by
          passing the data stream through various filtering modules to transform
          data.  This avoids context switches, provides better error handling,
          avoids most uses of temporary files, and allows for the pipeline to be
          dynamically changed.  A user mode pipeline can handle multiple files
          during the processing such as unpacking a tar archive, compressing
          each file separately, and then repacking the resulting files into
          another tar archive.

          In the following discussion, the various filtering modules are called
          devices even though the term is somewhat misleading since no hardware
          is involved.  However, data is read from them and is sent to them, and
          so the term has some validity.

          A user mode pipeline consists of a list of two or more devices
          arranged from "left" to "right" in a straight line.  The leftmost
          device is the "reading" device, and must generate data for the
          pipeline.  The reading device can generate data from external sources
          such as files or standard input, or from internal data such as a
          string or data generated by the calling program.  The rightmost device
          is the "writing" device and must consume data from the pipeline.  The
          writing device can send data to an external source such as files or
          standard output, or to internal data such as a string to be read by
          the calling program.  All devices in the middle of the pipeline (if
          any) are "filters" which pass the data through them, transforming the
          data as required.

          The interface to a device is uniform so that new types of devices can
          be easily added to the library.  The interface provides for storing
          input and output into buffers which are shared between adjacent
          devices to minimize copying.  The interface also provides
          synchronization between the devices for the correct transfer of file
          names, data, and end of files.

                                        - 1 -      Formatted:  November 21, 2001

     UP(3)                                                                 UP(3)

     MODES OF OPERATION
          Data in the pipeline can be processed using one of three different
          modes of operation.  Which of these modes is used depends on the needs
          of the calling program and its sophistication.

          Firstly, data can be "written" to the left end of the pipeline by the
          calling program which will "push" the data through the pipeline to
          some output device which consumes it (such as writing it to a file).
          In this mode of operation, the calling program blocks until the
          pipeline processes enough data to make room for storing the program's
          new data.

          Secondly, data can be "read" from the right end of the pipeline, which
          will "pull" the data through the pipeline from some input device which
          generates it (such as reading from a file).  In this mode of
          operation, the calling program blocks until the pipeline processes
          enough data to provide some to return to the calling program.

          Finally, data can be transferred "asychronously" through the pipeline
          by the calling program so that the program can generate input data if
          desired, consume output data if desired, and never block.  This mode
          of operation makes use of the "poll" system call if necessary, and
          allows the calling program to run the pipeline in an event-driven
          manner.

     CREATING A USER MODE PIPELINE
          A user mode pipeline is based on the UP structure which is allocated
          by calling the upAllocate routine.  The returned UP structure is then
          used for all further user mode pipeline calls.  Once allocated,
          devices can be attached sequentially to the pipeline.  The routines to
          add devices have names of the form upAddDevice, where Device is
          replaced by the desired device.  The argument list for each of these
          devices varies depending on the needs of the device.

          Devices are added from "left" to "right".  That is, the data in the
          pipeline will flow from the first device added to the last device
          added.  The first device must be one of the devices which can read
          data from some external source.  The last device must be one of the
          devices which can write data to some external source.  Intermediate
          devices must be one of the devices which are filters, which read data
          from another device and write data to another device.

          Some devices require that the size of their data be known before the
          data is seen.  (For example, this is true for the PipeTar device,
          because the headers for the files contain the file sizes.) If one of
          these devices is used, and the file size is not known from the
          previously added devices, then the special PipeFindSize device is
          automatically added before the device which requires it.  The
          PipeFindSize device will buffer a file's data in memory or in a
          temporary file, and will then supply the actual file size with the
          file name to the next device.

                                        - 2 -      Formatted:  November 21, 2001

     UP(3)                                                                 UP(3)

          The automatically added PipeFindSize device uses default parameters
          for the amount of memory it can consume, and whether or not file sizes
          should sometimes be believed.  You can explicity add your own
          PipeFindSize device in the pipeline in order to override these default
          parameters.

          When all of the devices have been added to the pipeline, the
          upSetupDone routine must be called.  After this call, no further
          devices can be added to the pipeline, and the pipeline is ready to do
          I/O.

     RUNNING THE PIPELINE
          The usual way to run the pipeline is asychrounously by calling
          upReadWrite repeatedly until the pipeline is finished.  Each call
          performs one cycle of I/O for the devices in the pipeline.

          The routine accepts a timeout value in milliseconds similar to select,
          so that the call does not need to block if that is not wanted.  A
          value of -1 means an indefinite wait, whereas a value of 0 means no
          wait at all.

          If there is an error, then the upReadWrite routine returns UP_FALSE.
          If there is no error, then the upIsFinished routine needs to be called
          to see whether or not the pipeline is finished.  If it is not
          finished, then you can call upReadWrite again to perform more I/O.

          When the pipeline is finished, you call upClose to close all of the
          devices and free their resources.  It returns UP_TRUE if there were no
          problems.  After the pipeline is closed, it can be re-used by new
          devices just like when it was first allocated.

          If the pipeline is no longer needed, calling upFree will close the
          devices (if necessary) and free the UP structure.

     LIST OF FUNCTIONS
          Function
               Description

          upAddReadEndpoint
               Add a device to allow reading from the pipeline

          upAddWriteEndpoint
               Add a device to allow writing to the pipeline

          upAddReadNull
               Add a device which reads no data

          upAddWriteNull
               Add a device which writes no data

                                        - 3 -      Formatted:  November 21, 2001

     UP(3)                                                                 UP(3)

          upAddReadFd
               Add a device which reads data from a file descriptor

          upAddWriteFd
               Add a device which writes data to a file descriptor

          upAddReadDirectory
               Add a devices which reads files from a directory

          upAddWriteDirectory
               Add a device which writes files into a directory

          upAddReadString
               Add a device which reads data from a string

          upAddWriteString
               Add a device which writes data to a string

          upAddReadFile
               Add a device which reads data from a file

          upAddReadFileList
               Add a device which reads data from a list of files

          upAddWriteFile
               Add a device which writes data to a file or a sequence of files

          upAddProcessString
               Add a device which sends data through an external process

          upAddProcessArgs
               Add a device which sends data through an external process

          upAddProgress
               Add a device which provides progress information

          upAddLogFile
               Add a device which logs information

          upAddCat
               Add a device which concatenates its input files into one output
               file

          upAddGnuZip
               Add a device which compresses its input files using GZIP

          upAddGnuUnzip
               Add a device which uncompresses its intput files using GUNZIP

          upAddCompress
               Add a device which compresses its input files using COMPRESS

                                        - 4 -      Formatted:  November 21, 2001

     UP(3)                                                                 UP(3)

          upAddUnCompress
               Add a device which uncompresses its input files using UNCOMPRESS

          upAddUUDecode
               Add a device which uuencodes its input files into one output file

          upAddUUEncode
               Add a device which uudecodes its input file into output files

          upAddArchive
               Add a device which archives its input files using AR

          upAddUnArchive
               Add a device which unarchives its input file using AR

          upAddTar
               Add a device which tars its input files using TAR

          upAddUnTar
               Add a device which untars its input files using TAR

          upAddList
               Add a device which produces a listing of the files

          upAddGrep
               Add a device which searches for lines which match a regular
               expression

          upAddFindSize
               Add a device which calculates the true size of files

          upAddFilter
               Add a device which selects which files are transferred through it

          upAddRemoveData
               Add a device which transmits file names and status but removes
               the file data

          upAddFindFileData
               Add a device which reads the data for the file names sent through
               it

          upAddMakeFileList
               Add a device which reads a file name list to produce empty output
               files

          upGetVersion
               Return the version of the up library

          upAllocate
               Allocate a new pipeline

                                        - 5 -      Formatted:  November 21, 2001

     UP(3)                                                                 UP(3)

          upClose
               Close a pipeline

          upFree
               Close and free a pipeline

          upSetupDone
               Mark a pipeline ready for use

          upReadWrite
               Run one cycle of I/O for a pipeline

          upPoll
               Wait for the pipeline to have some I/O

          upPollList
               Return a list of polling entries needed by the pipeline

          upRead
               Read data from the right end of a pipeline

          upWrite
               Write data to the left end of a pipeline

          upFlush
               Flush data through a pipeline

          upCanRead
               Return whether data can be read from the right end of a pipeline

          upCanWrite
               Return whether data can be written to the left end of a pipeline

          upIsFinished
               Return whether the pipeline has finished

          upSetSystemPath
               Set the PATH list to be used when securely executing external
               programs

          upGetSystemPath
               Return the PATH list used when securely executing external
               programs

          upSetTemporaryDirectoryPath
               Set the directory to be used for creating temporary files

          upGetTemporaryDirectoryPath
               Return the directory to be used for creating temporary files

                                        - 6 -      Formatted:  November 21, 2001

     UP(3)                                                                 UP(3)

          upClearError
               Clear the error information from the pipeline

          upGetErrorCode
               Get the error code for a pipeline error

          upGetDeviceCount
               Return the number of devices which are in the pipeline

          upGetErrorString
               Return a string describing the last error for the pipeline

          upPrintError
               Print the latest error on the pipeline to stderr

          upGetWarningCount
               Return the count of warnings which have occurred for the pipeline

          upGetMaximumWarnings
               Return the maximum number of warnings which can be printed to
               stderr

          upSetMaximumWarnings
               Set the maximum number of warnings which can be printed to stderr

          upSetDebugFlags
               Set debugging flags for printing of internal pipeline status

     SEE ALSO
          up(1) - A utility which creates and runs a user mode pipeline
          according to the supplied arguments.

     LICENSE
          The up library is licensed under the GNU Lesser Public License.  This
          means that you may freely use this library in your own program, using
          whatever license you wish for your program, and with no restrictions
          on the distribution or use of your program.  However, any
          modifications to the up library itself are covered by the license.

          You should have received a copy of the GNU Lesser General Public
          License along with this library; if not, write to the Free Software
          Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307
          USA

     AUTHOR
          David I. Bell
          dbell@canb.auug.org.au

                                        - 7 -      Formatted:  November 21, 2001