            README file for sgrep version 1.91-alpha -
    	a tool to search and index text, SGML, XML and HTML files using
    	structured patterns

            Copyright (C) 1998  University of Helsinki,
                                Department of Computer Science

            Authors: Jani Jaakkola           Jani.Jaakkola@cs.helsinki.fi
                     Pekka Kilpelainen       Pekka.Kilpelainen@cs.helsinki.fi

    ---------------------------------------------------------------------------

    This README file is intended for describing the new features of
    sgrep-1.91a. If you want to know what sgrep is and what the old features
    are, see:
    http://www.cs.helsinki.fi/~jjaakkol/sgrep.html

    See the section "NEW QUERY LANGUAGE FEATURES" for description of the
    new operators available in version 1.91a.

    Sgrep-1.91 supports 16-bit wide characters and Unicode in XML-documents.
    See the section "WIDE CHARACTER SUPPORT" for information on wide
    characters and UTF-8 and UTF-16 encodings.

    This file (and newer versions of this file) is available from
    http://www.cs.helsinki.fi/~jjaakkol/sgrep/README.txt

    Sgrep is distributed under GNU General Public License. See file COPYING
    for details.

    This piece of software is still under development. This means that:
    - New features might be included before final sgrep-2.0 release.
    - Existing features might be changed.
    - It is guaranteed to have bugs.
    - All suggestions are welcome.
    - All available documentation of the new features is contained in
      this file.

    ---------------------------------------------------------------------------
    NEW FEATURES
    ---------------------------------------------------------------------------

    Major new features since sgrep-1.0 which are already present:
    - Indexing of both structure and content.
    - SGML/XML/HTML scanner.
    - Official Win32 binary.
    - sgtool has been dumped. It never really worked and even when it
      did, it wasn't very useful.
    - Should be completely compatible with older versions of sgrep.
    - Sgrep now supports direct containment. In SGML and XML world this
      means, that you can query children or parents of given elements.
    - Sgrep uses GNU autoconf
    - Also the sources are now available
    - Operators for supporting direct containment
    - Nearness operators
    - 16-bit wide characters and Unicode support

    Features which will be present in sgrep-2.0:
    - Proper documentation
    - Support for querying notations, element type declarations and
      attribute list declarations inside SGML/XML document prolog
    - Scanning of all well-formed XML-documents.

    Features probably won't be present in sgrep-2.0:
    - Regular expressions, since they are probably better handled by other
      software, like Perl. However, sgrep still needs some new options for
      better perl support.

    ---------------------------------------------------------------------------
    Win32-BINARY RELEASE
    ---------------------------------------------------------------------------

    The Win32-binary release contains both sgrep binary and m4 binary.
    Sgrep binary is compiled with MSVC and requires no additional libraries.

    Please note that the examples in this README file and in the sgrep
    WWW-pages have been written using sh shell-syntax. When you use sgrep
    under the windows shell, "COMMAND.COM" you have to either use the -f option
    or translate query from:

    % sgrep 'word("foo") or word("bar")' foobar

    to

    C:\> sgrep "word(\"foo\") or word(\"bar\")" foobar

    Alternatively, you can install bash from the Cygnus Cygwin project.

    The m4 binary comes from the Cygnus Cygwin project. See
    http://sourceware.cygnus.com/cygwin/ for details.
    Included binary release of m4 requires the cygwin.dll DLL-library.
    Both of them are distributed under GNU General Public License (GPL).
    See file COPYING for details.

    ---------------------------------------------------------------------------
    SGML-SCANNER
    ---------------------------------------------------------------------------

    Sgrep has a built-in scanner for XML, SGML and HTML-documents. This means
    that complex macros for querying SGML-files are no longer
    needed. However, sgrep still does not contain a full blown
    SGML-parser: the thing which it does contain could be described as an
    SGML-scanner. It does not recognize any syntax errors, it does
    not provide a parse tree and it does not provide any event stream.
    It just recognizes regions from SGML-files corresponding to different
    SGML tokens: start tags, end tags, attributes, etc.

    Since version 1.90a the SGML-scanner maintains an element stack. This
    is needed for the ability to support direct containment in queries
    to SGML/XML-files. Query language primitive 'elements' returns
    all elements of queried XML/SGML-documents. (see the 'childrening'
    and 'parenting' operators for examples).

    Since version 1.91a sgrep has support for 16-bit wide characters in
    query terms and support for UTF-8 and UTF-16 encodings in the
    SGML-scanner. See the "WIDE CHARACTER SUPPORT" below.

    SGML has many features which make it very difficult to parse. The
    SGML-scanner implemented in sgrep does not attempt to be a complete
    and error free SGML-parser; valid SGML-documents might confuse it.
    However, my goal is that all well formed XML-documents will be
    parsed correctly.

    The scanner has two modes:
    - SGML/HTML-mode
      o Names are case insensitive
      o PIs end with '>'
    - XML-mode
      o Names are case sensitive
      o PIs end with '?>'

    Sgrep will recognize empty XML elements (<ELEMENT/>) in both
    modes.

    The scanner does not automatically include entity references.
    However, it can automatically add external parsed entities
    defined in the internal document type definition subset to scanned files.
    Eg. if you have a line
    	<!ENTITY chapter1 SYSTEM "chapter1.sgml">
    in your document, the scanner can automatically include file
    "chapter1.sgml" to the list of scanned files, when the scanner sees
    this line in the internal document type definition subset.
    To use this feature you need to use "-g include-entities" option.

    ---------------------------------------------------------------------------
    WIDE CHARACTER SUPPORT
    ---------------------------------------------------------------------------

    Sgrep version 1.91a introduces 16-bit wide character support in index terms
    and in the SGML-parser.

    Since the sgrep query language is still strictly 8-bit, wide characters
    in queries need to be encoded. I chose to use encoding which looks
    just like character entity references in SGML: "\#<decimal number>;"
    for character number in decimal and "\#x<hex number>;" for character
    number in hexadecimal. Therefore the ISO-8859-1 letter a with two
    dots on top of it, 'ä' assuming you are reading this file with ISO-8859-1
    font '&auml;'-entity in HTML, '&#228;' as a decimal character reference
    and '&#e4;' as a hexadecimal character reference can be encoded in sgrep
    query either as "\#228;" or as "\#xe4;".

    So the finnish word "älämölö" ("&auml;l&auml;m&ouml;l&ouml;" in HTML) can
    be queried either with query like 'word("älämölö")', since sgrep query
    language supports 8-bit characters, or with encoded query like
    'word("\#228;l\#228;m\#248;l\#248")' or 'word("\#xe4;l\#xe4;m\#xf8;l\#xf8")'.

    The SGML-parser supports UTF-8 and UTF-16 encodings. You can select
    the encoding with the -g option:

    - "-g encoding=utf-8" selects UTF-8 encoding. This is the default if you
      are using the SGML-scanner in XML-mode (with -g xml option).
    - "-g encoding=utf-16" selects UTF-16 encoding. Note that currently (in
      version 1.91a), this is a synonym for "-g encoding=utf-8"
      since sgrep switches automatically to UTF-16 mode from UTF-8 mode when
      it sees the byte order mark (this also means, that you must have the byte
      order mark, if you are using UTF-16).
    - "-g encoding=iso-8859-1" selects iso-8859-1 encoding, which is also the
      default encoding when SGML-scanner is any other mode than XML.

    The SGML-scanner recognizes character entity references currently only
    in character data content. Character entity references in attribute values
    or entity literals are not recognized. No other entity references
    than character entity references are expanded, not even "&amp;", "&gt;"
    and "&lt;". I plan  to fix this before next release.

    The XML-scanner recognizes the encoding parameter in XML-declarations
    and can switch encoding accordingly (if not overridden with -g encoding
    option). Currently "us-ascii", "iso-8859-1", "utf-8" and "utf-16"
    encodings are recognized. Note that in XML-mode the SGML-parser
    interprets all characters classified as "Letter" in the XML-spesification
    as word characters by default.

    Currently only the SGML-scanner is aware of different encodings. The
    output module does not do any conversions: it just dumps the result
    regions from query files exatly as they were encoded there, even when
    different files use different encodings (this probably needs to be
    fixed).

    Here is an example using Murata Makotos example XML-documents in Japanese
    (see http://www.oasis-open.org/cover/xmlJapaneseExamples.html ).
    The unicode character 0x771f represents word Murata in japanese.

    % sgrep -o"%f:%l\n" -g xml 'word("\#x771f")' pr-xml-little-endian.xml pr-xml-utf-16.xml pr-xml-utf-8.xml weekly-utf-8.xml
    pr-xml-little-endian.xml:2
    pr-xml-utf-16.xml:2
    pr-xml-utf-8.xml:3

    ---------------------------------------------------------------------------
    NEW QUERY LANGUAGE FEATURES
    ---------------------------------------------------------------------------

    The example file "example.sgml" and its DTD "example.dtd" are
    included in this distribution.

    New query language features in version 1.91a and later:

    * near(distance)

    Finds regions of left hand side and right hand side having at most
    'distance' bytes bytes between them.
    'A near(0) B' would return regions of A and B which "touch" each other
    (in other words, there is no bytes between them. I know that using bytes
    is not the best way to measure distance in a text search engine, but the
    way Sgrep works makes this kind of query very fast. If you really need
    nearnes operator with words as a measure of distance, you could use
    'join(distance,word("*")) containing A containing B'. However this query
    would take much more time and memory to evaluate.)

    In this example, I use Jon Bosaks religious text as example material:

    % sgrep -x index -o"%r\n" 'word("jesus") near(20) word("peter")'
    Jesus was come into Peter
    Jesus taketh Peter
    Peter, and said unto Jesus
    Jesus taketh with him Peter
    Peter said unto Jesus
    Jesus unto Peter
    Peter followed Jesus
    Jesus loved saith unto Peter
    Jesus saith to Simon Peter
    Peter, an apostle of Jesus
    % sgrep -x index -o"%r\n" 'word("adam") near(30) word("eve") near(40) word("cain")'
    Adam knew Eve his wife; and she conceived, and bare Cain

    * near_before(distance)

    Works like just like 'near', except that 'near_after' requires the regions
    in the left hand side to occur before regions in the right hand side.

    % sgrep -x index -o"%r\n" 'word("peter") near_before(20) word("jesus")'
    Peter, and said unto Jesus
    Peter said unto Jesus
    Peter followed Jesus
    Peter, an apostle of Jesus

    New query language features in version 1.90a and later:

    * elements

    Returns all SGML-elements. This example counts all elements from input
    documents:

    % sgrep -c elements sgreptest.sgml
    14

    * parenting

    Works like old "containing" operator, except that parenting returns
    left hand side regions directly containing right hand side regions
    instead of all regions containing right hand side regions.
    NOTE: parenting works right only if the left hand side expression
    does not contain overlapping regions (which is guaranteed, if the
    left hand side regions correspond to SGML-elements).

    % sgrep 'elements parenting word("Peletier")' sgreptest.sgml
    <CITEREF RID="rf38">Peletier et al. (1994)</CITEREF>

    * childrening

    Works like old "in" operator, except that childrening returns left hand
    side regions directly contained in right hand side regions
    instead of all regions contained in right hand side regions.
    This example counts children elements of SGREPTEST-element

    % sgrep -c 'elements childrening (
    	stag("SGREPTEST") .. etag("SGREPTEST"))' sgreptest.sgml
    13

    * first(n, expression) and last(n ,expression)

    First-operator selects first n regions of the regions returned
    by expression and last-operator selects last n regions of
    the regions returned by expression.

    This query selects first child element of last child element of
    third ACT-element from a file containing word "Hamlet":
    (In other words, TITLE of last SCENE of third ACT of shakespeares
    famous PLAY "The Tragedy of Hamlet, Prince of Denmark")

    % cat test/childrening
    first(1,
      elements childrening
        last(1,
          elements childrening
            last(1,first(3,
              stag("ACT") .. etag("ACT") in (file("*") containing word("Hamlet"))
            )
          )
        )
    )
    % sgrep -x hamlet-index -f test/childrening
    <TITLE>SCENE IV.  The Queen's closet.</TITLE>
    %

    * first_bytes(n, expression) and last_bytes(n,expression)

    Operator first_bytes(n,expression) truncates all regions returned from
    expression to n-byte length starting from regions start point.
    Operator last_bytes(n,expression) truncates all regions returned from
    expression to n-byte length starting from regions end point.

    This example returns the start tags of SGREPTEST-elements children:

    % sgrep 'stag("*") containing first_bytes(1,elements childrening (stag("SGREPTEST") .. etag("SGREPTEST")))' sgreptest.sgml
    <Partno><Partno><Partno><Partno><partno><partno><partno><partno><partno><partno><partno><CITEREF RID="rf38"><AUTHOR>

    This query returns the end tags of SGREPTEST-elements children:

    % sgrep 'etag("*") containing last_bytes(1,elements childrening (stag("SGREPTEST") .. etag("SGREPTEST")))' sgreptest.sgml
    </partno></partno></partno></partno></partno></partno></partno></partno></PARTNO></PARTNO></PARTNO></CITEREF></AUTHOR>

    New query features in version 1.70a and later

    * file("filename")

    Returns the region containing the named file.

    * pi("PITarget")

    Returns the regions containing the processing instructions beginning with
    the given PI target.

    % sgrep 'pi("example_pi")' example.sgml
    <?example_pi processing instruction>

    * attribute("attribute name")

    Returns the regions containing the named attribute.

    % sgrep 'attribute("ATT1")' example.sgml
    att1="value1"

    * attvalue("attribute value")

    Returns the regions containing the given attribute value.

    % sgrep 'attvalue("value2")' example.sgml
    "value2"
    % sgrep 'attribute("*") containing attvalue("value2")' example.sgml
    att2="value2"

    * stag("GI")

    Returns the regions containing the start tags with the given GI.

    % sgrep 'stag("EXAMPLE")' example.sgml
    <EXAMPLE att1="value1" att2="value2">

    * etag("GI")

    Returns the regions containing the end tags with the given GI.

    % sgrep 'etag("EXAMPLE")' example.sgml
    </EXAMPLE>

    * word("word")

    Returns the regions containing the given word.
    N.B: A query word("foo") does not recognize occurrences of word "foo"
    inside comments. (See operators comment and comment_word  below.)

    % sgrep 'word("example")' example.sgml
    example
    % sgrep '"\n"_."\n" containing word("example")' example.sgml
    This is an example SGML file to demonstrate new features in sgrep-2.0.

    * comments

    Returns the regions containing all SGML comments.

    % sgrep 'comments' example.sgml
    <!-- comment --><!-- another comment -->

    * comment_word("comment word")

    Returns the region containing the given word inside comments.

    % sgrep 'comment_word("another")' example.sgml
    another
    % sgrep 'comments containing comment_word("another")' example.sgml
    <!-- another comment -->

    * cdata

    Returns regions containing CDATA marked sections. Sgrep recognizes
    words also inside CDATA marked sections.

    % sgrep 'cdata' example.sgml
    <![CDATA[ <CDATA> <marked> &section ]]><![CDATA[ another marked section ]]>
    % sgrep 'cdata containing word("another")' example.sgml
    <![CDATA[ another marked section ]]>

    * entity("entity name")

    Returns the regions containing references to the given entity.
    (Entity references are currently recognized only in PCDATA.)

    % sgrep 'entity("entity1")' example.sgml
    &entity1;
    % sgrep 'stag("ELEM1") .. etag("ELEM1") containing entity("entity1")'
    example.sgml
    <ELEM1>&entity1;</ELEM1>

    * doctype("doctype name")

    Returns the regions containing given document type name inside the document
    type declaration.

    % sgrep 'doctype("*")' example.sgml
    EXAMPLE

    * doctype_pid("publicid")

    Returns the regions containing the given document type public id inside
    document type declarations.

    % sgrep 'doctype_pid("*")' example.sgml
    -//SID//DTD sgrep example//EN

    * doctype_sid("systemid")

    Returns the regions containing the given document type system id inside
    document type declarations.

    % sgrep 'doctype_sid("ex*")' example.sgml
    example.dtd

    * entity_declaration("entity name")

    Returns the regions containing the declaration of the given entity name.

    % sgrep 'entity_declaration("entity1")' example.sgml
    <!ENTITY entity1 "literal value">

    * entity_literal("entity name")

    Returns the regions containing the literal value of the given entity name.

    % sgrep 'entity_literal("entity1")' example.sgml
    literal value

    * entity_pid("entity public id")

    Returns the regions containing the given public id of an entity within its
    declaration.

    % sgrep 'entity_pid("*")' example.sgml
    -//SID//NONSGML entity example//EN

    * entity_sid("entity system id")

    Returns the regions containing the given system id inside an entity
    declaration.

    % sgrep 'entity_sid("*")' example.sgml
    figure.file

    * entity_ndata("notation name")

    Returns the regions containing the given notation name inside entity
    declarations.

    % sgrep 'entity_ndata("*")' example.sgml
    anotation
    % sgrep 'entity_declaration("*") containing entity_ndata("ANOTATION")'
    example.sgml
    <!ENTITY figure PUBLIC "-//SID//NONSGML entity example//EN"
                    "figure.file" NDATA anotation>

    * prologs

    Returns the regions containing document prologs.

    % sgrep 'pi("*") in prologs' example.sgml
    <?pi inside prolog>

    --------------------------------------------------------------------
     INDEXING
    --------------------------------------------------------------------

    Sgrep supports indexing of both structure and content of SGML, HTML
    and XML documents. Indexing is implemented by creating a separate index
    file, which contains a list of terms and of regions corresponding to these
    terms. This means that if you want to output the actual content of
    result regions you have to keep the original files around.

    If the query uses only the new SGML query features, the same query should
    return same results independent of whether an index was used or not.

    Indexes are stored in compressed binary files. Depending on the indexed
    material the index file size is 30-60% of the original files. This is
    not so bad, since in theory you can produce the full content of
    the original files from the index file, except for whitespace and
    punctuation marks.

    Maximum size of the indexed data is currently 2 gigabytes. If you
    want to index larger collections, you have to split the index
    to multiple index files. However, for optimal performance the
    index file size should be smaller than your available RAM-memory.

    Sgrep is switched to indexing mode by giving "-I" as first option in
    the command line. Command 'sgrep -I -h' will give you the summary of
    available indexing options:

    <CLIP>
    Usage: (sgindex | sgrep -I) <options> <files...>
    Use 'sgrep -h' for help on query mode options.

    Indexing mode options are:
      -C              display copyright notice
      -h              help (means this text)
      -i              fold all words to lower case when indexing
      -T              show statistics about created index files
      -V              display version information
      -v              verbose mode. Shows what is going on
      -c <index file> create new index file
      -F <file>       read list of input files from <file> instead of command line
      -g <option>     set scanner option:
          sgml        use SGML scanner
          html        use HTML scanner (currently same as sgml scanner)
          xml         use XML scanner
          sgml-debug  show recognized SGML tokens
          include-entities  automatically include system entities
      -l <limit>      make a list of possible stopwords
      -L <stop file>  write possible stopwords to file
      -S <stop file>  read stop word list from file
      -m <megabytes>  main memory available for indexing in megabytes
      -w <char list>  set the list of characters used to recognize words
            --              no more options

    Copyright (C) 1998 University of Helsinki. Use sgindex -C for details,
    </CLIP>

    Options -C, -h, and -V should be self explanatory.

    Indexes are created using the -c option and giving a file list to index.
    The file list can be directly in the command line, or with -F option
    (see below).

    <CLIP>
    % sgrep -I -c demo.index demo.sgml
    % ls -l demo.*
    -rw-------   1 jjaakkol grpd        53577 Aug 24 12:52 demo.index
    -rw-------   1 jjaakkol grpd        91536 Mar  6 13:40 demo.sgml
    </CLIP>

    First 13 lines of the index file contain statistics about the created
    index.

    <CLIP>
    % head -13 demo.index
    sgrep-index v0

    2441 terms
    11554 entries
    1024 bytes header (1%)
    9764 bytes term index (18%)
    17222 bytes strings (32%)
      26366 total strings
      9144 compressed with lcps (-34%)
    25545 bytes postings (47%)
    22 bytes file list (0%)
    53577 total index size
    --
    </CLIP>

    With -F <file> option you can give a list of the files to be indexed
    in the named file, instead of giving it on the command line.
    The file names given in the file have to separated by a newline.

    The -v option gives verbose progress reports while indexing:

    <CLIP>
    % sgrep -I -v -c demo.index demo.sgml
    Indexing 1/1 files 64/89K (71%)
    Writing index file of 52K
    Writing index 2048/2441 entries (83%)
    </CLIP>

    The entries added to the index are case sensitive by default.
    For example, word("Foo") is different from word("foo").
    With option -i you can instruct sgrep to
    fold all words (only words in content or comments, not any structural
    elements like element type names or attribute values) to lowercase.

    With option -T you can get some statistics (some useful for debugging,
    some less useful, and some mighty cryptic) about the created index.

    With option -L <term file> you can create a file containing list of
    all terms added to index. Each line in created file will contain the
    amount of bytes required by the term and the term itself.

    This example is using the XML WWW-page from http://www.sil.org/sgml/xml.html
    <CLIP>
    % sgrep -I -i -L terms -c xml.index xml.html
    % cat terms | sort -n | tail
    2043 eLI
    2094 wa
    2263 wto
    2543 wof
    2713 wand
    3414 eA
    3433 wxml
    4410 wthe
    5076 aHREF
    5390 sA
    </CLIP>

    With option -S <stop word list file> you can give indexer a stop word
    list to reduce the size of index. Stop word list consists of one
    stop word per line, with possibly including the amount of bytes
    in the term (so you can actually use a part of file originally created with
    the -L option to indexer).

    Here is an example using a simple English stop word list (containing words like
    "to", "and", "the" and so on) to reduce the size of the index:

    <CLIP>
    % sgrep -I -i -L terms -c xml.index xml.html
    % ls -l xml.index
    -rw-------   1 jjaakkol grpd       291599 Aug 24 14:13 xml.index
    % sgrep -I -i -S stoplist -c xml.index xml.html
    % ls -l xml.index
    -rw-------   1 jjaakkol grpd       259058 Aug 24 14:14 xml.index
    </CLIP>

    With using "-l <number>" option you can obtain information about the impact
    of stop word list on index size when all terms taking more space
    than a fraction of 1/number of the index would be considered as stop words.

    <CLIP>
    % sgrep -I -l 200 -i -c xml.index xml.html
    Possible stop words:
        4K (1.74%) 'aHREF'
        3K (1.17%) 'eA'
        1K (0.70%) 'eLI'
        5K (1.85%) 'sA'
        1K (0.70%) 'sLI'
        2K (0.72%) 'wa'
        2K (0.93%) 'wand'
        1K (0.62%) 'wfor'
        1K (0.67%) 'win'
        2K (0.87%) 'wof'
        4K (1.51%) 'wthe'
        2K (0.78%) 'wto'
        3K (1.18%) 'wxml'
    -------------
       38K (13.43%) total
    </CLIP>

    Using the "-g sgml" option you can select SGML mode scanner.

    Using the "-g xml" option you can select XML mode scanner.

    Using the "-g include-entities" option you can automatically include
    defined system entities to indexed files (see SGML scanner above).

    Using the "-g sgml-debug" option you can check the tokens which sgrep
    recognized from its input files:

    <CLIP>
    % sgrep -I -c demo.index -g sgml-debug sgreptest.sgml
    doctype("SGREPTEST"):dnSGREPTEST:(10,18)
    doctype_pid("-//SID//DTD Just something to test sgrep//EN"):dp-//SID//DTD Just
    something to test sgrep//EN:(29,72)
    doctype_sid("empty"):dsempty:(77,81)
    comment_word("Comment"):cComment:(93,99)
    comment(""):-:(88,110)
    pi("PI"):?PI:(111,115)
    ....
    word("third"):wthird:(1391,1395)
    etag("PARTNO"):ePARTNO:(1396,1404)
    etag("SGREPTEST"):eSGREPTEST:(1406,1417)
    </CLIP>

    With the "-m <megabytes>'" option you can adjust the amount of main memory
    indexer will use for postings spool before writing a temporary
    file. Default value for -m option is 20 megabytes. Creating the index
    completely in main is faster than using temporary files. However, if you
    give a value larger than available main memory, sgrep will probably start
    trashing and indexing will be slow.

    With "-w <char list>" option you can select which characters make up
    words. The default is "-w a-zA-Z". Here in Finland I would use
    "-w A-Za-ZÅÄÖåäö" (assuming you have iso-8859-1 font). If you want to index
    also numbers you could use "-w A-Za-z0-9"

    ---------------------------------------------------------------------------
    QUERIES
    ---------------------------------------------------------------------------

    Sgrep has some new options when used in the query mode.

    Options "-F <file>", "-w <word characters>" and "-g <scanner option>"
    have the same functions as in the indexing mode.

    Option "-x <index file>" is used to specify an index while when one
    is used. If -x option is used and no file list is specified either
    in the command line or with -F option, sgrep obtains the list of
    queried files straight from the index.

    ---------------------------------------------------------------------------
    EXAMPLE
    ---------------------------------------------------------------------------

    Here the input file xml.html is taken from Robin Covers excellent
    WWW-page at http://www.sil.org/sgml/xml.html

    Example: Find all P elements containing word "newsfax":
    <CLIP>
    % time sgrep -x xml.index 'stag("P") .. etag("P") containing word("newsfax")'
    <p>[August 19, 1998]  <a href="http://www.zenweb.com/robert/tools/">Tools and
    Utilities</a> from Robert Hanson: XML::Parser, LOTE NewsFax to XML Parsers,
    LOTE XML to Kingdom Summaries, XML Script Server Parser.</p>
    0.03user 0.03system 0:00.25elapsed 24%CPU (0avgtext+0avgdata 0maxresident)k
    </CLIP>

    The same example without using index:

    <CLIP>
    % time sgrep -i 'stag("P") .. etag("P") containing word("newsfax")' xml.html
    <p>[August 19, 1998]  <a href="http://www.zenweb.com/robert/tools/">Tools and
    Utilities</a> from Robert Hanson: XML::Parser, LOTE NewsFax to XML Parsers,
    LOTE XML to Kingdom Summaries, XML Script Server Parser.</p>
    0.82user 0.05system 0:01.18elapsed 73%CPU (0avgtext+0avgdata 0maxresident)k
    </CLIP>

    ---------------------------------------------------------------------------
    ANOTHER EXAMPLE
    ---------------------------------------------------------------------------

    This example uses Jon Bosak's XML-example material: religious texts
    and Shakespeare's works. Since this query is slightly more complex,
    it has been put together from smaller parts by using m4. File "filelist"
    contains the list of all XML-example files from Bosaks collection.

    Here is the file "query"
    <CLIP>
    # Finds elements having given name
    define(ELEMENT, (stag($1) .. etag($1)))

    # Finds LINE elements
    define(E_LINE, (ELEMENT("LINE")))

    # Finds SPEECH elements
    define(E_SPEECH, (ELEMENT("SPEECH")))

    # Finds SPEECH elements where HAMLET is speaking
    define(HAMLET_SPEAKING, (E_SPEECH containing (
                    ELEMENT("SPEAKER") containing word("HAMLET"))))

    # Finds LINE elements containing words to, be, not and question
    define(TOBENOTQUESTION, (E_LINE containing word("to") containing word("be")
       containing word("not") containing word("question")))

    # Finds the LINE where HAMLET says the famous words
    define(HAMLET_SAYS, (TOBENOTQUESTION in HAMLET_SPEAKING))
    </CLIP>

    Evaluate the query using plain search:

    <CLIP>
    % time sgrep -o "%f:\n %r\n" -f query -e HAMLET_SAYS -F filelist
    /xml/shakespeare.1.10.xml/hamlet.xml:
     <LINE>To be, or not to be: that is the question:</LINE>
    16.60user 0.78system 0:18.29elapsed 94%CPU (0avgtext+0avgdata 0maxresident)k
    0inputs+0outputs (327major+455minor)pagefaults 0swaps
    </CLIP>

    Create an index of the input texts:

    <CLIP>
    % time sgrep -I -c index -v -F filelist
    Indexing 43/43 files 14957/14958K (99%)
    Writing index file of 5472K
    Writing index 35840/36691 entries (97%)
    23.65user 4.56system 0:32.77elapsed 86%CPU (0avgtext+0avgdata 0maxresident)k
    0inputs+0outputs (94major+5928minor)pagefaults 0swaps
    </CLIP>

    Evaluate the query using index:

    <CLIP>
    % time sgrep -x index -o "%f:\n %r\n" -f query -e HAMLET_SAYS -F filelist
    /xml/shakespeare.1.10.xml/hamlet.xml:
     <LINE>To be, or not to be: that is the question:</LINE>
    1.24user 0.13system 0:01.43elapsed 95%CPU (0avgtext+0avgdata 0maxresident)k
    0inputs+0outputs (536major+728minor)pagefaults 0swaps
    </CLIP>

    ---------------------------------------------------------------------------
    THAT'S IT! Enjoy!
    ---------------------------------------------------------------------------

    Please send comments about sgrep-2.0 to
    Jani Jaakkola (jjaakkol@cs.helsinki.fi).