SREhttp/2 Manual || View your log files

## Using SREhttp/2 Common-Log Audit Files

### I. Introduction

In keeping with http server standards, SREhttp/2 will maintain common-log format audit files.

Besides offering a compact synopsis, a number of tools exist to help you analyze common-log files. Furthermore, as an adjunct to the common-log audit file, SREhttp/2 can also record the browser used by each request (as well as the referer) in seperate log files.

This document discusses how to modify the LOGS.CFG configuration file. LOGS.CFG (which should be in the SREHTTP2\CFG directory) contains the configuration options for these three log files. These options include the host-specific filenames for these log files, audit-suppression instructions, and a few display options.

LOGS.CFG is a general configuration file -- its values are used for all hosts (that is, there is no differentiation between hosts that are defined as superceding or non-superceding). In addition, it can not be changed on-the-fly; you have to restart SRE2003 for its changes to take effect.

### II) Basic configuration

Configuration is accomplished by setting variables in LOGS.CFG. The following lists the variables, with details below:

ADD_HOSTNICKNAMES | add the Host-nickname to common-log entries
APACHE_MODE | Select format for Browser and Referer logs
BROWSERLOG BROWSERLOG.hostn | Default browser log file Browser log file for the "hostn" host.
COMMONLOG COMMONLOG.hostn | Default common-log file Common-log file for the "hostn" host (hostn is a host-nickname)>
COMBINEDLOG COMBINEDLOG.hostn | Default combined (common/referer/browser) log file Cominbedlog file for the "hostn" host.
FORCE_REFERER | Use a " - " when no referer request header is available
LOGFILE_MAXSIZE | rename log files once they get to this size
NOOPTIONS | Do not record the portion of the request selector following the ?
NOHTTP | Do not record the HTTP/1.0 portion of the request string
NOCODES | List of "return codes" and "http methods" to not record.
NOUSER | List of IP addresses, and SREhttp/2 "usernames", to not record
NOREFERER | List of IP addresses to not record in REFERER.LOG (COMMON, COMBINED and BROWSER logs do NOT pay attention to this parameter).
NOURL | Rquest selectors to not record
DISPLAY | Display log entries in PMPRINTF window or SRE2003 status screen
REFERERLOG REFERERLOG.hostn | Default referer log file Referer log file for the "hostn" host.

### III) Configuration Details:

#### COMMONGLOG, COMBINEDLOG, BROWSERLOG and REFERERLOG

The COMMONLOG, COMBINEDLOG, BROWSERLOG, and REFERERLOG "stem" variables all have the same structure. The basic notion is that SREhttp/2 will take the host nickname (of the host to whom the request was directed), and attempt to find matching COMMONLOG (and COMBINEDLOG, BROWSERLOG and REFERERLOG) variables. If a match is not found, or if the request is to the default host, the "default" versions of these four files are used.

You should set these stem variable to equal filenames. If you use a relative file name (i.e.; FOO.BAR or DIR1\FOO.BAR), the file is assumed to be relative to the SREHTTP2\LOG directory.

If you do not want to record information to a common-log file, then set `COMMONLOG=0`. Or, `COMMONLOG.HOSTN=0` to not record common-log information for requests to the `HOSTN` host.

Similarly, the COMBINEDLOG, BROWSERLOG and REFERERLOG variables are treated the same way. For example, to suppress the REFERERLOG, set `REFERERLOG=0`.

Example |
     COMMONLOG=COMMON.LOG
     COMMONLOG.ZOO=D:\CIRCUS\WWW\COMMON.LOG
For requests to the `ZOO` host, record common-log information to `D:\CIRCUS\WWW\COMMON.LOG`. For request to all other hosts (including the default host), record common-log information to `D:\SRE\SREHTTP2\LOG\COMMON.LOG` (assuming that SREhttp/2 was installed to D:\SRE\SREHTTP2).
Note: | if you use `COMMONGLOG=0` syntax to suppress writing to a common-log file (or other log file), the `LOG` permission is ignored.

#### The NOxxx,and other options

The followings options allow you to modify the output to the common-log file, and to suppress the recording of selected requests.

ADD_HOSTNICKNAMES | Add a host-nickname to the `request` portion of a log entry. Note that this is not standard practice. Also note that if you've specified unique log files for each host (and for each log file), then adding a host-nickname is not particularly useful.
Syntax: | `ADD_HOSTNICKNAMES=0` : do not add the host-nickname `ADD_HOSTNICKNAMES=1` : always add, even for DEFAULT host `ADD_HOSTNICKNAMES=2` : add, but not if default host
APACHE_MODE | If `APACHE_MODE=1`, then the referer and browser log's are written using an "Apache" mode (that is, something similar to what Apache server's write). If `APACHE_MODE=0`, then a "GoHTTP" mode is used. For example:
          APACHE_MODE (browser and referer logs):
    [14/Mar/1996:17:45:08] Mozilla/2.0 (X11; I; HP-UX A.09.05 9000/735)
    [14/Mar/1996:17:48:10] http://guide-p.infoseek.com/Titles -> /~sret1/analog/
           GoHTTP:
    [03/Jun/1997:19:35:16 -0500] DAL40-29.PPP.IADFW.NET Mozilla/2.02E (OS/2; I)
    [2/Mar/1997:18:23:37 -0500] "/imgs/srefiltr.gif" http://ship.float.net/
FORCE_REFERER | If `FORCE_REFERER=1`, then in cases where there is no referer, a `" - "` is used.
LOGFILE_MAXSIZE | LOGFILE_MAXSIZE is used to limit the size of the LOG files. Once a log file exceeds this size (as specified in Kbytes), then it will be renamed (using .001, .002, etc.). The renamed files will be moved to the SREHTTP2\LOG\OLD directory.
Syntax: | `LOGFILE_MAXIZE=nnn` where `nnn` is in kilobytes.
Examples: | `LOGFILE_MAXIZE=1000` -- maximum size of 1 Megabyte `LOGFILE_MAXIZE=0` -- no maximum size
NOOPTIONS | If `NOOPTIONS=1`, then the portion of the request selector following a `?` will not be included. Since this can be quite long (as when you are using the GET method in an HTML FORM), inclusion of this information can lead to very long (and hard to read) entries. For example: given a request for: `/LOOK1?arg1=sub1/&arg2=1&argx=1&xmac=0` if `NOOPTIONS=1`, the "request" portion of the common-log (and referers log) will be `/LOOK1`
NOHTTP | Technically, a request string should end with the HTTP version. Since this is not necessarily interesting, and since it's not clear that the common-log format requires this information, you can suppress this portion of the request string by setting `NOHTTP=1`.
NOCODES | NOCODES should contain a space delimited list of 3 digit return status codes, and http-methods, that you do not want to record. If the response code, or the request method, matches one of these NOCODES, then none of the log files are written to. Examples: you may not be interested in requests that generated a "authorization response" (a 401 response). If so, set: `NOCODES=401` If you are also uninterested in "moved" responses, you can set: `NOCODES=401 301 302` If you are not interested in "authorization responses" and responses to HEAD method requests: `NOCODES=401 HEAD` And if you are sort of perverse, and are not interested in normal responses, you can set: `NOCODES=200`
- NOUSER | NOUSER entries should contain a space delimited list of numeric IP addresses, or domain names. Requests from these clients will not be to logged. You can also contain "usernames" (from SREhttp/2's USERS.CFG username/password database) -- requests from these usernames will not be recorded. One sensible use of this is to include the webmaster's IP address (since you probably don't care about requests you make to yourself), as well as the "usernames" of the various SUPERUSERs. Example: `NOUSER=joe.bars.net *.wah.com 151.121.55.6 ADMINX` Note that `ADMINX` is a possible username (possibly of a user with SUPERUSER privileges). Notes: | you can use * as a wildcard for a field. Thus, `ab.wah.com` and `cd.wah.com` would both match `*.wah.com`, but `www.ef.wah.com` would not If a match to any of these NOCODES occurs, then none of the log files are written to. You can have multiple NOUSER entries (they will all be used). Thus, an alternative to the above example would be:
         NOUSER=joe.bars.net
         NOUSER=*.wah.com
         NOUSER=151.121.55.6
         NOUSER=ADMINX
NOUSER entries are not host-specific. In particular, `username` suppression is not host-specific.
- NOREFERER | This should contain a space delimited list of IP addresses and domain names that you do NOT wish to record in the REFERER.LOG file. That is, requests that were refered from these addresses will NOT be recorded in REFERER.LOG. However, these requests WILL be recorded in the common.log, combined.log, and browser.log files. Example: `NOREFERER= *.my.site.net backup.site.org` Notes: | you can use * as a wildcard for a field (see the above example). You can have multiple NOREFERER entries (they will all be used). Thus, an alternative to the above example would be:
         NOREFERER=*.my.site.net
         NOREFEER= backup.site.org
NOREFERER entries are not host-specific. NOREFERER only looks at the domain name portion of a request-URI. It does not look at the selector portion (the stuff after the `http://foo.bar.net/`)
NOURL | The NOURL entries (there can be more than one) should contain URLs to not record. That is, if a request selector matches one of these URLs, it will NOT be recorded. For example:
            NOURL =  IMGS/*
            NOURL =  BORING/TEDIOUS.HTM
The preceding examples will be used for all requests (to any host). Alternatively, you can include a host-nickname:
            NOURL =  DEFAULT:/IMGS/*
            NOURL =  ZOO:/BORING/TEDIOUS.HTM
Notes | You can include * as a wildcard. Thus, IMGS/HORSE.GIF matches IMGS/* If a request selector matches one of these NOURL variables, then none of the log files are written to.
DISPLAY | Use `DISPLAY` to also display the common-log entry on screen. `DISPLAY=1` : display common-log entries in the PMPRINTF window (you can use PMPRINTF.EXE to display the PMPRINTF window). `DISPLAY=0` : do not display common-log entries on-screen `DISPLAY=-1` : display common-log entries in the message area of the SRE2003 status screen. If you choose this, it is strongly recommended that you increase the OS/2 box size using the OS/2 MODE command. For example, to resize the OS/2 box to 95 columns x 35 rows, execute the following before running SRE2003. `d:\sre>MODE CO95,35`

### IV) Notes

- The following parameters, etc. are used to determine if a request is logged:
  - If a NO_LOG permission is specified, the request is not logged
  - if the SREhttp/2 LOG_SUPPRESS variable equals 1, and a LOG permission is not specified, then the request is not logged
  - If COMMONLOG=0 (or COMMONLOG.host-nickname=0), the request is not logged to the COMMON log file. SImilar rules hold for the other LOG files. Note that it is possible (perhaps even typical) to suppress logging to a subset of the LOG files; say, to suppress the combined-log and referer-log files.

- If you are running a single-host server, and you are happy to log all requests in a standard fashion, then you can use the default version of LOGS.CFG. Similarly, if you are running multiple-hosts, and are happy to log all requests to a single file, use the default version of LOGS.CFG.

  If you choose to use the default LOGS.CFG file, log information will be written to the following log files in the SREHTTP2\LOG directory.

- COMMON.LOG -- the common-log audit file
- BROWSERS.LOG -- the browser-log file
- REFERER.LOG -- the referer-log file

- SREhttp/2 will "lazy write" the audit files (about once a minute).

- In addition to the common, etc. log files, SREhttp/2 has several other means of auditing requests:
- The REQUESTS.CNT file is a running count of the total number of requests for each URI.
- You can write audit information to the SRE2003 log file
- If you've set `DISPLAY=1`, you can use `PMPRINTF.EXE` to display and to save log information.
- You can write your own Post-Response procedure

### V) Structure of .LOG files

(the following description is adapted from http://www.apache.org/docs/mod/mod_log_common.html)

    i) The common-log file format

    The common-log file contains a separate line for each request.
    A line is composed of several items (tokens) separated by spaces:
         host ident authuser date request status bytes
    If a token does not have a value then it is represented by a hyphen (-).

    The meanings and values of these tokens are as follows:

    client
      The fully-qualified domain name of the client, or its IP number if the
      name is not available.
    ident
       The identd identity information: NOT SUPPORTED BY SREhttp/2.
    authuser
       The client's username, as provided in the AUTHORIZATION request header.
       Note that the common-log standard suggests that "if the request was for a
       password protected document, then this is the username used in the request".
       However, SREhttp/2 will always include the authuser (if available), even
       if the particular request did not require authorization.
    date
       The date and time of the request, in the following format:
          date = [day/month/year:hour:minute:second zone]
       where:
             day = 2 digit
             month = 3 letter
             year = 4 digit
             hour = 2 digit
             minute = 2 digit
             second = 2 digit
             zone = (`+' | `-') 4 digit
    request
         The request line from the client, enclosed in double quotes
    status
       The three digit status code returned to the client.
    bytes
       The number of bytes in the object returned to the client, not including
       any headers.

    For example:
     acl.wear.org - - [18/Mar/1996:16:24:08 -0500] "GET /SMP/HLO.HTM HTTP/1.0" 200 1251

    In the above example, the HTTP/1.0 substring may not be standard (you can
    suppress it by setting the NOHTTP option in SREFLOGS.INI). Also note that
    -0500 corresponds to the EST offset from GMT.

    ii) The BROWSERS log file

    The browsers log file has the following structure:
       [date] client : user-agent
    For example:
      [24/Feb/1997:18:14:30 -0500] 151.121.65.143 : IBM-WebExplorer-DLL/v1.1b

    iii) The REFERERS log file

    The referers log file has the following structure:
       [date]  "sel"  referer_url
    Where SEL is from the request string, and the referer_url is from the
    REFERER request header.
    For example:
       [24/Feb/1997:18:18:19 -0500] "/SAMPLES/samp1.htm" http://dh.ag.gov/i.htm
    (http://dh.ag.gov/i.htm contained a link to /samples/samp1.htm)

    Note: if a request does not have a REFERER request header (for example, WebEx
    does not provide a REFERER request header):
      if FORCE_REFERER=0, an entry will NOT be added to the   REFERERS log.
         FORCE_REFERER=1, a referer of " - " is used.

    iv)The COMBINEDLOG log file

    The  combined log file combines the common, referer, and browser logs.
    It has the following structure:
      PARROT.HALL.NET - - [04/Jun/1997:22:38:16 -0400] "GET /imgs/justscal.gif"
      200 516 "http://mysite.heights.edu/SAMPLES/guide.htm"
      "Mozilla/2.02E (OS/2; I)"
    (but all on one long line).

     Last updated 26 June 2002