
                           1996 03 18
                              KNEWS

INTRODUCTION
------------
KNews is a radically new approach to storing news on a machine, that
is several orders of magnitude less resource intensive than traditional
approaches to storing news (such as Cnews, etc).  KNews is even suitable
for "underpowered" machines, like a 386/486 class processor, whereas
traditional news systems require the latest and fastest CPUs and disks.

This beta release focuses on the following:

	o	introducing KNews to the QNX community,

	o	gauging interest in KNews

	o	examining the performance increase with KNews over
		traditional news systems

	o	providing a free/low_cost version of the ultimate
		KNews product, which will be a commercial product
		available late '96.

KNews consists of two major components, a newsfeeder program ("knews")
and a virtual file system manager ("VFsys.knews").  These two programs
cooperate with each other to implement KNews.

WHAT IS KNEWS?
--------------
For a detailed answer, see the article in the May 1996 Doctor Dobb's
Journal.  Here's the short answer:

In traditional news systems (like those based on CNews, etc), when
articles are stored to disk (from the UUCP or NNTP site that is
providing them), there are a number of inefficiencies that occur:

	o	fragmentation -- the disk head is moving all over the disk
		in an attempt to put a large number of tiny files in their
		respective newsgroups,

	o	space wastage -- if you are on a filesystem that has 512 byte
		blocks, then statistically 256 bytes are wasted.  On a typical
		news system, with 200k articles, this wasted space is a
		whopping 51 MEGABYTES.

		Another form of space wastage that occurs is the directory
		entries themselves on disk.  There are a few issues here;
		the directory structure consumes about 48 bytes per file entry,
		the directory structure has a minimum size of 2k, and the
		directory structure never shrinks.

	o	expiry takes many hours -- if you are an experienced sys admin
		for a UNIX-style news system, you know that one of the main
		problems is the fact that expiration takes forever.  On some
		machines, the news just doesn't expire fast enough, and the
		disk overflows, resulting in much grief.

Under KNews, as the articles come in, they are placed into a
few large files, based upon when the articles are going to expire
(basically, that's what most news *does* anyway!).  Then, we map the
individual article files (like /usr/spool/news/comp/os/qnx/1143)
into a portion of the expiry-ordered file.

How does this solve all of the problems?  When news is
coming in, we're only writing the data to one of a few files,
depending upon when the article is to expire.  There are no great
amounts of head movement, as the file system is able to efficiently
cache the writes.

Another advantage of this organization is that we aren't wasting
any additional storage space with the "statistical holes".  One
article begins right where the previous article leaves off, with
no intervening wasted space.  On a 500 MB hard disk, with 200k
articles, this is an immediate 10% space savings.

To expire news, we just "rm" a single file, and unlink our RAM-based
pointers which pointed at the file.  This is typically done in under
1 second on a "slow" 486/33.

WHAT DOES IT DO?
----------------
This initial beta release has the following limitations, some due to
design / time, and others intentional, so as not to compete with the
commercial product:

	o	operates as an end-node (ie:  doesn't provide the ability
		to construct files for other sites)

	o	all expiry is fixed.  This means that all news expires
		some number of days (configurable) after it was
		received, regardless of the date of posting, or the
		EXPIRY line in the article.

	o	only operates with an NNTP host.  Ie:  your service provider
		must allow you access to their host in order to read
		news (this is common practice).

	o	source is provided, support isn't.

Some of the features that KNews allows:

	o	allows the selection of newsgroups to receive

	o	configurable expiry granularity.  If you like, you can expire
		once every minute (!), although a reasonable limit is something
		like once per hour, or every 6 hours, etc.

	o	presents /usr/spool/news to applications, and maintains an
		active file.  This has been found to work just fine with
		newsreaders like "tin" for example.

	o	the program is constantly updated.

To get a first-hand feel for the speed of KNews, before you install it,
go to your existing /usr/spool/news directory and do a "find .", perhaps
timing it.  Then, when you have a full news feed of KNews news, repeat
the experiment.

LICENSE
-------
The whole idea of this program is threefold:

	o	obtain development money

	o	illustrate how to write a virtual filesystem under QNX 4

	o	give a free sample taste of what the commercial product
		will be like, and hence promote the commercial product

Therefore, you can do anything with it that doesn't violate the
above intentions.  Ie:  I wouldn't be too pleased if you copied
it wholesale, and claimed it as your own work.  Or, if you started
up an internet site, used the software heavily, and made lots of
money without at least sending the recommended contribution.

REGISTRATION
------------
This product is several hundered hours of work, from conception to
implementation.  A financial contribution is welcome; this will
help to create other useful, low-cost software products.

The recommended individual contribution is $20US/$25CDN.  Large sites
are encouraged to obtain a site license, contact the author for
details.

Benefits of registration include notification of updates, and a
10% discount on the commercial version, when available.

COMMERCIAL VERSION
------------------
As far as the commercial product goes, the current plans are to
support the following feature list (subject to change without
notice):

	o	allow functionality to feed other nodes

	o	configurable expiry, based upon newsgroup

	o	compression (25% on UUENCODED files, ~50% on text)

	o	writable file system
		(both for news articles, and other non-article files)

The commercial software will not be distributed in source form,
and will come with a limited amount of support.  Software
vendors interested in licensing the commercial offering are
invited to contact the author.

RELATED PRODUCTS
----------------
The entire concept of virtual filesystems has led to the
concept of a virtual filesystem toolkit.  This toolkit would
contain the source for a number of virtual filesystems, including:

	o	VFsys.tar
		This file system allows you to "cd" into a ".tar" file
		and perform read/lseek operations, instead of unpacking
		the tar file, doing the operations, and repacking it.

	o	VFsys.map
		With the use of a ASCII text configuration file, this
		virtual filesystem allows you to map any virtual filename
		onto either any subset of a physical file (ie:  basically
		what VFsys.knews does), or any number of physical files.
		The latter is useful for log buffers -- you would map
		one large virtual file to be the logical concatenation
		of log files from each day of the week.  As the log files
		expired, you would just change the mapping, rather than
		copying the rest of the large file to a different log
		file.

	o	VFsys.compress
		Allows compressed files to be treated as if they were
		uncompressed for read/lseek operations.  This way, instead
		of having to run a special command to uncompress a file,
		look at it, and then recompress it, the read operation
		could be done directly through the virtual filesystem.

The toolkit would provide full, well-documented source, and also
include a tutorial on how to write a virtual filesystem under QNX 4.
		
WHERE CAN I GET A COPY?
-----------------------
Probably the easiest place to get a copy is on Quics, QNX Software
System's "bulletin board".  As of March 18th, 1996, it will be
placed in /usr/free/news/knews, and you can download either the
entire archive ("knews-1.tar.gz"), or you can cd into "knews-1"
and download individual files.

WHO WROTE IT?
-------------
KNews is written by Robert Krten.

I can be reached at:

	Robert Krten
	278 Equestrian Drive,
	Kanata, ON
	K2M 1C5
	CANADA

	rk@parse.com			(preferred)
	rkrten@quics.qnx.com
	http://www.parse.com

Enjoy!
