.\" $Revision: 1.34 $
.TH NEWSFEEDS 5
.SH NAME
newsfeeds \- determine where Usenet articles get sent
.SH DESCRIPTION
The file
.\" =()<.I @<_PATH_NEWSFEEDS>@>()=
.I /usr/local/news/etc/newsfeeds
specifies how incoming articles should be distributed to other sites.
It is parsed by the InterNetNews server
.IR innd (8)
when it starts up, or when directed to by
.IR ctlinnd (8).
.PP
The file is interpreted as a set of lines according to the following rules.
If a line ends with a backslash, then the backslash, the newline, and any
whitespace at the start of the next line is deleted.
This is repeated until the entire ``logical'' line is collected.
If the logical line is blank, or starts with a number sign (``#''), it
is ignored.
.PP
All other lines are interpreted as feed entries.
An entry should consist of four colon-separated fields; two of the fields
may have optional sub-fields, marked off by a slash.
Fields or sub-fields that take multiple parameters should be separated
by a comma.
Extra whitespace can cause problems.
Except for the site names, case is significant.
The format of an entry is:
.RS
.nf
sitename[/exclude,exclude,...]\e
	:pattern,pattern...[/distrib,distrib...]\e
	:flag,flag...\e
	:param
.fi
.RE
Each field is described below.
.PP
The
.I sitename
is the name of the site to which a news article can be sent.
It is used for writing log entries and for determining if an
article should be forwarded to a site.
If
.I sitename
already appears in the article's Path header, then the article will not
be sent to the site.
The name is usually whatever the remote site uses to identify itself in
the Path line, but can be almost any word that makes sense; special local
entries (such as archivers or gateways) should probably end with an
exclamation point to make sure that they do not have the same name as any
real site.
For example, ``gateway'' is an obvious name for the local entry that
forwards articles out to a mailing list.
If a site with the name ``gateway'' posts an article, when the local site
receives the article it will see the name in the Path and not send the
article to its own ``gateway'' entry.
See also the description of the ``Ap'' flag, below.
If an entry has an exclusion sub-field, then the article will not be sent
to that site if any of the names specified as
.IR exclude s
appear in the Path header.
The same
.I sitename
can be used more than once \(em the appropriate action will be taken for
each entry that should receive the article, regardless of the name \(em although
this is recommended only for program feeds to avoid confusion.
Case is not significant in site names.
.PP
The
.I patterns
specify which groups to send to the site and are interpreted to build
a ``subscription list'' for the site.
The default subscription is to get all groups.
The patterns in the field are
.IR wildmat (3)-style
patterns, and are matched in order against the list of newsgroups that the
local site receives.
If the first character of a pattern is an exclamation mark, then any groups
matching the pattern are removed from the subscription, otherwise any
matching groups are added.
For example, to receive all ``comp'' groups, but only comp.sources.unix
within the sources newsgroups, the following set of patterns can be
used:
.RS
.nf
comp.*,!comp.sources.*,comp.sources.unix
.fi
.RE
There are three things to note about this example.
The first is that the trailing ``.*'' is required.
The second is that, again, the result of the last match is the most important.
The third is that ``comp.sources.*'' could be written as ``comp.sources*''
but this would not have the same effect if there were a
``comp.sources-only'' group.
.PP
There is also a way to subscribe to a newsgroup negatively.  That is to
say, do not send this group even if the article is cross-posted to a
subscribed newsgroup.
If the first character of a pattern is an atsign ``@'', it means that any article
posted to a group matching the pattern will not be sent even though the
article may be cross-posted to a group which is subscribed.  The same rules
of precedence apply in that the last match is the one which counts.
For example, if you want to prevent all articles posted to any
"alt.binaries.warez" group from being propagated even if it is
cross-posted to another "alt" group or any other group for that
matter, then the following set of patterns can be used:
.RS
.nf
alt.*,@alt.binaries.warez.*,misc.*
.fi
.RE
If you reverse the alt.* and alt.binaries.warez.* patterns, it would
nullify the atsign because the result of the last
match is the one that counts.  Using the above example, if an article
is posted to one or more of the alt.binaries.warez.* groups and is
cross-posted to misc.test, then the article is not sent.
.PP
See
.IR innd (8)
for details on the propagation of control messages.
.PP
A subscription can be further modified by specifying ``distributions'' that
the site should or should not receive.
The default is to send all articles to all sites that subscribe to any of
the groups where it has been posted , but if an article has a Distribution
header and any
.IR distrib s
are specified, then they are checked according to the following rules:
.IP 1.
If the Distribution header matches any of the values in the sub-field,
then the article is sent.
.IP 2.
If a distrib starts with an exclamation point, and it matches the
Distribution header, then the article is not sent.
.IP 3.
If Distribution header does not match any distrib in the site's
entry, and no negations were used, then the article is not sent.
.IP 4.
If Distribution header does not match any distrib in the site's entry, and
any distrib started with an exclamation point, then the article is sent.
.PP
If an article has more than one distribution specified, then each one
is according to the above rules.
If any of the specified distributions indicate that the article
should be sent, it is; if none do, it is not sent \(em the rules are used
as a ``logical or.''
It is almost definitely a mistake to have a single feed that specifies
distributions that start with an exclamation point along with some that don't.
.PP
Distributions are text words, not patterns; entries like ``*'' or ``all''
have no special meaning.
.PP
The
.I flags
parameter specifies miscellaneous parameters.
They may be specified in any order; flags that take values
should have the value immediately after the flag letter with no
whitespace.
The valid flags are:
.TP
.BI < size
An article will only be sent to the site if it is less than
.I size
bytes long.
The default is no limit.
.TP
.BI > size
An article will only be sent to the site if it is greater than
.I size
bytes long.
The default is no limit.
.TP
.BI A checks
An article will only be sent to the site if it meets the requirements specified
in the
.IR checks ,
which should be chosen from the following set:
.nf
	d	Distribution header required
	p	Do not check Path header before propagating
.fi
.TP
.BI B high/low
If a site is being fed by a file, channel, or exploder (see below), the server
will normally start trying to write the information as soon as possible.
Providing a buffer may give better system performance and help smooth out
overall load if a large batch of news comes in.
The value of the this flag should be two numbers separated by a slash.
The first specifies the point at which the server can start draining
the feed's I/O buffer, and the second specifies when to stop writing
and begin buffering again; the units are bytes.
The default is to do no buffering, sending output as soon as it is
possible to do so.
.TP
.BI F name
This flag specifies the name of the file that should be used if it is
necessary to begin spooling for the site (see below).
If
.I name
is not an absolute pathname, it is taken to be relative to
.\" =()<.IR @<_PATH_BATCHDIR>@ .>()=
.IR /news/out.going .
Then, if the destination is a directory, the file
.I togo
in that directory will be used as filename.
.TP
.BI G count
If this flag is specified, an article will only be sent to the site if
it is posted to no more than
.I count
newsgroups.
.TP
.BI H count
If this flag is specified, an article will only be sent to the site
if it has
.I count
or fewer sites in its Path line.
This flag should only be used as a rough guide because of the loose
interpretation of the Path header; some sites put the poster's name
in the header, and some sites that might logically be considered to be
one hop become two because they put the posting workstation's name in
the header.
The default value for
.I count
is one.
.TP
.BI I size
The flag specifies the size of the internal buffer for a file feed.
If there are more file feeds then allowed by the system, they will
be buffered internally in least-recently-used order.
If the internal buffer grows bigger then
.I size
bytes, however, the data will be written out to the appropriate file.
.\" =()<The default value is @<SITE_BUFFER_SIZE>@ bytes.>()=
The default value is (16 * 1024) bytes.
.TP
.BI N modifiers
The newsgroups that a site receives are modified according to the
.IR modifiers ,
which should be chosen from the following set:
.nf
	m	Only moderated groups
	u	Only unmoderated groups
.fi
.TP
.BI S size
If the amount of data queued for the site gets to be larger than
.I size
bytes, then the server will switch to spooling, appending to a file
specified by the ``F'' flag, or
.\" =()<.I @<_PATH_BATCHDIR>@/ sitename>()=
.I /news/out.going/ sitename
if the ``F'' flag is not specified.
Spooling usually happens only for channel or exploder feeds.
.TP
.BI T type
This flag specifies the type of feed for the site.
.I Type
should be a letter chosen from the following set:
.nf
	c	Channel
	f	File
	l	Log entry only
	m	Funnel (multiple entries feed into one)
	p	Program
	x	Exploder
.fi
Each feed is described below in the section on
.IR "feed types" .
The default is
.IR Tf .
.TP
.BI W items
If a site is fed by file, channel, or exploder, this flag controls what
information is written.
If a site is fed by a program, only the asterisk (``*'') has any effect.
The
.I items
should be chosen from the following set:
.nf
	b	Size of the article in bytes
	f	Article's full pathname
	g	The newsgroup the article is in;
		if cross-posted, then the first of the groups this
		site gets
	m	Article's Message-ID
	n	Article's pathname relative to the spool directory
	p	The time the article was posted as seconds since epoch.
.\" =()<.ie '@<IPADDR_LOG>@'DONT' \{\>()=
.ie 'DO'DONT' \{\
	s	The site that fed the article to the server;
		from the Path header\}
.el \{\
	s	The IP address of the site that sent the article\}
	t	Time article was received as seconds since epoch
	*	Names of the appropriate funnel entries;
		or all sites that get the article
	D	Value of the Distribution header;
		? if none present
	H	All headers
	N	Value of the Newsgroups header
	O	Overview data
	R	Information needed for replication
.fi
More than one letter can be used; the entries will be separated by a
space, and written in the order in which they are specified.
The default is
.IR Wn .
.sp
The ``H'' and ``O'' items are intended for use by programs that create
news overview databases.
If ``H'' is present, then the all the article's headers are written followed
by a blank line.
An Xref header (even if one does not appear in the filed article)
and a Bytes header, specifying the article's size, will also be part of
the headers.
If used, this should be the only item in the list; if preceeded by
other items, however, a newline will be written before the headers.
The ``O'' generates input to the
.IR overchan (8)
program.
It, too, should be the only item in the list.
.sp
The asterisk has special meaning.
It expands to a space-separated list of all sites that received the
current article.
If the site is the target of a funnel however (i.e., it is named by other
sites which have a ``Tm'' flag), then the asterisk expands
to the names of the funnel feeds that received the article.
If the site is fed by a program, then an asterisk in the
.I param
field will be expanded into the list of funnel feeds that received the article.
A site fed by a program cannot get the site list unless it is the target
of other ``Tm'' feeds.
.PP
The interpretation of the
.I param
field depends on the type of feed, and is explained in more detail below
in the section on
.IR "feed types" .
It can be omitted.
.PP
The site named
.I ME
is special.
There should only be one such entry, and it should be the first entry
in the file.
If the
.I ME
entry has a subscription list, then that list is automatically
prepended to the subscription list of all other entries.
For example, ``*,!control,!junk,!foo.*'' can be used to set up the
initial subscription list for all feeds so that local postings are
not propagated unless ``foo.* explicitly appears in the site's subscription
list.
Note that most subscriptions should have ``!junk,!control'' in their pattern
list; see the discussion of ``control messages'' in
.IR innd (8).
(Unlike other news software, it does not affect what groups are received;
that is done by the
.IR active (5)
file.)
.PP
If the
.I ME
entry has a distribution subfield, then only articles that match
the distribution list are accepted; all other articles are rejected.
A commercial news server, for example, might have ``/!local'' to reject
local postings from other, misconfigured, sites.
.SS "FEED TYPES"
.I Innd
provides four basic types of feeds: log, file, program, and channel.
An exploder is a special type of channel.
In addition, several entries can feed into the same feed; these are
funnel feeds, that refer to an entry that is one of the other types.
Note that the term ``feed'' is technically a misnomer, since the server
does not transfer articles, but reports that an article should be sent to
the site.
.PP
The simplest feed is one that is fed by a log entry.
Other than a mention in the news logfile, no data is ever written out.
This is equivalent to a ``Tf'' entry writing to
.I /dev/null
except that no file is opened.
.PP
A site fed by a file is simplest type of feed.
When the site should receive an article, one line is written to the file
named by the
.I param
field.
If
.I param
is not an absolute pathname, it is taken to be
relative to
.\" =()<.IR @<_PATH_BATCHDIR>@ .>()=
.IR /news/out.going .
If empty, the filename defaults to
.\" =()<.IR @<_PATH_BATCHDIR>@/ sitename.>()=
.IR /news/out.going/ sitename.
This name should be unique.
.PP
When a site fed by a file is flushed (see
.IR ctlinnd ),
the following steps are performed.
The script doing the flush should have first renamed the file.
The server tries to write out any buffered data, and then closes the file.
The renamed file is now available for use.
The server will then re-open the original file, which will now get created.
.PP
A site fed by a program has a process spawned for every article that
the site receives.
The
.I param
field must be a
.IR sprintf (3)
format string that may have a single
.I %s
parameter, which will be given a pathname for the article, relative
to the news spool directory.
The full path name may be obtained by prefixing the
.I %s
in the
.I param
field by the news spool directory prefix.
Standard input will be set to
the article or
.I /dev/null
if the article cannot be opened for some reason.
Standard output and error
will be set to the error log.
The process will run with the user and group ID of the
.\" =()<.I @<_PATH_INNDDIR>@>()=
.I /usr/local/news/run
directory.
.I Innd
will try to avoid spawning a shell if the command has no shell
meta-characters; this feature can be defeated by appending a semi-colon
to the end of the command.
The full pathname of the program to be run must be specified; for security,
PATH is not searched.
.PP
If the entry is the target of a funnel, and if the ``W*'' flag is used,
then a single asterisk may be used in the
.I param
field where it will be replaced by the names of the sites that fed into
the funnel.
If the entry is not a funnel, or if the ``W*'' flag is not used, then the
asterisk has no special meaning.
.PP
Flushing a site fed by a program does no action.
.PP
When a site is fed by a channel or exploder, the
.I param
field names the process to start.
Again, the full pathname of the process must be given.
When the site is to receive an article, the process receives a line on its
standard input telling it about the article.
Standard output and error, and the user and group ID of the all
sub-process are set as for a program feed, above.
If the process exits, it will be restarted.
If the process cannot be started, the server will spool input to a
file named
.\" =()<.IR @<_PATH_BATCHDIR>@/ sitename.>()=
.IR /news/out.going/ sitename.
It will then try to start the process some time later.
.PP
When a site fed by a channel or exploder is flushed, the server closes down its
end of the pipe.
Any pending data that has not been written will be spooled; see the description
of the ``S'' flag, above.
No signal is sent; it is up to the program to notice EOF on
its standard input and exit.
The server then starts a new process.
.PP
Exploders are a superset of channel feeds.
In addition to channel behavior, exploders can be sent command lines.
These lines start with an exclamation point, and their interpretation
is up to the exploder.
The following messages are generated automatically by the server:
.RS
.nf
newgroup group
rmgroup group
flush
flush site
.fi
.RE
These messages are sent when the
.I ctlinnd
command of the same name is received by the server.
In addition, the ``send'' command can be used to send an arbitrary
command line to the exploder child-process.
The primary exploder is
.IR buffchan (8).
.PP
Funnel feeds provide a way of merging several site entries into a
single output stream.
For a site feeding into a funnel, the
.I param
field names the actual entry that does the feeding.
.PP
For more details on setting up different types of news feeds, see the
INN installation manual.
.SH EXAMPLES
.RS
.nf
##  Initial subscription list and our distributions.
ME:*,!junk,!foo.*/world,usa,na,ne,foo,ddn,gnu,inet\e
	::
##  Feed all moderated source postings to an archiver
.\" =()<.ds R$ @<_PATH_NEWSBIN>@>()=
.ds R$ /usr/local/news/bin
source-archive!:!*,*sources*,!*wanted*,!*.d\e
	:Tc,Wn:\*(R$/archive \-f \-i \e
	    /usr/spool/news.archive/INDEX
##  Watch for big postings
watcher!:*\e
	:Tc,Wbnm\e
	:exec awk '$1 > 1000000 { print "BIG", $2, $3 }' >/dev/console
##  A UUCP feed, where we try to keep the "batching" between 4 and 1K.
ihnp4:/world,usa,na,ddn,gnu\e
	:Tf,Wnb,B4096/1024:
##  Usenet as mail; note ! in funnel name to avoid Path conflicts.
##  Can't use ! in "fred" since it would like look a UUCP address.
fred:!*,comp.sources.unix,comp.sources.bugs\e
	:Tm:mailer!
barney@bar.com:!*,news.software.nntp,comp.sources.bugs\e
	:Tm:mailer!
mailer!:!*\e
	:W*,Tp:/usr/ucb/Mail -s "News article" *
##  NNTP feeds fed off-line via nntpsend or equivalent.
feed1::Tf,Wnm:feed1.domain.name
peer.foo.com:foo.*:Tf,Wnm:peer.foo.com
##  Real-time transmission.
.\" =()<.ds R$ @<_PATH_NEWSBIN>@>()=
.ds R$ /usr/local/news/bin
mit.edu:/world,usa,na,ne,ddn,gnu,inet\e
	:Tc,Wnm:\*(R$/nntplink -i stdin mit.edu
##  Two sites feeding into a hypothetical NNTP fan-out program:
nic.near.net:\e
	:Tm:nntpfunnel1
uunet.uu.net/uunet:!ne.*/world,usa,na,foo,ddn,gnu,inet\e
	:Tm:nntpfunnel1
nntpfunnel1:!*\e
	:Tc,Wmn*:\*(R$/nntpfanout
##  A UUCP site that wants comp.* and moderated soc groups
uucpsite!comp:!*,comp.*/world,usa,na,gnu\e
	:Tm:uucpsite
uucpsite!soc:!*,soc.*/world,usa,na,gnu\e
	:Tm,Nm:uucpsite
uucpsite:!*\e
	:Tf,Wnb:/usr/spool/batch/uucpsite
.fi
.RE
.PP
The last two sets of entries show how funnel feeds can be used.
For example, the
.I nntpfanout
program would receive lines like the following on its standard input:
.RS
.nf
<123@litchi.foo.com> comp/sources/unix/888 nic.near.net uunet.uu.net
<124@litchi.foo.com> ne/general/1003 nic.near.net
.fi
.RE
Since the UUCP funnel is only destined for one site, the asterisk
is not needed and entries like the following will be written into the file:
.RS
.nf
<qwe#37x@snark.uu.net> comp/society/folklore/3
<123@litchi.foo.com> comp/sources/unix/888
.fi
.RE
.SH HISTORY
Written by Rich $alz <rsalz@uunet.uu.net> for InterNetNews.
.de R$
This is revision \\$3, dated \\$4.
..
.R$ $Id: newsfeeds.5,v 1.34 1996/09/28 17:31:09 brister Exp $
.SH "SEE ALSO"
active(5),
buffchan(8),
ctlinnd(8),
innd(8),
wildmat(3).
