		Hopfield-style network simulator
	       Arun Jagota (jagota@cs.buffalo.edu)

If you plan to install to use this simulator, I suggest that you
drop a line to the above e-mail address. It will help me keep you informed
about newer developments on the network (like new applications, papers etc..)
or the simulator (new versions etc) as and when that happens.

Installation
------------
1. This s/w is all in C and should install without change on
any 32-bit Unix machine. To create the executable,
cd src
make
-- The executable 'mlhn' will be created in the directory 'src'.
-- Move it to 'bin' and try out one of the examples as follows
cd ../examples
../bin/mlhn room -c < room.train
-- You should be in the directory which contains the example.

2. To install it on a 16-bit machine (Unix or MSDOS), change the value 
of the constants MIN_VIGILANCE and MAX_W (in main.c) from -200000000 to
-32000 and 2500000 to 25000 respectively. This is quite necessary. These
 are the only required changes and have been tested on an IBM XT.
A file called Makefile.DOS for the MSDOS make utility has been provided
in the 'src' directory and might be useful.
3. It will probably also work on any other non-Unix machine since it is 
all written in standard C.

Changes
-------
Within a year, I will probably announce a newer version with added 
features.  It will make it easier for you to "roll" your changes 
(if any) to this s/w to the new version if you track the changes.

Usage
-----
	mlhn <network> [{-t}] [-c]

	-t   : Test only. mlhn will read standard input and do
	       energy-descent.
	-c   : Assume all units in input are clamped to 1 unless
	       <unit-name>*0.  mlhn will read the standard input
	       as containing training data
	[for other flags/options, see the header of the file main.c]

<network>.cfg has to be made in the directory from which mlhn is
invoked. It is a configuration file which must contain 5 integers
on a single line. The meaning and typical range of the numbers is as
follows.

max_units lambda lambda_R vigilance       #size units
10-1000   5 or 7  1-5   -10 to -4000      1-12

Eg

mlhn mynet -t

Files : mynet.cfg

mynet.cfg (* the following 1 line in the file mynet.cfg*)

400     5    1  -4000 10   

Input format (during training)

{ a b c }
....

Input Format (to mlhn, during testing)


{ a b c }

{ a*1 b*1 c*1 }

{ a*0 b*1 c }

{ 90-a 80-b 7-c }   (* Optional confidence values in interval [1-100] *)

-- The only delimiters in sets are blank[s], tab[s] or end-of-line[s].
-- So {a b c } would be an illegal set and might cause the symbol 'a' to
-- be ignored.

Debugging features (Commands accepted from stdin)
-------------------------------------------------

w 1 <first-set> <second-set>
-- Will display weights in matrix format as follows.
-- rows are 'first-set' and columns are 'second-set'
Eg,
w 1 { a b c } { d e f g }

-- will display
num11 num12 num13 num14
num21 num22 num22 num24
num31 num32 num33 num34

-- where the rows and columns correspond to
   d      e     f    g
a
b
c

r 1 <first-set>
-- Will display the internal resistance (threshold) of each unit.

Example
-------

1. Training file (mynet.train)
{ a b c }
{ c d e }
.......


2. mynet.cfg
100 7 1 -20 1

/* Train */
mlhn mynet -c < mynet.train

/* Test if network has stored all memories cleanly */

mlhn mynet -t < mynet.train > mynet.tst


NOTE : network is proven to store all memories 'cleanly' ONLY when
the domain has a combinatorial structure (See paper, 
A Hopfield-style network for content-addressable memories - 90-02)

If your application domain has a combinatorial structure, that's fine
otherwise, any n-dimensional binary domain can be converted to one with 
a combinatorial structure as follows. Need a unit for each value {0,1}
at each position. This requires 2n units.

Example : Store the 3-d vectors 101, 111,  011
It is recommended you use other symbols (like {y,n}) instead of
{0,1}. Use numerals to identify positions.
Based on above, we want to store yny, yyy, nyy

Store as follows:

{ y1 n2 n3 } 
{ y1 y2 y3 }
{ n1 y2 y3 }

Modifying individual weights
----------------------------
It is also possible to modify individual weights by presenting the
corresponding two units as a binary set. For eg, if you have to 
modify the weight w_ab, then present { a b } during training. However,
this should be done USING the -ns flag to mlhn which tells mlhn not
to automatically include the size unit during training. Otherwise
{ a b } will be considered a binary memory and not just a weight
in perhaps a larger memory. In some sense, this allows you to 
customise the learning rule (within limits). The script 'unlearn_i',
for eg, uses this because during unlearning we don't want to unlearn
all pairs of connections in a spurious state, ONLY those connections
between units in initial state and units that were switched ON. Without
the '-ns' option, there would be no way to accomplish this.

Exploiting order in the net O/P
-------------------------------
For any 'set' I/P, the net O/P is another 'set'. Actually this 
'set' has some order information in it that can be exploited if
necessary. If we know that the network switched k units ON during
energy descent, then the k left-most elements in O/P set will be in
the reverse order of the switches, the 1st element (from left) being the
last switch ON and the kth element being the first switch ON. 
At this time, unfortunately, the O/P set contains no information about
the order in which units (if any) were switched OFF or which was the
first element that was switched ON, if k is unknown. The latter can
be found by subtracting the I/P set from the O/P set but preserving
the order in the O/P set. If the result is non-empty, then the result
is a set which contains all and only those units that were switched ON 
and in reverse order. The former (info about switch OFF order) will
be provided in some subsequent version of simulator.

Files created while training
----------------------------
Suppose you train a network as follows

mlhn mynet -c < mynet.train

where mynet.train contains the training set.

mlhn will create the following files (in your current directory) in 
which it will store the network parameters.

mynet.pn1 - file contains all symbols used in the training set.
mynet.R1  - value of internal resistance for each unit
mynet.w1  - weight matrix (run-length encoded)
mynet.wup1- Not used for 1-layer net but should be left untouched

Numeric details
---------------
No 'real' arithmetic is performed. +ve weight values in [0,1] 
are done in practice in the integer interval 0..SCALE-1, where
'SCALE' is a constant declared in the file 'std.h'. To change the 
precision, simply change it's value and recompile everything
(typing 'make' will do it). If 'SCALE' is changed, 'SCALE_POWER'
should also be changed accordingly. 'SCALE' should always be a power
of 2.
Similarly, R_i, the internal resistance of each unit is in the integer
interval 0..THRESH_SCALE. To change the precision, change THRESH_SCALE
and THRESH_SCALE_POWER in file 'std.h' and recompile (type 'make').
Again, THRESH_SCALE should be a power of 2. It is recommended that
SCALE and THRESH_SCALE should always have the same value.
Initial values of units (recall units can be continous-valued in test
i/p) are in the integer range 0..HIGH-1. As before, to change the
precision, change HIGH (and HIGH_POWER) in std.h. Again, HIGH should
always be a power of 2.

Pre/Postprocessing utilities
----------------------------
The following utilities are included that do the following.

some_sequence -> wrd_in.c -> { s1 o2 m3 e4 _5 s6 e7 q8 u9 c10 e11 }
{ s1 o2 m3 e4 _5 s6 e7 q8 u9 c10 e11 } -> wrd_out.c -> some_sequence

....xxx..
.xxxxxx..  -> vis_in.c -> { .1 .2 .3 .4 x5 x6 ......... }
xxx.xxx..

                                      ....xxx..
{ .1 .2 .3 x4 x5 ...} -> vis_out.c -> .xxxxxx..  
                                      xxx.xxx..


Additional documentation
------------------------
1) Read the shell scripts train, tst and unlearn_i, especially
the last one.
2) Read the headers of the source files main.c, wrd_in.c

Known Bugs
----------
1. At least 1 size unit has to be put into the configuration file.
This causes no loss of generality.

1. Very rarely, when the input sets are too large, the network
gives segmentation fault after having processed 200-300 of them.
It may be related to running out of memory at some point in time.
All results before that are correct. Simply run the network until
a segmentation fault occurs, and re-run it on the remaining input.
This will work if the input is test input to the network. If it is
training input then it won't work. Break up the input into multiple
parts such that no segmentation fault occurs. This phenomenon is
quite rare.

Relevant Papers and Reports
---------------------------
The following papers are relevant (in priority order) and their LaTeX
sources can be made available by e-mail (send request to 
jagota@cs.buffalo.edu). The first 3 papers have minimal overlap and
are recommended reading.

%T A new Hopfield-style network for content-addressable memories
%A Arun Jagota (1990)
%P Technical Report 90-02, Dept Of Computer Science, Suny Buffalo, NY
%X Mathematical description of the model and its properties. A must.

%T Applying a Hopfield-style network to Degraded Text Recognition
%A Arun Jagota (1990). 
%P To appear in Proceedings of the International Joint Conference on 
   Neural Networks, San Diego, 1990
%X real-world application

%T Applying a Hopfield-style network to Degraded Printed Text Restoration
%A Arun Jagota (1990). 
%P To appear in Proceedings of the 3rd conference on neural networks
   and PDP, Indiana-Purdue university
%X real-world application

%T Knowledge representation in a multi-layered Hopfield network
%A Arun Jagota, Oleg Jakubowicz
%P Proceedings of the International Joint Conference on 
   Neural Networks, Wash DC, June 1989, I-435
%X multi-layer version of above model. Architecture designed with care
   but properties not fully explored. Simulator has partial implementa-
   tion of above architecture. For info, send e-mail. 

%T A neural lexicon in a Hopfield-style network
%A Arun Jagota, Y.S. Hung
%P Proceedings of the International Joint Conference on 
   Neural Networks, Wash DC, Jan 1990, II-607
%X Precursor to application-I (Degraded text recognition). 
   Detailed experiments with large word sets. Most, though not all
   the information in this paper can be infered from TR 90-02 or the
   Degraded Text recognition paper.
