Contents
=========

  1. mozg Introduction.

  2. "Soft" rules.

  3. mozg Programmer's guide.

    3.1 MLPerceptron class (high level).

      3.1.1 High level functions.

      3.1.2 Code example of a network learning.

      3.1.3 Code example of a saved net using.

    3.2 MLPerceptron class (middle & low level).

      3.2.1 Middle level functions.

      3.2.2 Low level functions(layer interactions functions).

      3.2.3 How to insert other output function.

        3.2.3.1 For net = sum(w_{ij}*y_j).

        3.2.3.2 For RBF_net = -.5*sum(m_{ij}-y_j)^2/sigma_i^2.

      3.2.4 How to change basic types.

    3.3 Kernel library (mozgVector,mozgMatrix classes templates).

      3.3.1 Description.

      3.3.2 Optimisation technique.

  4. Bibliography.

*****************************************************************************

                        1. mozg Introduction
                       ======================

    mozg is a flexible, fast neurolibrary. Up to the moment it allows one
    to create, learn and use multi-layer perceptron (MLP) [1,9], which is the
    most popular artificial neural network (ANN) for the large number of
    problems solved by a ANN.

      Structural parts of the mozg are:

        1. Kernel library is the library of fast vector and matrix operations. 
    It consists of mozgVector and mozgMatrix class temlates [2] and functions 
    templates using them (see mozgVectorMatrix.cc, mozgVectorMatrix.hh).
    Explanations of its appearance see in 3.3.1 (Description).
    
        2. MLPerceptron class (using the Kernel library) simulates the MLP and
    its behaviour. One can by the constructor create an ANN with desiring the 
    number of layers, the number of neurons (called units) in them, output
    function in each layer (except input layer storing network's inputs), 
    with/without bias term, learn, use, save and load a network
    (see MLPerceptron.cc, MLPerceptron.hh).

      Dynamic memory allocations have been placed in the constructor of MLP
    (and in putLParams (b)) and don't influence on the MLP learning and using
    CPU times. Constructors of vectors and matrices is called only in MLP con-
    structor and in putLParams (b), so there's no unnecessary vector & matrix
    constructor calls.

      Output of i-th neuron of a layer is computed as [1,3]:

        output_i = f(net_i) = f(sum_j(w_{ij}*input_j+theta_i)),

    where f(net_i) -- neuron's output function,
          net_i -- weighted inputs sum of i-th neuron
          w_{ij} -- weight for j-th input of i-th neuron,
          input_j -- j-th input of its,
          theta_i -- bias term or simply bias of its, for binary neuron it's 
    called threshhold.

      Sometimes they use neuron's activation function:

        a_i(t) = g(a_i(t-1),net_i),

        output_i(t) = f(a_i(t)),

    to use dependensy of neuron's output from time. It hasn't been implemented
    in the simulator.

      The temperature (T) in the output function allows to change slope para-
    meter of the output function [3]:

        output_i = f(net_i/T).

    Its decreasing allows to sharpen error surface (called energy landscape)
    and conversly its increasing allows to soften the surface, so one can
    manage learning by it.

      For any layer (except input one storing network's inputs) neurons' output
    function may be choosed from (standard set):

        1. 1/(1+exp[-net/T]),
        2. th(net/T),
        3. net/T,
        4. Radial Basis Function --
           RBF = exp(-.5*sum(m_{ij}-y_j)^2/(sigma_i^2*T)),
        5. 2/Pi arctg(net/T),

    or desiring one may be inserted (see mozg programmer's guide). RBF is a
    "local" function and differs from the others by the "net", which isn't
    weighted input sum as for those. Due to it the learning rule changes
    considerably for the RBF layer and previous one. I've deduced and imple-
    mented in the simulator error backpropagation rule for three cases:

      a) RBF for hidden and output layer,

      b) RBF for hidden, sigmoid for output layer,

      c) sigmoid for hidden, RBF for output layer,

    that allow to use RBF in any combinations with sigmoid on different layers.
    Example of deducing error backpropagation rule for RBF for output and
    sigmoid for hidden layer see in [5]. Local RBF nature may lead to a
    "sharp sweep". It is the bad phenomenon caused by lack of learning examples
    or them bad "scattering" in the input space and clear shows itself on the
    function approximation problem. This trouble one can overcome if one put
    the RBF_sigma_min > 0 (large value results in the network learningless)
    or good scattering of learning examples.

      Implemented learning rule is the "vanilla" backpropagation extended by
    the momentum term [1,2,7], Langevin noise [4], weight decay [4,7]:

        w_{ij}(t+1) = w_{ij}(t) + delta w_{ij}(t) =
                    = w_{ij}(t) -
                      - eta * d E / d w_{ij} +
                      + alpha * delta w_{ij}(t-1) +
                      + Langevin_noise_term -
                      - decay_coef * w_{ij}(t).

      Any from three energy functions may be used in learning of a network:

        1) summed squared error E = sum_p sum_i (t_i^p-o_i)^2,

        2) Cross-Entropy E = sum_p sum_i ((1-t_i^p)*ln(1-o_i)-t_i^p*ln(o_i)),

        3) log-squared error E = - sum_p sum_i ln(1-(o_i-t_i^p)^2),

    Surely, they may be used in turn for learning of one network.

      Possibility to learn network with random order of learning examples
    using has been implemented in simulator, it improves network generaliza-
    tion (property of network to recognize examples not used in learning
    correctly) [4].

      Sometimes I had trouble with learning 1,2,...-hidden layers network
    with linear output function for all layers(it is meaningless, but for
    the sake interest). Learning lost stability and simulator crashed.
    I don't know why it occurs.

      Wasilx Urazmetow rendered noticeble assistance and can be named second
    mozg author.

      Please, send me lost bugs and suggestions related to the mozg to the
    e-mail filin@desert.ihep.su. Good luck!
                                                Alexey P. Filin 19/09/99

*******************************************************************************

                        2. "Soft" rules
                       =================

        1. Chosen network inputs and outputs influence on learning results
        considerably, for a bad case network can't be learned. There isn't 
        guaranty of good learning result for the task at all. Your expe-
        rience is your advicer.

        2. Normalize inputs and outputs [6], i.e. make transformation

            new_input = ( old_input [- old_input_min] )
                          / | old_input_max - old_input_min |

        [- old_input_min] -- isn't certainly

        3. Divide your set of examples in two subset:
             - learning set (is used for learning),
             - test set (isn't used for learning)

        4. Try to use "vanilla" backpropagation rule [4] (without extentions)
        i.e. use function 2.a) (putLParams) for learning parameters settings 
        then optimised weights adjustment is used.
           Momentum term and Langevin noise are used for "flat spot" elimi-
        nation in errors-weights space [7]. When error derivative is too small
        for learning by "vanilla" backpropagation, backpropagation rule exten-
        tions have to be used. Langevin_sigma != 0 leads to N-FOLD CPU time in-
        creasing, only use Langevin noise for your need.
           Weight decay [4,7] is used for network architecture optimisation, it
        decreases absolute weights' values. It may lead to bad learning result.
           Bias term may be very usefull sometimes.

        5. The number of learning examples is recomended be >> the number of
        network weights or ANN will not generalize [4], it will remember of
        learning examples with their noise.

        6. Usually three successive regions of test errors (errors for
        examples not being used in learning) exist in the learning phase [6]:
             - test error decreasing,
             - test error optimal value,
             - test error increasing (overfitting).

  NOTE: error for learning examples usually decreases all learning time.
        Because of it test error checking is very usefull for learning result
        observation (see code example).
        
        7. Try to use different numbers of layers, numbers of hidden units,
        output functions. Learning results may be very different (That's
        life :).


*******************************************************************************

                   3. mozg Programmer's guide
                  ============================

    Three levels of called functions may be extracted:

        * level of interface functions providing network interaction with
    outer objects and network management. They have access "public" in the
    MLPerceptron (except operator-functions ">>" and "<<").

        * level of member-functions of the MLPerceptron with access
    "protected" (except propLayer()). These functions provide interaction of
    neighbouring network layers,

        * low level(kernel library). Functions for vector and matrix opera-
    tions. Those functions use classes templates mozgVector and mozgMatrix.

      Let's go into details of those levels.

3.1 MLPerceptron class (high level):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
3.1.1 High level functions for MLPerceptron using are (see MLPerceptron.hh and
MLPerceptron.cc):

member-functions:

        1. MLPerceptron (int  number_of_layers,
                         int* num_outs_layers,
                         int* func_number,
                         float weight_dispersion,
                         bool bias_term_flag);
                         
              int number_of_layers -- number of layers (including input layer),
                it's usually enough three layers,sometimes four,

              int* num_outs_layers -- pointer to the massive wich contains
                numbers of units at layers; if the bias term is used, numbers
                of units in a layer MUST INCLUDE one extra unit -- bias unit,
                which models bias input for the next layer, that is

                       units_number = real_units_number + 1,

                EXCEPT output layer, which doesn't contain bias unit (because
                the layer is last);

          NOTE: at the first layer the number of units also must include bias
                unit, but network input vector dimension equals to the number
                of layer units WITHOUT bias unit,

              int* func_number -- pointer to the massive wich contains output
                 function numbers at layers. One can put it equal to:
                                  
                    1 -- 1/(1+exp[-net/T]),
                    2 -- th(net/T),
                    3 -- net/T,
                    4 -- Radial Basis Function,
                    5 -- 2/Pi arctg(net/T),
                    0 -- user defined function(see 3.2.3 of the text),
                    6 -- user defined function(the same).

                  For example, { 0, 4, 2}:
                    0 on the input layer (it may be any, because there's no
                      output function at the layer),
                    4 at the hidden layer,
                    2 at the output layer,

              float weight_dispersion -- dispersion of begin weight distribu-
                tion, bouth very small value and very large one leads to 
                network failed to be learned; try to use about 0.4;

              bool bias_term_flag -- flag of bias term, true (1) for its switch
                on and false (0) for its switch off; it is modeled by neuron's
                input, which allways equals to one -- bias input; sometimes it
                can be usefull, for example, for function approximation,
                without the term network's outputs is zero for zero network's
                inputs and odd neuron's output function (atg, th) so the net-
                work can't approximate function not equaling to zero for all
                zero inputs.

        create network, if RBF isn't used, then RBF_sigma_min doesn't
        influence on computations,


        2.a) putLParams (float learning_rate,
                         float temperature,
                         float RBF_sigma_minimum
		         int energy_function);

                float learning_rate -- common learning rate, usually value
                    about 0.3 is a good one. Learning rate for each layer is
                    computed as (rule of fan-in):

                       layer's_eta = learning_rate / layer's_inputs_number

                float temperature -- temperature in the output function, very
                    small value makes very sharp energy landscape, so learning
                    can be lost in local minima, very large one makes that
                    too sloping and network is learned too long.
                    at first try to use 1,

                float RBF_sigma_minimum -- allowed RBF sigma minimum in the 
                    learning phase, if RBF isn't used, it doesn't influence 
                    on computations (at first try to use zero);

                int energy_function -- you can use following energy function:

                      1 -- summed squared error:
                         E = sum_p sum_i (t_i^p-o_i)^2,

                      2 -- Cross-Entropy(don't use it for odd sigmoids):
                         E = sum_p sum_i (t_i^p*ln(o_i) +
                                         + (1-t_i^p)*ln(1-o_i)),

                      3 -- log-squared error:
                         E = - sum_p sum_i ln(1-(o_i-t_i^p)^2);

          put learning parameters for "vanilla" backpropagation rule,

          b) putLParams (float learning_rate,
                         float temperature,
                         float RBF_sigma_minimum,
		         int energy_function,
		         float mom_term_coef,
		         float lang_noise_begin_dispersion,
		         float lang_noise_disp_decr_coef,
		         float weight_decay_coef);

                float mom_term_coef -- momentum term coefficient(alpha) in
                    weight changing rule:

                    w_{ij}(t+1) = w_{ij}(t) + delta w_{ij}(t) =
                                = w_{ij}(t) + ... + alpha * delta w_{ij}(t-1)

                    is recomended to be in interval 0...1.

	        float lang_noise_begin_dispersion begin value of the disper-
                    sion of the Langevin noise(Gauss noise) term in a weight
                    changing rule:

                    w_{ij}(t+1) = w_{ij}(t) + ... + noise_term

                    It must be sufficiently small for successful learning.

	        float lang_noise_disp_decr_coef -- decreasing coefficient of
                    the Langevin dispersion in learning phase:

                    Langevin_sigma(t+1) =
                          = Langevin_sigma(t) * lang_noise_disp_decr_coef

                    where t+1 -- next call of the AdjustWeights (). It must be 
                    putted approximatly 0.99.

	        float weight_decay_coef -- weight decay coefficient for decay
                    term in a weight changing rule:

                    w_{ij}(t+1) = w_{ij}(t) + ... -
                                - weight_decay * w_{ij}(t).

                    It must be << 1, then a network can be learned(for a
                    normalized inputs);

        put learning parameters for backpropagation with extensions rule,
 
    ALL RECOMENDED VALUES INTEND FOR NORMALIZED INPUTS AND OUTPUTS (see "Soft"
    rules).

  NOTE: you can don't call the function then default values is used for lear-
        ning:

           layer[i].eta = 0.3 / num_outputs_layers[i];
           layer[i].temperature = 1.;
           RBF_sigma_min             = 0.;
           switch_of_energy_function = 1;
           alpha                     = 0.;
           sigma_Langevin            = 0.;
           sigma_Langevin_decrease   = 0.;
           decay                     = 0.;
           flag_of_optimisation      = true;
           error_threshold           = 0.;
           rate_coeff = 1.;
           temperature_coeff = 1.;


        3.a) learnNet (float* linputs,   // learning input vector
                       float* loutputs)  // learning output vector

          learn net for the learning example (one error backpropagation), by
          the function you can self choose order of learning examples in epochs
          what usefull if you want to use them in random order (it assists
          improve a network's generalization)

          b) learnNet (float** linputs,  // array of learning input vectors
                       float** loutputs, // array of learning output vectors
                       int lssize,       // learning set size
		       int vec_order,    // order of vectors used in learning
                       int epnum,        // number of epochs network must be
                                         // learned
                       int pepnum)       // answerMessage() (9.) is called for
                                         // each pepnum-th epoch

          learn network:

	    1) vec_order = 0 -- in random order of learning examples (regular
               distribution is used), that helps to improve network genera-
               lisation; in the case some examples will be used larger and some
               less number of times, but common number of error backpropagation
               is equal to one in a);

            2) vec_order = 1 -- in successive order of learning examples given
               by the array, one error backpropagation for each learning 
               example in epoch is used,

          each pepnum-th epoch learning error is printed.

    NOTE: learning error isn't the mean squared error on learning set,
          it is the mean of output network's error squares for learning set:

               learning_error = sum_i(target_i-realouput_i)^2/lssize

          where target_i -- i-th target output vector,
                realoutput -- i-th real output vector
          and each sum's term is calculated after a weight change,

          c) learnNet (float** linputs,  // array of learning input vectors
                       float** loutputs, // array of learning output vectors
                       int lssize,       // learning set size
		       int vec_order,    // order of vectors used in learning
                       int epnum,        // number of epochs network must be
                                         // learned
                       int pepnum,       // answerMessage() (9.) is called for
                                         // each pepnum-th epoch
                       float** tinputs,  // array of test input vectors
                       float** toutputs, // array of test output vectors
                       int tssize);      // test set size

          learn network (one error backpropagation for each learning example),
          each pepnum-th epoch the learning error and the test error is prin-
          ted.

    NOTE: test error is real mean squared error on test set:

                 test_error = (sum_i(target_i-realoutput_i)^2)/tssize ,

          where i (= 1...tssize) -- the number of test examples,
                target_i -- i-th target output vector,
                realoutput -- i-th real output vector,


        4. testNet (float** tinp, // array of test input vectors
                    float** tout, // array of test output vectors
                    int tssize);  // test set size

          test network and return test error.

    NOTE: test error is real mean squared error on test set,


        5.a) propInputs (float* invec,   // input vector
                         float* outvec); // output vector(allocated by
                                         // programmer)
          propagate input vector invec through network and place output vector
          in outvec

          b) propInputs (float** invecarr,  // input vector massive
                         float** outvecarr, // output vector massive (allocated
                                            // by programmer)
                         int ssize);        // vectors number
          propagate input vector massive through network and place output
          vector massive in outvecarr,


not member-functions:

        6. output << net
           ofstream& output -- output file stream,
           MLPerceptron& net -- network to be saved,
        save the network in the file,


        7. input >> p_p_net
            ifstream& input -- input file stream,
            MLPerceptron** p_p_net -- address of the pointer to the network,

        load the network from the file,

  NOTE: network is loaded to the dynamic memory wich is allocated in the 
        operator-function, so you must self destroy the network if it isn't
        need


        8. output << net
            ostream& output -- output stream (cout),
            MLPerceptron& net -- network to be writed in the output,

        show the network.


PLEASE DEFINE IT:

        9. bool answerMessage(float learn_error, // learning error
			      float test_error); // test error

     The function is intended for error checking and management of learning
   "on fly". It is called by learnNet b),c) each pepnum-th epoch and is got
   learning error and test error if it is calculated. If test error isn't cal-
   culated (for learnNet b)) zero is put in second signature parameter of the
   function. If you like graphics use a graphic library then you can admire
   learning in real time.
     The function MUST RETURN:
       - true if network learning must be ended (error is small enough), then
   return from learnNet() takes place,
       - and false in the other case (continue learning).
     A simple example:

        bool MLPerceptron::answerMessage(float learn_error,
        				 float test_error)
        {
          cout << "Overall number of epoch since learning begin: "
        // overall_epnum -- member-datum of MLPerceptron class (float)
               << overall_epnum << endl
               << "  learning error: " << learn_error << endl
               << "  test error    : " << test_error << endl;

        // is_call_aM -- member-datum of MLPerceptron class (bool);
        // to enter test error once
          if (!is_call_aM) {
            is_call_aM = true;
            cout << "Enter test error which can be allowed: ";

        // error_threshold -- member-datum of MLPerceptron class (float)
            cin >> error_threshold;
          }
          return (test_error < error_threshold);
        }

    I don't use static-values in answerMessage() instead of member-data
    overall_epnum, is_call_aM and error_threshold, because I had troubles
    with them. When I had made several networks, static-values was common.
    Please, don't forget it, if you want to use static-value.
      In learning process you can change some learning parameters, learning
    rate or temperature for each layer except input one, for it you can write
    in answerMessage():

          if (!is_call_aM) {

            is_call_aM = true;
            cout << "Enter rate_coeff and temperature_coeff:";
            cin >> rate_coeff >> temperature_coeff;
          }

          for (int i=1; i<num_layers; i++) {

            layers[i].rate *= rate_coeff;
            layers[i].temperature *= temperature_coeff;
          }

  NOTE: you can continue learning of the network by the other call of learnNet
        (any from a),b),c)) then overall_epnum DOESN'T be zeroed and equals to
        overall number of epochs for several call of learnNet().


3.1.2 Code example of a network learning:

        ...

        float** learning_set_inputs,
             ** learning_set_outs,
             ** test_set_inputs,
             ** test_set_outs,
                eta,
                begin_wts_sigma,
                learning_rate,
                temperature,
                RBF_sigma_minimum;

        int   number_layers,
            * num_outputs_at_layers,
            * num_output_f,
              learning_set_size,
              vec_order,
              energy_function;

        bool flag_of_bias_term;

        // Initialisations of inputs & outputs vector arrays
        ...

        // Enter network and its learning parameters
        params_input (...);

        // Network is created:
        MLPerceptron net (number_layers,
                          num_outputs_at_layers,
                          num_output_f,
                          begin_wts_sigma,
                          flag_of_bias_term);

        // Learning parameters initialisation:
        net.putLParams (learning_rate,
                        temperature,
                        RBF_sigma_minimum,
		        energy_function);

        // Network is learned:
        net.learnNet (learning_set_inputs,
                      learning_set_outputs,
                      learning_set_size,
                      vec_order,
                      epoch_num,
                      print_epoch_num,
                      test_set_inputs,
                      test_set_outs,
                      test_set_size);


        // and is saved:
        bool flag_of_saving;
        char net_file_name[100];

        cout << "Save network? (yes-1, no-0)";
        cin >> flag_of_saving;

        if (flag_of_saving) {

          cout << "Enter file name network must be saved in: ";
          cin >> net_file_name;

          ofstream out_file (net_file_name);
          out_file << net;
        }
        ...

3.1.3 Code example of a saved net using:

        float* set_inputs,
             * input_vec,
             * output_vec;
        ...

        MLPerceptron* p_net;

        ifstream from_file (net_file_name);

        from_file >> &p_net;

        // I want to see loaded net...
        cout << *p_net;

        output_vec = new float[num_outs];

        p_net->propInputs (input_vec,
                           output_vec);
        ...

3.2 MLPerceptron class (middle & low level).
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
3.2.1 Middle level functions:

        10. putInputs (inputs)
            float* inputs -- pointer to begin of the massive containing network
                inputs,

        put the inputs in the network input layer,

        11. compOutputErrors (target)
           float* target -- pointer to begin of the desiring outputs massive,

        compute network out errors and place square of a error vector module
        in (t_i-y_i)^2 the error_over_outs,

        12. backpropErrors ()

        propagate out errors back through the network,

        13. adjustWeights ()

        adjust network weights,

        14. getOutputs(net_outputs)
           float* net_outputs -- pointer to massive output vector had been
                                placed in,

        put output layer outputs to the net_outputs (which allocated by
        programmer),


3.2.2 Low level(layer interactions functions):

        1. propLayer (...), DON'T FORGET BIAS TERM if you use
        the function (for example, see definition of learnNet).

        propagate signal through a layer,

following functions have access "protected":

        2. compute_lay_outs (...)

        compute outs of a hidden layer,

        3. compute_output_errors (...)

        compute errors of the out layer,

        4. compute_lay_errors (...)

        compute errors of a hidden layer,

        5. adjust_lay_weights (...)

        update weights of neurons of a layer.


3.2.3 How to insert other output function.

    All you need to insert other output function is to write in MLPOutFunc.cc
    desiring function, its derivative  and recompile mozg. In the following I
    consider two case, the case of weighted inputs net and the case of RBF net.

        1. For weighted input sum, net = sum(w_{ij}*y_j) you must write
	desiring output function in mozgweinet(net,sigm) and its derivative in
    	mozgDweinet(out,net,sigm) (see MLPOutFunc.cc).

      	For example, you want to insert y = sin(net), then you must write in
    	mozgweinet (net,sigm):

	  mozgflt mozgweinet (mozgflt net,mozgflt sigm)
	  {
  	    return sin(net);
	  }

    	Then, because of y' = cos(x) = sqrt(1-y^2), you must write in the
    	mozgDweinet (out,net,sigm):

	  mozgflt mozgDweinet (mozgflt out,mozgflt net,mozgflt sigm)
	  {
  	    return cos(net);
	  }
    	or (for some functions using of "out" instead of "net" is more
        convenient)

	  mozgflt mozgDweinet (mozgflt out,mozgflt net,mozgflt sigm)
	  {
  	    return sqrt(1 - out * out);
	  }

  NOTE: the net is neuron's net devided by temperature yet.

        2. If you want to use RBFnet = -.5*sum(m_{ij}-y_j)^2/sigma_i^2, for
     	example y = sin(RBFnet), you must write in mozgrbfnet (RBFnet,sigm):

	  mozgflt mozgrbfnet (mozgflt RBFnet,mozgflt sigm)
	  {
 	    return sin(-.5 * RBFnet / (sigm * sigm));
	  }

    	and, because of y' = cos(RBFnet) = sqrt(1-y^2), you must write in the
    	mozgDrbfnet (out,RBFnet,sigm):

	  mozgflt mozgDrbfnet (mozgflt out,mozgflt RBFnet,mozgflt sigm)
	  {
  	    return cos(-.5 * RBFnet / (sigm * sigm)) / (sigm * sigm);
	  }

    	or(for some functions using of out instead of RBFnet is more
	convenient)

	  mozgflt mozgDrbfnet (mozgflt out,mozgflt RBFnet,mozgflt sigm)
	  {
  	    return sqrt(1 - out * out) / (sigm * sigm);
	  }

  NOTE: the RBFnet is neuron's RBFnet multiplied by gain (slope parameter) yet.

    For frequent change of output function it is better to extract
    MLPOutFunc.cc from the library only to recompile it after each change of
    output function.


3.2.4 How to change basic types.

    If you want to change basic value type "float" by "double" (for example for
    precision increasing) and value type "int" by "long", you need to replace
    "float" by "double" and "int" by long in "types.hh" and recompile mozg.
    Then ALL mozg's "float" values are replaced by "double" values and ALL
    "int" values are replaced by "long" values.


3.3 Kernel library(mozgVector,mozgMatrix classes templates).
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
3.3.1 Description.

    Extracting of vector and matrix operations from neurosimulator is conse-
    quence of ANN model, that has been described in terms of vector and matrix
    operations. Calculations with vectors and matrices nearly occupy all time
    of a neurosimulator's run only leave some percents or even some shares of
    percent for rest, so to extract them in separate library is very conve-
    nient for further acceleration of the neurosimulator, because you only
    need to change some library's functions by faster ones (for example,
    by written in assembler or even hard implemented ones) to get consi-
    derable acceleration. If you want to do it I'm ready to answer your ques-
    tions about the library.
      Kernel library consists of the mozgVector and mozgMatrix class templates
    (with 7 + 7 member-functions), 32 optimised kernel functions templates
    and 5 aid functions. Those functions may be used independent from
    MLPerceptron as a library for vector and matrix operations. But it isn't
    universal vector and matrix library, because it has been created for
    requirements of the neurosimulator (Some of those will be used in
    following mozg versions). Explicit instantiation of those templates
    functions for float, int value types and int size type has been made in
    VectorMatrix.cc.
      If you want to use kernel library (for example, for your neurosimulator
    writing) then vectors and matrices dimensions coincidences checkings in
    some functions (see VectorMatrix.cc) is very usefull. To use them type

      ./configure --with-debug

    in time of the installation process.


3.3.2 Optimisation technique.

      1. I actively used address arithmetic, for example, in the function of
    out vector product computation instead of:

           matr_t** matr_prod = matrix->matr ();
           vec_t* temp1 = v1.vec (),
                * temp2;

           vec_size_t size1 = v1.size (),
                      size2 = v2.size (),
                      i;

           for (i=0; i<size1; i++)
             {
               temp2 = v2.vec ();

               for (j=0; j<size2; j++)
                 matr_prod[i][j] = temp1[i] * temp2[j];
             }

    I've written:

           matr_t** matr_prod = matrix->matr () [0];
           vec_t* temp1 = v1.vec (),
                * temp2;

           vec_size_t size1 = v1.size (),
                      size2 = v2.size (),
                      i;

           while (size1--)
             {
               temp2 = v2.vec ();
               i = size2;

               while (i--)
       	         *temp3++ = *temp1 * *temp2++;

               temp1++;
             }

    because I didn't want rely upon cleverness of compilator designers :).

      2. The method was used for vector-matrix product:

            - matrix is allocated in one string,

            - instead of:

                for(..i..) { 
                  for(..j..) {
                    result_vector[i] += vector[j] * matrix[j][i];
                  }
                }

              has been written:

                
                for(..i..) {
                  mattrix = matrix[0][i];
                  for(..j..) {
                    *result_vector += *vector++ * *mattrix;
                    mattrix += matrix_columns_num;
                  }
                }.

    matrix[j][i] is compiled to *(*(matrix+j)+i), that requires more CPU time 
    than matrix += matrix_columns_num, so we get considerable gain for inten-
    sive vector-matrix operations (in a neurosimulator, for example). If we
    should use transponed matrix it led to the extra memory consumption
    undesirable for big matrices.

*******************************************************************************

                    4. Bibliography
                   =================

    [1]	J.Freeman, D.Scapura; {\it Neural Network: algorithms, applications, 
programming techniques.} Addison-Wesley Publishing Company, 1991, --422 p.

    [2] B. Stroustrup {\it The C++ programmig language,} third edition,
Addison-Wesley Publishing Company, 1997, --9XX p.

    [3]	C.Peterson, Th.R\"{o}gnvaldsson; {\it An Introduction to Artificial 
Neural networks,} Department of Theoretical Phisics, University of Lund Sweden.

    [4]	C.Peterson, Th.R\"{o}gnvaldsson; {\it JETNET 3.0 --- A Versatile
Neural Network Package,} Department of Theoretical Phisics, University of Lund
Sweden.

    [5]	J.Proriol; {\it Multi-modular neural network for the classification of 
$e^+ e^-$ hadronic events;} NIMA 337 (1994).

    [6]	C.Svarer; {\it Neural Network for Signal Processing,} Ph.D. Thesis,
ph.d. nr. 91-0112-134, CONNECT, Electronic Institute, Technical University of
Denmark, 1995.

    [7]	A.Zell et. al.; {\it Stuttgart Neural Network Simulator: User manuale, 
Version 4.2;} University of Stutgart, Institute for parallel and distributed
high performance systems (IPVR); University of T\"{u}bingen, 
Wilhelm-Schickard-Institute for Computer Science.

    [8] http://world.std.com/~nr

    [9] ftp://ftp.sas.com/pub/neural/FAQ.html
