### mozg-0.4.3 README file

    Contents
    =========

      1. mozg Introduction.

      2. "Soft" rules.

      3. mozg Programmer's guide.

        3.1 MLPerceptron class (high level).

          3.1.1 High level functions.

          3.1.2 Code example of a network learning.

          3.1.3 Code example of a saved net using.

        3.2 MLPerceptron class (middle & low level).

          3.2.1 Middle level functions.

          3.2.2 Low level functions(layer interactions functions).

          3.2.3 How to insert other output function.

            3.2.3.1 For net = sum(w_{ij}*y_j).

            3.2.3.2 For RBF_net = -.5*sum(m_{ij}-y_j)^2/sigma_i^2.

          3.2.4 How to change basic types.

        3.3 Kernel library (mozgVector,mozgMatrix classes templates).

          3.3.1 Description.

          3.3.2 Optimisation technique.

      4. Bibliography.

    *****************************************************************************

                            1. mozg Introduction
                           ======================

        mozg is a flexible, fast neurolibrary. Up to the moment it allows one
        to create, learn and use multi-layer perceptron (MLP) [1,9], which is the
        most popular artificial neural network (ANN) for the large number of
        problems solved by a ANN.

          Structural parts of the mozg are:

            1. Kernel library is the library of fast vector and matrix operations.
        It consists of mozgVector and mozgMatrix class temlates [2] and functions
        templates using them (see mozgVectorMatrix.cc, mozgVectorMatrix.hh).
        Explanations of its appearance see in 3.3.1 (Description).

            2. MLPerceptron class (using the Kernel library) simulates the MLP and
        its behaviour. One can by the constructor create an ANN with desiring the
        number of layers, the number of neurons (called units) in them, output
        function in each layer (except input layer storing network's inputs),
        with/without bias term, learn, use, save and load a network
        (see MLPerceptron.cc, MLPerceptron.hh).

          Dynamic memory allocations have been placed in the constructor of MLP
        (and in putLParams (b)) and don't influence on the MLP learning and using
        CPU times. Constructors of vectors and matrices is called only in MLP con-
        structor and in putLParams (b), so there's no unnecessary vector & matrix
        constructor calls.

          Output of i-th neuron of a layer is computed as [1,3]:

            output_i = f(net_i) = f(sum_j(w_{ij}*input_j+theta_i)),

        where f(net_i) -- neuron's output function,
              net_i -- weighted inputs sum of i-th neuron
              w_{ij} -- weight for j-th input of i-th neuron,
              input_j -- j-th input of its,
              theta_i -- bias term or simply bias of its, for binary neuron it's
        called threshhold.

          Sometimes they use neuron's activation function:

            a_i(t) = g(a_i(t-1),net_i),

            output_i(t) = f(a_i(t)),

        to use dependensy of neuron's output from time. It hasn't been implemented
        in the simulator.

          The temperature (T) in the output function allows to change slope para-
        meter of the output function [3]:

            output_i = f(net_i/T).

        Its decreasing allows to sharpen error surface (called energy landscape)
        and conversly its increasing allows to soften the surface, so one can
        manage learning by it.

          For any layer (except input one storing network's inputs) neurons' output
        function may be choosed from (standard set):

            1. 1/(1+exp[-net/T]),
            2. th(net/T),
            3. net/T,
            4. Radial Basis Function --
               RBF = exp(-.5*sum(m_{ij}-y_j)^2/(sigma_i^2*T)),
            5. 2/Pi arctg(net/T),

        or desiring one may be inserted (see mozg programmer's guide). RBF is a
        "local" function and differs from the others by the "net", which isn't
        weighted input sum as for those. Due to it the learning rule changes
        considerably for the RBF layer and previous one. I've deduced and imple-
        mented in the simulator error backpropagation rule for three cases:

          a) RBF for hidden and output layer,

          b) RBF for hidden, sigmoid for output layer,

          c) sigmoid for hidden, RBF for output layer,

        that allow to use RBF in any combinations with sigmoid on different layers.
        Example of deducing error backpropagation rule for RBF for output and
        sigmoid for hidden layer see in [5]. Local RBF nature may lead to a
        "sharp sweep". It is the bad phenomenon caused by lack of learning examples
        or them bad "scattering" in the input space and clear shows itself on the
        function approximation problem. This trouble one can overcome if one put
        the RBF_sigma_min > 0 (large value results in the network learningless)
        or good scattering of learning examples.

          Implemented learning rule is the "vanilla" backpropagation extended by
        the momentum term [1,2,7], Langevin noise [4], weight decay [4,7]:

            w_{ij}(t+1) = w_{ij}(t) + delta w_{ij}(t) =
                        = w_{ij}(t) -
                          - eta * d E / d w_{ij} +
                          + alpha * delta w_{ij}(t-1) +
                          + Langevin_noise_term -
                          - decay_coef * w_{ij}(t).

          Any from three energy functions may be used in learning of a network:

            1) summed squared error E = sum_p sum_i (t_i^p-o_i)^2,

            2) Cross-Entropy E = sum_p sum_i ((1-t_i^p)*ln(1-o_i)-t_i^p*ln(o_i)),

            3) log-squared error E = - sum_p sum_i ln(1-(o_i-t_i^p)^2),

        Surely, they may be used in turn for learning of one network.

          Possibility to learn network with random order of learning examples
        using has been implemented in simulator, it improves network generaliza-
        tion (property of network to recognize examples not used in learning
        correctly) [4].

          Sometimes I had trouble with learning 1,2,...-hidden layers network
        with linear output function for all layers(it is meaningless, but for
        the sake interest). Learning lost stability and simulator crashed.
        I don't know why it occurs.

          Wasilx Urazmetow rendered noticeble assistance and can be named second
        mozg author.

          Please, send me lost bugs and suggestions related to the mozg to the
        e-mail filin@desert.ihep.su. Good luck!
                                                    Alexey P. Filin 19/09/99

    *******************************************************************************

                            2. "Soft" rules
                           =================

            1. Chosen network inputs and outputs influence on learning results
            considerably, for a bad case network can't be learned. There isn't
            guaranty of good learning result for the task at all. Your expe-
            rience is your advicer.

            2. Normalize inputs and outputs [6], i.e. make transformation

                new_input = ( old_input [- old_input_min] )
                              / | old_input_max - old_input_min |

            [- old_input_min] -- isn't certainly

            3. Divide your set of examples in two subset:
                 - learning set (is used for learning),
                 - test set (isn't used for learning)

            4. Try to use "vanilla" backpropagation rule [4] (without extentions)
            i.e. use function 2.a) (putLParams) for learning parameters settings
            then optimised weights adjustment is used.
               Momentum term and Langevin noise are used for "flat spot" elimi-
            nation in errors-weights space [7]. When error derivative is too small
            for learning by "vanilla" backpropagation, backpropagation rule exten-
            tions have to be used. Langevin_sigma != 0 leads to N-FOLD CPU time in-
            creasing, only use Langevin noise for your need.
               Weight decay [4,7] is used for network architecture optimisation, it
            decreases absolute weights' values. It may lead to bad learning result.
               Bias term may be very usefull sometimes.

            5. The number of learning examples is recomended be >> the number of
            network weights or ANN will not generalize [4], it will remember of
            learning examples with their noise.

            6. Usually three successive regions of test errors (errors for
            examples not being used in learning) exist in the learning phase [6]:
                 - test error decreasing,
                 - test error optimal value,
                 - test error increasing (overfitting).

      NOTE: error for learning examples usually decreases all learning time.
            Because of it test error checking is very usefull for learning result
            observation (see code example).

            7. Try to use different numbers of layers, numbers of hidden units,
            output functions. Learning results may be very different (That's
            life :).

    *******************************************************************************

                       3. mozg Programmer's guide
                      ============================

        Three levels of called functions may be extracted:

            * level of interface functions providing network interaction with
        outer objects and network management. They have access "public" in the
        MLPerceptron (except operator-functions ">>" and "<<").

            * level of member-functions of the MLPerceptron with access
        "protected" (except propLayer()). These functions provide interaction of
        neighbouring network layers,

            * low level(kernel library). Functions for vector and matrix opera-
        tions. Those functions use classes templates mozgVector and mozgMatrix.

          Let's go into details of those levels.

    3.1 MLPerceptron class (high level):
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    3.1.1 High level functions for MLPerceptron using are (see MLPerceptron.hh and
    MLPerceptron.cc):

    member-functions:

            1. MLPerceptron (int  number_of_layers,
                             int* num_outs_layers,
                             int* func_number,
                             float weight_dispersion,
                             bool bias_term_flag);

                  int number_of_layers -- number of layers (including input layer),
                    it's usually enough three layers,sometimes four,

                  int* num_outs_layers -- pointer to the massive wich contains
                    numbers of units at layers; if the bias term is used, numbers
                    of units in a layer MUST INCLUDE one extra unit -- bias unit,
                    which models bias input for the next layer, that is

                           units_number = real_units_number + 1,

                    EXCEPT output layer, which doesn't contain bias unit (because
                    the layer is last);

              NOTE: at the first layer the number of units also must include bias
                    unit, but network input vector dimension equals to the number
                    of layer units WITHOUT bias unit,

                  int* func_number -- pointer to the massive wich contains output
                     function numbers at layers. One can put it equal to:

                        1 -- 1/(1+exp[-net/T]),
                        2 -- th(net/T),
                        3 -- net/T,
                        4 -- Radial Basis Function,
                        5 -- 2/Pi arctg(net/T),
                        0 -- user defined function(see 3.2.3 of the text),
                        6 -- user defined function(the same).

                      For example, { 0, 4, 2}:
                        0 on the input layer (it may be any, because there's no
                          output function at the layer),
                        4 at the hidden layer,
                        2 at the output layer,

                  float weight_dispersion -- dispersion of begin weight distribu-
                    tion, bouth very small value and very large one leads to
                    network failed to be learned; try to use about 0.4;

                  bool bias_term_flag -- flag of bias term, true (1) for its switch
                    on and false (0) for its switch off; it is modeled by neuron's
                    input, which allways equals to one -- bias input; sometimes it
                    can be usefull, for example, for function approximation,
                    without the term network's outputs is zero for zero network's
                    inputs and odd neuron's output function (atg, th) so the net-
                    work can't approximate function not equaling to zero for all
                    zero inputs.

            create network, if RBF isn't used, then RBF_sigma_min doesn't
            influence on computations,

            2.a) putLParams (float learning_rate,
                             float temperature,
                             float RBF_sigma_minimum
    		         int energy_function);

                    float learning_rate -- common learning rate, usually value
                        about 0.3 is a good one. Learning rate for each layer is
                        computed as (rule of fan-in):

                           layer's_eta = learning_rate / layer's_inputs_number

                    float temperature -- temperature in the output function, very
                        small value makes very sharp energy landscape, so learning
                        can be lost in local minima, very large one makes that
                        too sloping and network is learned too long.
                        at first try to use 1,

                    float RBF_sigma_minimum -- allowed RBF sigma minimum in the
                        learning phase, if RBF isn't used, it doesn't influence
                        on computations (at first try to use zero);

                    int energy_function -- you can use following energy function:

                          1 -- summed squared error:
                             E = sum_p sum_i (t_i^p-o_i)^2,

                          2 -- Cross-Entropy(don't use it for odd sigmoids):
                             E = sum_p sum_i (t_i^p*ln(o_i) +
                                             + (1-t_i^p)*ln(1-o_i)),

                          3 -- log-squared error:
                             E = - sum_p sum_i ln(1-(o_i-t_i^p)^2);

              put learning parameters for "vanilla" backpropagation rule,

              b) putLParams (float learning_rate,
                             float temperature,
                             float RBF_sigma_minimum,
    		         int energy_function,
    		         float mom_term_coef,
    		         float lang_noise_begin_dispersion,
    		         float lang_noise_disp_decr_coef,
    		         float weight_decay_coef);

                    float mom_term_coef -- momentum term coefficient(alpha) in
                        weight changing rule:

                        w_{ij}(t+1) = w_{ij}(t) + delta w_{ij}(t) =
                                    = w_{ij}(t) + ... + alpha * delta w_{ij}(t-1)

                        is recomended to be in interval 0...1.

    	        float lang_noise_begin_dispersion begin value of the disper-
                        sion of the Langevin noise(Gauss noise) term in a weight
                        changing rule:

                        w_{ij}(t+1) = w_{ij}(t) + ... + noise_term

                        It must be sufficiently small for successful learning.

    	        float lang_noise_disp_decr_coef -- decreasing coefficient of
                        the Langevin dispersion in learning phase:

                        Langevin_sigma(t+1) =
                              = Langevin_sigma(t) * lang_noise_disp_decr_coef

                        where t+1 -- next call of the AdjustWeights (). It must be
                        putted approximatly 0.99.

    	        float weight_decay_coef -- weight decay coefficient for decay
                        term in a weight changing rule:

                        w_{ij}(t+1) = w_{ij}(t) + ... -
                                    - weight_decay * w_{ij}(t).

                        It must be << 1, then a network can be learned(for a
                        normalized inputs);

            put learning parameters for backpropagation with extensions rule,

        ALL RECOMENDED VALUES INTEND FOR NORMALIZED INPUTS AND OUTPUTS (see "Soft"
        rules).

      NOTE: you can don't call the function then default values is used for lear-
            ning:

               layer[i].eta = 0.3 / num_outputs_layers[i];
               layer[i].temperature = 1.;
               RBF_sigma_min             = 0.;
               switch_of_energy_function = 1;
               alpha                     = 0.;
               sigma_Langevin            = 0.;
               sigma_Langevin_decrease   = 0.;
               decay                     = 0.;
               flag_of_optimisation      = true;
               error_threshold           = 0.;
               rate_coeff = 1.;
               temperature_coeff = 1.;

            3.a) learnNet (float* linputs,   // learning input vector
                           float* loutputs)  // learning output vector

              learn net for the learning example (one error backpropagation), by
              the function you can self choose order of learning examples in epochs
              what usefull if you want to use them in random order (it assists
              improve a network's generalization)

              b) learnNet (float** linputs,  // array of learning input vectors
                           float** loutputs, // array of learning output vectors
                           int lssize,       // learning set size
    		       int vec_order,    // order of vectors used in learning
                           int epnum,        // number of epochs network must be
                                             // learned
                           int pepnum)       // answerMessage() (9.) is called for
                                             // each pepnum-th epoch

              learn network:

    	    1) vec_order = 0 -- in random order of learning examples (regular
                   distribution is used), that helps to improve network genera-
                   lisation; in the case some examples will be used larger and some
                   less number of times, but common number of error backpropagation
                   is equal to one in a);

                2) vec_order = 1 -- in successive order of learning examples given
                   by the array, one error backpropagation for each learning
                   example in epoch is used,

              each pepnum-th epoch learning error is printed.

        NOTE: learning error isn't the mean squared error on learning set,
              it is the mean of output network's error squares for learning set:

                   learning_error = sum_i(target_i-realouput_i)^2/lssize

              where target_i -- i-th target output vector,
                    realoutput -- i-th real output vector
              and each sum's term is calculated after a weight change,

              c) learnNet (float** linputs,  // array of learning input vectors
                           float** loutputs, // array of learning output vectors
                           int lssize,       // learning set size
    		       int vec_order,    // order of vectors used in learning
                           int epnum,        // number of epochs network must be
                                             // learned
                           int pepnum,       // answerMessage() (9.) is called for
                                             // each pepnum-th epoch
                           float** tinputs,  // array of test input vectors
                           float** toutputs, // array of test output vectors
                           int tssize);      // test set size

              learn network (one error backpropagation for each learning example),
              each pepnum-th epoch the learning error and the test error is prin-
              ted.

        NOTE: test error is real mean squared error on test set:

                     test_error = (sum_i(target_i-realoutput_i)^2)/tssize ,

              where i (= 1...tssize) -- the number of test examples,
                    target_i -- i-th target output vector,
                    realoutput -- i-th real output vector,

            4. testNet (float** tinp, // array of test input vectors
                        float** tout, // array of test output vectors
                        int tssize);  // test set size

              test network and return test error.

        NOTE: test error is real mean squared error on test set,

            5.a) propInputs (float* invec,   // input vector
                             float* outvec); // output vector(allocated by
                                             // programmer)
              propagate input vector invec through network and place output vector
              in outvec

              b) propInputs (float** invecarr,  // input vector massive
                             float** outvecarr, // output vector massive (allocated
                                                // by programmer)
                             int ssize);        // vectors number
              propagate input vector massive through network and place output
              vector massive in outvecarr,

    not member-functions:

            6. output << net
               ofstream& output -- output file stream,
               MLPerceptron& net -- network to be saved,
            save the network in the file,

            7. input >> p_p_net
                ifstream& input -- input file stream,
                MLPerceptron** p_p_net -- address of the pointer to the network,

            load the network from the file,

      NOTE: network is loaded to the dynamic memory wich is allocated in the
            operator-function, so you must self destroy the network if it isn't
            need

            8. output << net
                ostream& output -- output stream (cout),
                MLPerceptron& net -- network to be writed in the output,

            show the network.

    PLEASE DEFINE IT:

            9. bool answerMessage(float learn_error, // learning error
    			      float test_error); // test error

         The function is intended for error checking and management of learning
       "on fly". It is called by learnNet b),c) each pepnum-th epoch and is got
       learning error and test error if it is calculated. If test error isn't cal-
       culated (for learnNet b)) zero is put in second signature parameter of the
       function. If you like graphics use a graphic library then you can admire
       learning in real time.
         The function MUST RETURN:
           - true if network learning must be ended (error is small enough), then
       return from learnNet() takes place,
           - and false in the other case (continue learning).
         A simple example:

            bool MLPerceptron::answerMessage(float learn_error,
            				 float test_error)
            {
              cout << "Overall number of epoch since learning begin: "
            // overall_epnum -- member-datum of MLPerceptron class (float)
                   << overall_epnum << endl
                   << "  learning error: " << learn_error << endl
                   << "  test error    : " << test_error << endl;

            // is_call_aM -- member-datum of MLPerceptron class (bool);
            // to enter test error once
              if (!is_call_aM) {
                is_call_aM = true;
                cout << "Enter test error which can be allowed: ";

            // error_threshold -- member-datum of MLPerceptron class (float)
                cin >> error_threshold;
              }
              return (test_error < error_threshold);
            }

        I don't use static-values in answerMessage() instead of member-data
        overall_epnum, is_call_aM and error_threshold, because I had troubles
        with them. When I had made several networks, static-values was common.
        Please, don't forget it, if you want to use static-value.
          In learning process you can change some learning parameters, learning
        rate or temperature for each layer except input one, for it you can write
        in answerMessage():

              if (!is_call_aM) {

                is_call_aM = true;
                cout << "Enter rate_coeff and temperature_coeff:";
                cin >> rate_coeff >> temperature_coeff;
              }

              for (int i=1; i<num_layers; i++) {

                layers[i].rate *= rate_coeff;
                layers[i].temperature *= temperature_coeff;
              }

      NOTE: you can continue learning of the network by the other call of learnNet
            (any from a),b),c)) then overall_epnum DOESN'T be zeroed and equals to
            overall number of epochs for several call of learnNet().

    3.1.2 Code example of a network learning:

            ...

            float** learning_set_inputs,
                 ** learning_set_outs,
                 ** test_set_inputs,
                 ** test_set_outs,
                    eta,
                    begin_wts_sigma,
                    learning_rate,
                    temperature,
                    RBF_sigma_minimum;

            int   number_layers,
                * num_outputs_at_layers,
                * num_output_f,
                  learning_set_size,
                  vec_order,
                  energy_function;

            bool flag_of_bias_term;

            // Initialisations of inputs & outputs vector arrays
            ...

            // Enter network and its learning parameters
            params_input (...);

            // Network is created:
            MLPerceptron net (number_layers,
                              num_outputs_at_layers,
                              num_output_f,
                              begin_wts_sigma,
                              flag_of_bias_term);

            // Learning parameters initialisation:
            net.putLParams (learning_rate,
                            temperature,
                            RBF_sigma_minimum,
    		        energy_function);

            // Network is learned:
            net.learnNet (learning_set_inputs,
                          learning_set_outputs,
                          learning_set_size,
                          vec_order,
                          epoch_num,
                          print_epoch_num,
                          test_set_inputs,
                          test_set_outs,
                          test_set_size);

            // and is saved:
            bool flag_of_saving;
            char net_file_name[100];

            cout << "Save network? (yes-1, no-0)";
            cin >> flag_of_saving;

            if (flag_of_saving) {

              cout << "Enter file name network must be saved in: ";
              cin >> net_file_name;

              ofstream out_file (net_file_name);
              out_file << net;
            }
            ...

    3.1.3 Code example of a saved net using:

            float* set_inputs,
                 * input_vec,
                 * output_vec;
            ...

            MLPerceptron* p_net;

            ifstream from_file (net_file_name);

            from_file >> &p_net;

            // I want to see loaded net...
            cout << *p_net;

            output_vec = new float[num_outs];

            p_net->propInputs (input_vec,
                               output_vec);
            ...

    3.2 MLPerceptron class (middle & low level).
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    3.2.1 Middle level functions:

            10. putInputs (inputs)
                float* inputs -- pointer to begin of the massive containing network
                    inputs,

            put the inputs in the network input layer,

            11. compOutputErrors (target)
               float* target -- pointer to begin of the desiring outputs massive,

            compute network out errors and place square of a error vector module
            in (t_i-y_i)^2 the error_over_outs,

            12. backpropErrors ()

            propagate out errors back through the network,

            13. adjustWeights ()

            adjust network weights,

            14. getOutputs(net_outputs)
               float* net_outputs -- pointer to massive output vector had been
                                    placed in,

            put output layer outputs to the net_outputs (which allocated by
            programmer),

    3.2.2 Low level(layer interactions functions):

            1. propLayer (...), DON'T FORGET BIAS TERM if you use
            the function (for example, see definition of learnNet).

            propagate signal through a layer,

    following functions have access "protected":

            2. compute_lay_outs (...)

            compute outs of a hidden layer,

            3. compute_output_errors (...)

            compute errors of the out layer,

            4. compute_lay_errors (...)

            compute errors of a hidden layer,

            5. adjust_lay_weights (...)

            update weights of neurons of a layer.

    3.2.3 How to insert other output function.

        All you need to insert other output function is to write in MLPOutFunc.cc
        desiring function, its derivative  and recompile mozg. In the following I
        consider two case, the case of weighted inputs net and the case of RBF net.

            1. For weighted input sum, net = sum(w_{ij}*y_j) you must write
    	desiring output function in mozgweinet(net,sigm) and its derivative in
        	mozgDweinet(out,net,sigm) (see MLPOutFunc.cc).

          	For example, you want to insert y = sin(net), then you must write in
        	mozgweinet (net,sigm):

    	  mozgflt mozgweinet (mozgflt net,mozgflt sigm)
    	  {
      	    return sin(net);
    	  }

        	Then, because of y' = cos(x) = sqrt(1-y^2), you must write in the
        	mozgDweinet (out,net,sigm):

    	  mozgflt mozgDweinet (mozgflt out,mozgflt net,mozgflt sigm)
    	  {
      	    return cos(net);
    	  }
        	or (for some functions using of "out" instead of "net" is more
            convenient)

    	  mozgflt mozgDweinet (mozgflt out,mozgflt net,mozgflt sigm)
    	  {
      	    return sqrt(1 - out * out);
    	  }

      NOTE: the net is neuron's net devided by temperature yet.

            2. If you want to use RBFnet = -.5*sum(m_{ij}-y_j)^2/sigma_i^2, for
         	example y = sin(RBFnet), you must write in mozgrbfnet (RBFnet,sigm):

    	  mozgflt mozgrbfnet (mozgflt RBFnet,mozgflt sigm)
    	  {
     	    return sin(-.5 * RBFnet / (sigm * sigm));
    	  }

        	and, because of y' = cos(RBFnet) = sqrt(1-y^2), you must write in the
        	mozgDrbfnet (out,RBFnet,sigm):

    	  mozgflt mozgDrbfnet (mozgflt out,mozgflt RBFnet,mozgflt sigm)
    	  {
      	    return cos(-.5 * RBFnet / (sigm * sigm)) / (sigm * sigm);
    	  }

        	or(for some functions using of out instead of RBFnet is more
    	convenient)

    	  mozgflt mozgDrbfnet (mozgflt out,mozgflt RBFnet,mozgflt sigm)
    	  {
      	    return sqrt(1 - out * out) / (sigm * sigm);
    	  }

      NOTE: the RBFnet is neuron's RBFnet multiplied by gain (slope parameter) yet.

        For frequent change of output function it is better to extract
        MLPOutFunc.cc from the library only to recompile it after each change of
        output function.

    3.2.4 How to change basic types.

        If you want to change basic value type "float" by "double" (for example for
        precision increasing) and value type "int" by "long", you need to replace
        "float" by "double" and "int" by long in "types.hh" and recompile mozg.
        Then ALL mozg's "float" values are replaced by "double" values and ALL
        "int" values are replaced by "long" values.

    3.3 Kernel library(mozgVector,mozgMatrix classes templates).
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    3.3.1 Description.

        Extracting of vector and matrix operations from neurosimulator is conse-
        quence of ANN model, that has been described in terms of vector and matrix
        operations. Calculations with vectors and matrices nearly occupy all time
        of a neurosimulator's run only leave some percents or even some shares of
        percent for rest, so to extract them in separate library is very conve-
        nient for further acceleration of the neurosimulator, because you only
        need to change some library's functions by faster ones (for example,
        by written in assembler or even hard implemented ones) to get consi-
        derable acceleration. If you want to do it I'm ready to answer your ques-
        tions about the library.
          Kernel library consists of the mozgVector and mozgMatrix class templates
        (with 7 + 7 member-functions), 32 optimised kernel functions templates
        and 5 aid functions. Those functions may be used independent from
        MLPerceptron as a library for vector and matrix operations. But it isn't
        universal vector and matrix library, because it has been created for
        requirements of the neurosimulator (Some of those will be used in
        following mozg versions). Explicit instantiation of those templates
        functions for float, int value types and int size type has been made in
        VectorMatrix.cc.
          If you want to use kernel library (for example, for your neurosimulator
        writing) then vectors and matrices dimensions coincidences checkings in
        some functions (see VectorMatrix.cc) is very usefull. To use them type

          ./configure --with-debug

        in time of the installation process.

    3.3.2 Optimisation technique.

          1. I actively used address arithmetic, for example, in the function of
        out vector product computation instead of:

               matr_t** matr_prod = matrix->matr ();
               vec_t* temp1 = v1.vec (),
                    * temp2;

               vec_size_t size1 = v1.size (),
                          size2 = v2.size (),
                          i;

               for (i=0; i<size1; i++)
                 {
                   temp2 = v2.vec ();

                   for (j=0; j<size2; j++)
                     matr_prod[i][j] = temp1[i] * temp2[j];
                 }

        I've written:

               matr_t** matr_prod = matrix->matr () [0];
               vec_t* temp1 = v1.vec (),
                    * temp2;

               vec_size_t size1 = v1.size (),
                          size2 = v2.size (),
                          i;

               while (size1--)
                 {
                   temp2 = v2.vec ();
                   i = size2;

                   while (i--)
           	         *temp3++ = *temp1 * *temp2++;

                   temp1++;
                 }

        because I didn't want rely upon cleverness of compilator designers :).

          2. The method was used for vector-matrix product:

                - matrix is allocated in one string,

                - instead of:

                    for(..i..) {
                      for(..j..) {
                        result_vector[i] += vector[j] * matrix[j][i];
                      }
                    }

                  has been written:

                    for(..i..) {
                      mattrix = matrix[0][i];
                      for(..j..) {
                        *result_vector += *vector++ * *mattrix;
                        mattrix += matrix_columns_num;
                      }
                    }.

        matrix[j][i] is compiled to *(*(matrix+j)+i), that requires more CPU time
        than matrix += matrix_columns_num, so we get considerable gain for inten-
        sive vector-matrix operations (in a neurosimulator, for example). If we
        should use transponed matrix it led to the extra memory consumption
        undesirable for big matrices.

    *******************************************************************************

                        4. Bibliography
                       =================

        [1]	J.Freeman, D.Scapura; {\it Neural Network: algorithms, applications,
    programming techniques.} Addison-Wesley Publishing Company, 1991, --422 p.

        [2] B. Stroustrup {\it The C++ programmig language,} third edition,
    Addison-Wesley Publishing Company, 1997, --9XX p.

        [3]	C.Peterson, Th.R\"{o}gnvaldsson; {\it An Introduction to Artificial
    Neural networks,} Department of Theoretical Phisics, University of Lund Sweden.

        [4]	C.Peterson, Th.R\"{o}gnvaldsson; {\it JETNET 3.0 --- A Versatile
    Neural Network Package,} Department of Theoretical Phisics, University of Lund
    Sweden.

        [5]	J.Proriol; {\it Multi-modular neural network for the classification of
    $e^+ e^-$ hadronic events;} NIMA 337 (1994).

        [6]	C.Svarer; {\it Neural Network for Signal Processing,} Ph.D. Thesis,
    ph.d. nr. 91-0112-134, CONNECT, Electronic Institute, Technical University of
    Denmark, 1995.

        [7]	A.Zell et. al.; {\it Stuttgart Neural Network Simulator: User manuale,
    Version 4.2;} University of Stutgart, Institute for parallel and distributed
    high performance systems (IPVR); University of T\"{u}bingen,
    Wilhelm-Schickard-Institute for Computer Science.

        [8] http://world.std.com/~nr

        [9] ftp://ftp.sas.com/pub/neural/FAQ.html