# TOKENINIT()

Initializes a token environment

Syntax

  TOKENINIT (<[@]cString>], [<cTokenizer>], [<nSkipWidth>],
  [<@cTokenEnvironment>]) -> lState

Arguments

  <[@]cString> is the processed string <cTokenizer> is a list of characters separating the tokens in <cString> Default: chr(0)+chr(9)+chr(10)+chr(13)+chr(26)+ chr(32)+chr(32)+chr(138)+chr(141)+ ",.;:!\?/\\<>()#&%+-*" <nSkipWidth> specifies the maximum number of successive tokenizing characters that are combined as ONE token stop, e.g. specifying 1 can yield to empty token Default: 0, any number of successive tokenizing characters are combined as ONE token stop <@cTokenEnvironment> is a token environment stored in a binary encoded string

Returns

  <lState> success of the initialization

Description

  The TOKENINIT() function initializes a token environment. A token environment is the information about how a string is to be tokenized. This information is created in the process of tokenization of the string <cString> - equal to the one used in the TOKEN() function with the help of the <cTokenizer> and <nSkipWidth> parameters.
  This token environment can be very useful when large strings have to be tokenized since the tokenization has to take place only once whereas the TOKEN() function must always start the tokenizing process from scratch.
  Unlike CTIII, this function provides two mechanisms of storing the resulting token environment. If a variable is passed by reference as 4th parameter, the token environment is stored in this variable, otherwise the global token environment is used. Do not modify the token environment string directly !
  Additionally, a counter is stored in the token environment, so that the tokens can successivly be obtained. This counter is first set to 1. When the TOKENINIT() function is called without a string a tokenize, the counter of either the global environment or the environment given by reference in the 4th parameter is rewind to 1.
  Additionally, unlike CTIII, tokeninit() does not need the string <cString> to be passed by reference, since one must provide the string in calls to TOKENNEXT() again.

Examples

          tokeninit (cString)   // tokenize the string  with default
                                // rules and store the token environment globally
                                // and eventually delete an old global TE
          tokeninit (@cString)  // no difference in result, but eventually faster,
                                // since the string must not be copied
          tokeninit()           // rewind counter of global TE to 1
          tokeninit ("1,2,3",",",1) // tokenize constant string, store in global TE
          tokeninit (cString,,1,@cTE1)  // tokenize cString and store TE in
                                        // cTE1 only without overriding global TE
          tokeninit (cString,,1,cTE1)  // tokenize cString and store TE in
                                       // GLOBAL TE since 4th parameter is
                                       // not given by reference !!!
          tokeninit (,,,@cTE1)         // set counter in TE stored in cTE1 to 1

Status

  Ready

Compliance

  TOKENINIT() is compatible with CTIII's TOKENINIT(), but there is an additional parameter featuring local token environments.

Platforms

  All

Files

  Source is token2.c, library is libct.

See Also

- TOKEN()
- TOKENEXIT()
- TOKENNEXT()
- TOKENNUM()
- TOKENAT()
- SAVETOKEN()
- RESTTOKEN()
- TOKENEND()