Tokenization

Tokenization involves detection of white space characters and other symbols that separate words from each other and that are not relevant to the matching process. It is part of the linguistic analysis, where text is split into word entities. More complex tokenization is used for CJK languages, where semantic analysis is required to identify word boundaries. FAST ESP provides a highly configurable tokenization, which enables you to configure whether or not special characters like -‘./ shall be discarded, treated as white space or indexed as normal characters.