Asian language tokenization

Tokenization (word segmentation) for Asian languages requires special treatment. These languages do not allow text to be split into word entities by referring to whitespace or other separators. Asian language text needs to be split into tokens that can be treated as words during document processing and matching.