tokenizing 1 topic · 1 source Technology› ProgrammingPerlSplitting up a program text into tokens. Also known as "lexing", in which case you get "lexemes" instead of tokens.Perl — Perl Glossary Cite