Tokenization functions

Reference documentation for Expressif functions in the text/tokenization scope.

Name	Overview
token	Returns the token at the specified index in the argument value. The index of the first token is 0, the second token is 1, and so on. By default, the tokenization is executed based on any white-space characters. If a character is specified then the tokenization is executed based on this character to separate two tokens.
token-count	Returns the count of token within the argument value. By default, the tokenization is executed based on any white-space characters. If a character is specified then the tokenization is executed based on this character to separate two tokens.
token-count-lexical	Returns the number of lexical tokens in the argument value, including punctuation and symbols.
tokenize	Returns all tokens in the argument value in source order. By default, tokenization uses white-space characters as delimiters. If a character is specified, that character delimits the tokens.
tokenize-camel	Returns tokens from a camelCase name using case and acronym transitions as boundaries.
tokenize-kebab	Returns normalized tokens from a hyphen-separated name, preserving escaped hyphens within tokens.
tokenize-lexical	Returns lexical tokens in source order, preserving punctuation and symbols as separate tokens.
tokenize-lines	Returns lines in source order, recognizing CR, LF, and CRLF as separators. Preserves spaces and empty lines, including a final empty line after a trailing separator. Returns an empty array for null or empty input.
tokenize-pascal	Returns tokens from a PascalCase name using case and acronym transitions as boundaries.
tokenize-regex	Returns segments separated by regular expression matches in source order. Preserves spaces and empty segments without including captured delimiters. Returns an empty array for null or empty input.
tokenize-snake	Returns normalized tokens from an underscore-separated name, preserving escaped underscores within tokens.
tokenize-words	Returns word tokens using separators, punctuation, symbols, case transitions, and acronym transitions as boundaries.