tokenize-lexical

text →
tokenize-lexical() → array

Returns lexical tokens in source order, preserving punctuation and symbols as separate tokens.

Parameters

This function has no parameters.

"Hello, David!" | tokenize-lexical → {"Hello", ",", "David", "!"}

Kind: Function
Scope: text/tokenization
Aliases: text-to-tokenize-lexical