Class Tokenizer

    • Constructor Detail

      • Tokenizer

        public Tokenizer​(CharSequence text)
        Create a Tokenizer for the specified text.
        Parameters:
        text -
    • Method Detail

      • parse

        public static List<Token> parse​(CharSequence text)
        Convenience method to parse text into tokens. This method will throw an exception if the text is invalid. If you need a partial token sequence, use the iterator directly.
        Parameters:
        text - text to parse
        Returns:
        list of tokens
        Throws:
        InvalidSyntaxException - if the text is invalid