class Poco::StreamTokenizer

Overview

A stream tokenizer splits an input stream into a sequence of tokens of different kinds. More…

#include <StreamTokenizer.h>

class StreamTokenizer
{
public:
    // structs

    struct TokenInfo;

    // construction

    StreamTokenizer();
    StreamTokenizer(std::istream& istr);

    // methods

    void
    attachToStream(std::istream& istr);

    void
    addToken(Token* pToken);

    void
    addToken(
        Token* pToken,
        bool ignore
        );

    const Token*
    next();
};

Detailed Documentation

A stream tokenizer splits an input stream into a sequence of tokens of different kinds.

Various token kinds can be registered with the tokenizer.

Construction

StreamTokenizer()

Creates a StreamTokenizer with no attached stream.

StreamTokenizer(std::istream& istr)

Creates a StreamTokenizer with no attached stream.

Methods

void
attachToStream(std::istream& istr)

Attaches the tokenizer to an input stream.

void
addToken(Token* pToken)

Adds a token class to the tokenizer.

The tokenizer takes ownership of the token and deletes it when no longer needed. Comment and whitespace tokens will be marked as ignorable, which means that next() will not return them.

void
addToken(
    Token* pToken,
    bool ignore
    )

Adds a token class to the tokenizer.

The tokenizer takes ownership of the token and deletes it when no longer needed. If ignore is true, the token will be marked as ignorable, which means that next() will not return it.

const Token*
next()

Extracts the next token from the input stream.

Returns a pointer to an EOFToken if there are no more characters to read. Returns a pointer to an InvalidToken if an invalid character is encountered. If a token is marked as ignorable, it will not be returned, and the next token will be examined. Never returns a NULL pointer. You must not delete the token returned by next().