The Token Type |
Introduction
Header 'wave/context.hpp' synopsis
Template parameters
Member functions
The token type in Wave is the main carrier of information. It is returned by dereferencing the main iterator. The tokens are originally generated by the lexer ("An entity that lexically transforms the subject of parsing to a sequence of objects (called tokens) more suitable for subsequent parsing."). The Wave library contains two different, interchangable C++ lexers, which may be used as a starting point during developing your own application. The lexer generated tokens are transformed by the preprocessing engine (macro expansion, include file processing etc.) and after this returned to the user of the Wave library.
You can use arbitrary token types in conjunction with your own lexer as long as these implement some required interface. The required token type interface is described below and is implemented by the wave::cpplexer::lex_token template, the required lexer interface is described here.
In the following is described the token type predefined inside the Wave library, which is used in conjunction with the two predefined C++ lexers conatined in the Wave library too. If you need to use your own token type, it is required to implement the interafce described below.
namespace
wave { namespace cpplexer { template <typename PositionT> class lex_token { public: typedef std::basic_string
string_t; typedef PositionT position_t; lex_token(); lex_token(token_id id, string_t const &value, PositionT const &pos); // accessors operator token_id() const; string_t const &get_value() const; position_t const &get_position() const; void set_token_id (token_id id); void set_value (string_t const &newval); void set_position (position_t const &pos); }; } // namespace cpplexer } // namespace wave
The predefined token type uses a template parameter PositionT, which allows to specify the type to be used to carry the file position information contained inside the generated tokens. This type should contain at least the filename, the line number and the column number of the position, where the token was recognized. It defaults to a simple file_position template class described here.
lex_token(); lex_token(token_id id, string_t const &value, PositionT const &pos);
The first (default) constructor is for generating an end of stream token, which is used for indicating the end of the underlying input stream.
The second constructor initializes the newly created token object with its token id (for a list of valid token id's please look here), the string representation of its value and the file position, describing the position inside the input stream , where this token was recognized.
operator token_id() const;
Allows the access to the token id of the token. This access function is implemented as an implicit conversion function to allow the operation of Spirit parsers directly on top of a token stream. The possible token id's are described here.
This function does not throw in any case.
string_t const &get_value() const;
Returns the value of the token, as it was recognized in the input stream. Even for constant tokens (as keywords or operators etc.) the returned value reflects the character sequence as found in the input stream.
This function does not throw in any case.
PositionT const &get_position() const;
Returns the position of the token in the input stream, where it was recognized. The position contains information about the filename, the line number and the column number of the token. By default the Wave library uses a file_position template for this purpose, which is described in more detail here.
This function does not throw in any case.
void set_token_id(token_id id);
Changes the token id of the token to the new value. The possible token id's are described here. Propably this function is of little value for the library user, but it is required under certain circumstances for correct operation of the preprocessing engine.
This function does not throw in any case.
void set_value(string_t const &newval);
Changes the value stored inside the token to the new value. Propably this function is of little value for the library user, but it is required under certain circumstances for correct operation of the preprocessing engine.
void set_position(PositionT const &newpos);
Changes the position stored inside the token to the new value. This is used for instance for implementing the functionality required for to implement the #line directive.
Copyright © 2003 Hartmut Kaiser
Permission to copy, use, modify, sell and distribute this document
is granted provided this copyright notice appears in all copies. This document
is provided "as is" without express or implied warranty, and with
no claim as to its suitability for any purpose.