The Lexer Interface

Introduction
Header 'wave/cpplexer/cpp_lexer_interface.hpp' synopsis
Template parameters
Member functions

Introduction

The wave::cpplexer::lex_interface template is designed to be used as a base class for any C++ lexer, which should be plugged in into the Wave preprocessor engine. It is an abstract base class, which allows to abstract the lexer itself from the other parts of the library. Additionally it contains a static functions, through which a new instance of the lexer object should be instantiated. For this to function correctly you need to supply an implementation for the new_lexer_gen template, which actually is used to generate a new instance of your lexer object. This lexer object is expected to have a constructor taking three parameters: the begin and end iterators of the underlying input stream to analyse and the file_position of the point in the input stream, which corresponds to the first of the given iterators.

Header wave/cpplexer/cpp_lexer_interface.hpp synopsis

namespace wave {
namespace cpplexer {
 
    template <typename IteratorT, typename PositionT>
    struct new_lexer_gen
    {
        static lex_input_interface<lex_token<PositionT> > *
        new_lexer(IteratorT const &first, IteratorT const &last, 
            PositionT const &pos);
    };

    template <typename TokenT>
    struct lex_input_interface 
    {
        typedef typename TokenT::position_t position_t;
    
        virtual TokenT get() = 0;
        virtual void set_position(position_t const &pos) = 0;

        virtual ~lex_input_interface() {}

    //  The new_lexer function allows the opaque generation of 
    //  a new lexer object. It is coupled to the token type 
    //  to allow to distinguish different lexer/token 
    //  configurations at compile time.
        template <typename IteratorT>
        static lex_input_interface *
        new_lexer(IteratorT const &first, IteratorT const &last, 
            position_t const &pos)
        { 
            return new_lexer_gen<IteratorT, position_t>::
                new_lexer (first, last, pos); 
        }
    }
};

}   // namespace cpplexer
}   // namespace wave

Template parameters

The lex_input_interface itself needs the token type to be supplied as the only template parameter. It specifies the token type to be returned by the actual C++ lexer. The predefined C++ lexers inside the Wave library use the lex_token template (described here) for this purpose.

The new_lexer_gen structure needs to be implemented besides the actual lexer object. It takes two template parameters: the iterator type of the underlying input stream to use and the type of the file_position struct to be used by the C++ lexer. If the second parameter is omitted, it should default to the PositionT template parameter used by the token type.

Member functions

destructor

    virtual ~lex_input_interface();

The destructor is defined as virtual to garant the correct destruction of the derived lexers. This is very important to avoid memory leaks.

get

    virtual TokenT get();

This function is called, whenever the next token from the input stream is needed. It should return this token, if it is available otherwise the T_EOF token (see The Token Identifiers), signaling the end of the underlying input stream.

set_position

    virtual void set_position(
        position_t const &pos);

This function is called, whenever the current position of the input stream has to be changed. Note, that this function should not move the iterators of the input stream, it should instead store the new position and take it as the reference point for all subsequent generated tokens. For instance this function is called , whenever a valid #line directive is recognized in the input stream.

new_lexer_gen::new_lexer

    static lex_input_interface<TokenT> *
    new_lexer(IteratorT const &first, IteratorT const &last, 
            PositionT const &pos);

Copyright © 2003 Hartmut Kaiser

Permission to copy, use, modify, sell and distribute this document is granted provided this copyright notice appears in all copies. This document is provided "as is" without express or implied warranty, and with no claim as to its suitability for any purpose.