{"id":567,"date":"2009-11-20T17:31:30","date_gmt":"2009-11-21T01:31:30","guid":{"rendered":"http:\/\/boost-spirit.com\/home\/?page_id=567"},"modified":"2010-08-18T15:07:27","modified_gmt":"2010-08-18T22:07:27","slug":"creating-your-own-parser-component-for-spirit-qi","status":"publish","type":"page","link":"http:\/\/boost-spirit.com\/home\/articles\/qi-example\/creating-your-own-parser-component-for-spirit-qi\/","title":{"rendered":"Creating Your Own Parser Component for Spirit.Qi"},"content":{"rendered":"

Several people have been asking the question of how it would be possible to access the current iterator position from an semantic action. Different solutions have been proposed, all of them somehow abusing the predefined Qi<\/em> directive raw[]<\/a><\/span> which normally exposes as its attribute the pair of iterators pointing to the range in the input stream matched by the embedded parser. I thought this to be a nice incentive to write about how you can create your own parser components.<\/p>\n

Spirit<\/em> is a very modular library. It has been designed with full extensibility in mind, so writing additional components is fairly straight forward. Let\u2019s write a parser component for Qi<\/em> called iter_pos<\/span> which exposes the current iterator position as its attribute. In the end we will have implemented a parser primitive usable in the same way as any of the predefined parsers, nicely integrated with Qi\u2019s<\/em> predefined attribute propagation and merging rules.<\/p>\n

The new parser primitive will have the following expression semantics: iter_pos<\/span> will create a new parser which never matches anything while always succeeding (very much like the eps<\/a><\/span> parser). It\u2019s attribute type is iterator_type<\/span>, the type of the iterator used to access the underlying input stream.<\/p>\n

In order to create a custom parser component we need to perform 4 easy steps as outlined below.<\/p>\n

Defining the Placeholder<\/h4>\n

The first step is to define the placeholder representing our new component when building parser expressions (that\u2019s the iter_pos<\/span> symbol). This can<\/span> can be done by using the predefined macro BOOST_SPIRIT_TERMINAL:<\/p>\n

 namespace custom_parser { BOOST_SPIRIT_TERMINAL(iter_pos); } <\/pre>\n
which can be placed in any namespace. We assume our custom parser component will be created in the namespace custom_parser<\/span>. This macro defines two types and a const<\/span> instance of one of the types. It essentially expands to:<\/p>\n
\r\nnamespace custom_parser\r\n{\r\n    namespace tag { struct iter_pos{}; } \/\/ tag identifying placeholder\r\n    typedef unspecified<tag::iter_pos> iter_pos_type;\r\n    iter_pos_type const iter_pos = {};   \/\/ placeholder itself\r\n}\r\n<\/pre>\n
We will use the type tag::iter_pos<\/span> later to identify our component, whereas the variable iter_pos<\/span> is the one to be used as the placeholder representing our parser component in more complex grammars.<\/p>\n
Implementing the Enabler<\/h4>\n
The placeholder needs to be associated with the appropriate extension mechanism (Spirit<\/em> has separate extension points for primitive parsers, parser directives, operators, etc.). We need to implement an enabler for the custom parser in a way allowing the library to recognize iter_pos<\/span> as a simple terminal only, as we don\u2019t want it to be valid if used as a directive, etc. The extension point used for simple terminals is boost::spirit::use_terminal<><\/span>. We enable our custom parser by providing a specialization of this extension point, which has to be placed in the namespace boost::spirit<\/span>:<\/p>\n
\r\nnamespace boost { namespace spirit\r\n{\r\n    \/\/ We want custom_parser::iter_pos to be usable as a terminal only,\r\n    \/\/ and only for parser expressions (qi::domain).\r\n    template <>\r\n    struct use_terminal<qi::domain, custom_parser::tag::iter_pos>\r\n      : mpl::true_\r\n    {};\r\n}}\r\n<\/pre>\n
Spirit<\/em> will pick up this template specialization whenever it sees our placeholder custom_parser::iter_pos<\/span> while compiling a parser expression (that\u2019s why the qi::domain<\/span>).<\/p>\n
Implementing the Parser itself<\/h4>\n
So far, everything we saw was scaffolding allowing to integrate our new parser with the component framework of Spirit<\/em>. This step describes the actual interface to be implemented in order to expose the required functionality as a parser component. Here is the full code, please note we are placing this code into our own namespace:<\/p>\n
\r\nnamespace custom_parser\r\n{\r\n    struct iter_pos_parser\r\n      : boost::spirit::qi::primitive_parser<iter_pos_parser>\r\n    {\r\n        \/\/ Define the attribute type exposed by this parser component\r\n        template <typename Context, typename Iterator>\r\n        struct attribute\r\n        {\r\n            typedef Iterator type;\r\n        };\r\n\r\n        \/\/ This function is called during the actual parsing process\r\n        template <typename Iterator, typename Context\r\n          , typename Skipper, typename Attribute>\r\n        bool parse(Iterator& first, Iterator const& last\r\n          , Context&, Skipper const& skipper, Attribute& attr) const\r\n        {\r\n            boost::spirit::qi::skip_over(first, last, skipper);\r\n            boost::spirit::traits::assign_to(first, attr);\r\n            return true;\r\n        }\r\n\r\n        \/\/ This function is called during error handling to create\r\n        \/\/ a human readable string for the error context.\r\n        template <typename Context>\r\n        boost::spirit::info what(Context&) const\r\n        {\r\n            return boost::spirit::info("iter_pos");\r\n        }\r\n    };\r\n}\r\n<\/pre>\n
The code above shows 4 notable details any parser component needs to implement, all of them are based on conceptual requirements for parsers (for more information see the Spirit documentation about this here<\/a>).<\/p>\n
We derive our parser implementation from boost::spirit::qi::primitive_parser<><\/span> to associate our component with the correct parser concept, in this case a PrimitiveParser<\/a><\/span>.<\/p>\n
The embedded meta function attribute<\/span> is a template which will be instantiated with a Context<\/span> type (we can ignore this for our small example) and with the type of the Iterator<\/span> the parser component is\u00a0being used with. It needs to have defined an embedded type definition type<\/span>. This will be interpreted as the attribute type exposed by the parser component. Obviously, we expose the supplied Iterator<\/span> type as our attribute.<\/p>\n
The member function parse()<\/span> is where the actual parsing takes place. It will be invoked with a pair of iterators (first, last)<\/span>, an instance of the Context (which we will ignore), the reference to the used skipper\u00a0instance (skipper)<\/span>, and the reference to the attribute instance where the parser is supposed to store its result (attr)<\/span>. If no skipper is used to invoke this parser the Skipper<\/span> will be of the type unused_type<\/span>. Similarily, if no attribute needs to be extracted the Attribute<\/span> will be of the type unused_type<\/span>. As we are designing a parser representing the PrimitiveParser<\/a><\/span> concept we need to perform a pre-skip operation on function entry by calling the function qi::skip_over<\/span>. Our iter_pos<\/span> parser is not performing any real parsing, but is supposed to return the current iterator position as it\u2019s attribute. We achieve this by assigning the iterator first<\/span> (which pints to the current position in the input stream) to the attribute. For this assignment to take place we invoke the customization point assign_to<\/span><\/a>, allowing us not to worry about whether\u00a0an attribute has been supplied or not. The parse function needs to return whether it succeeded, in our case it will always return true<\/span>.<\/p>\n
The function what()<\/span> is invoked by the library whenever a human readable string is needed identifying this parser. This is\u00a0most notably used for error handling, allowing to generate a nicely formatted description about the error context.<\/p>\n
This part is clearly the most complex step required to write a parser component, but I believe it is not too complex to understand. As mentioned above, Spirit<\/em> has been designed with extensibility in mind, and we wanted to make it as simple as possible to be extended.<\/p>\n
Instantiating the Parser<\/h4>\n
The last required piece of code is a parser generator function object which will be used by the library to instantiate a new instance of our parser component. Non-surprisingly, the name of the function object we need to specialize is make_primitive<\/span>, and since we were using qi::domain<\/span> in the first step above, we now need to place this specialization into the namespace boost::spirit::qi<\/span>.<\/p>\n
namespace boost { namespace spirit { namespace qi\r\n{\r\n    \/\/ This is the factory function object invoked in order to create\r\n    \/\/ an instance of our iter_pos_parser.\r\n    template <typename Modifiers>\r\n    struct make_primitive<custom_parser::tag::iter_pos, Modifiers>\r\n    {\r\n        typedef custom_parser::iter_pos_parser result_type;\r\n\r\n        result_type operator()(unused_type, unused_type) const\r\n        {\r\n            return result_type();\r\n        }\r\n    };\r\n}}}\r\n<\/pre>\n
You can think of this function object as of a factory for our parser object. Our specialization is again based on the tag::iter_pos<\/span> as defined above. This identifies our parser component. The function object make_primitive<\/span> has to expose the type of the component it creates as its embedded type definition result_type<\/span>. Additionally it exposes a function operator as the actual factory function. You probably already realized that unused_type<\/span> is Spirits<\/em> fancy way of saying: \u2018I don\u2019t care\u2019, and as we don\u2019t care about the two (required) parameters we use unused_type<\/span> instead (ok, if you insist to know the details: the first parameter is a reference to the iter_pos<\/span> placeholder instance resulting in this factory being invoked, and the second parameter is a reference to an object instance of the type Modifiers<\/span>, which is needed only for directives like no_case[]<\/a>).<\/p>\n
Using the iter_pos<\/span> Parser Component<\/h4>\n
The only things left to show is how to use our newly created parser. As I mentioned before, the iter_pos<\/span> component is usable like any other predefined parser.<\/p>\n
\r\n    std::string prefix, suffix;           \/\/ attributes receiving the\r\n    std::string::iterator position;       \/\/ parsed values\r\n\r\n    std::string input("prefix1234567");\r\n    std::string::iterator first = input.begin();\r\n    bool result =\r\n        qi::parse(first, input.end()\r\n          , +qi::alpha >> custom_parser::iter_pos >> +qi::digit\r\n          , prefix, position, suffix);\r\n<\/pre>\n
This code snippet utilizes the new parser component to retrieve the position in the input at which the letters end and the digits start. If the parse is successful, prefix<\/span> will hold \u201cprefix\u201d<\/span>, suffix<\/span> will hold \u201c1234567\u201d<\/span>, and position<\/span> will point to the 7th character after the beginning of the input string.<\/p>\n
Conclusion<\/h4>\n
The methodology as outlined above is applicable for simple components not exposing any additional functionality, parameterization, or member functions on its own. But since most of Spirits<\/em> predefined parsers are written using this technique I assume it will allow you to go a long way before you need to apply more powerful tools. For now, I leave the description of how to implement one of those more complex parsers for a future article. Stay tuned!<\/p>\n
If you are interested in learning more about this topic, here<\/a> you can find more material about this in the main Spirit documentation. The complete source code for this example can be found in the Boost<\/a> SVN. It consists of two files, the header file iter_pos.hpp<\/a><\/span>, which is self contained and ready to be reused in your projects, and the example source file iter_pos_parser.cpp<\/a><\/span>.<\/p>\n
Share this:<\/h3>