{"id":659,"date":"2009-12-05T17:02:31","date_gmt":"2009-12-06T01:02:31","guid":{"rendered":"http:\/\/boost-spirit.com\/home\/?page_id=659"},"modified":"2009-12-05T17:45:15","modified_gmt":"2009-12-06T01:45:15","slug":"creating-your-own-generator-component-for-spirit-karma","status":"publish","type":"page","link":"http:\/\/boost-spirit.com\/home\/articles\/karma-examples\/creating-your-own-generator-component-for-spirit-karma\/","title":{"rendered":"Creating Your Own Generator Component for Spirit.Karma"},"content":{"rendered":"

In the good tradition of highlighting Spirits<\/em> both major sub-libraries based on similar use cases I will talk about Karma<\/em> today. In a previous installment (see Creating Your Own Parser Component for Spirit.Qi<\/a>) I presented the steps needed to create a parser terminal. Our topic here is the creation of a new generator directive allowing to group output elements in columns. For instance, instead of being restricted to printing a vector of integers in one row only:<\/p>\n

\r\n1 2 3 4 5 6 7 8 9\r\n<\/pre>\n
we will build the facilities needed to automatically insert a line break after each N-th element:<\/p>\n
\r\n1 2 3 4 5\r\n6 7 8 9\r\n<\/pre>\n
This is a very useful feature in general still missing from Karma<\/em>. For this reason the described directive has been recently added to the library making it available with the next release. But to simplify matters the code presented here will be an abridged version of the full implementation as contained in the main code base. We chose the following syntax for our new directive columns<\/span>, where the embedded expression (in the example the *int_<\/span>) can be any other generator:<\/p>\n
\r\ncolumns[*int_]\r\n<\/pre>\n
As you might already know, all generators are invoked with a special delimiting generator as one of their parameters, which is used to insert special delimiters in between the output emitted by Karma\u2019s<\/em> primitives. We will see in the example below how to specify it. This delimiting generator is very similar to Qi\u2019s<\/em> skip parser. As mentioned in an earlier article<\/a>\u00a0Karma<\/em> is very much a mirrored image of Qi<\/em>, which applies nicely in this case as well.<\/p>\n
Now, the columns<\/span> directive will substitute the delimiting generator for the embedded generator with a generator which not only wraps and invokes the old delimiter, but additionally emits an additional column delimiter after its N-th invocation. The original delimiter will be restored after the embedded generator expression finished its work. As the chosen syntax above does not allow to specify neither the number N nor the column delimiter to use we will assume for this example that we always want to generate columns by inserting a newline after each 5th invocation of our special delimiting generator (the \u2018real\u2019 columns<\/span> directive as added to Karma<\/em> does support specifying both parameters, for more information please consult its documentation<\/a>).<\/p>\n
Creating a custom directive is a bit more difficult than implementing a primitive component. In addition to the 4 similar steps required for the primitive (if you have not seen the related article, please see Creating Your Own Parser Component for Spirit.Qi<\/a>) we need to perform one more step. Even if this will be a bit like a repetition of what I wrote in that other article, I will explain the necessary steps in detail.<\/p>\n
Defining the Placeholder<\/h4>\n
The first step is again to define the placeholder representing our new component when building generator expressions (that\u2019s the columns<\/span> symbol). This has to be done by using the predefined macro BOOST_SPIRIT_TERMINAL<\/span>:<\/p>\n
\r\nnamespace custom_generator { BOOST_SPIRIT_TERMINAL(columns); }\r\n<\/pre>\n
which can be placed in any namespace. We assume our custom generator component will be created in the namespace custom_generator<\/span>. This macro defines two types and a const<\/span> instance of one of the types. It essentially expands to:<\/p>\n
\r\nnamespace tag { struct columns {}; } \/\/ tag identifying placeholder\r\ntypedef unspecified<tag::columns> columns_type;\r\ncolumns_type const columns = {};     \/\/ placeholder itself\r\n<\/pre>\n
We will use the type tag::colums<\/span> later to identify our component, whereas the variable columns<\/span> is the one to be used as the placeholder representing our generator directive in more complex grammars.<\/p>\n
Implementing the Enabler<\/h4>\n
The placeholder needs to be associated with the appropriate extension mechanism (Spirit<\/em> has separate extension points for primitive generators, generator directives, operators, etc.). We need to implement an enabler for the custom generator in a way allowing the library to recognize columns<\/span> as a directive only, as we don\u2019t want it to be valid if used as a primitive generator, etc. The extension point used for directives is boost::spirit::use_directive<><\/span>. We enable our custom generator by providing a specialization of this extension point, which has to be placed in the namespace boost::spirit<\/span>:<\/p>\n
\r\nnamespace boost { namespace spirit\r\n{\r\n    \/\/ We want custom_generator::columns to be usable as a directive only,\r\n    \/\/ and only for generator expressions (karma::domain).\r\n    template <>;\r\n    struct use_terminal<karma::domain, custom_generator::tag::columns>\r\n      : mpl::true_ {};\r\n}}\r\n<\/pre>\n
Spirit<\/em> will pick up this template specialization whenever it sees our placeholder custom_generator::columns<\/span> while compiling a generator expression (that\u2019s why the karma::domain<\/span>).<\/p>\n
Implementing the Generator itself<\/h4>\n
Everything we have seen so far is boilerplate code allowing to integrate our new generator with the component framework of Spirit<\/em>. This step describes the actual interface to be implemented in order to expose the required functionality as a generator component. Here is the full code, please note everything is placed into our own namespace custom_generator<\/span>.<\/p>\n
The class simple_columns_generator<\/span> is slightly more complicated than the iter_pos<\/span> parser component developed earlier, but if you look closer you will see a very similar, equally well defined interface we need to implement in order to fit the new generator into the existing framework.<\/p>\n
We derive our generator implementation from boost::spirit::karma::unary_generator<><\/span> to associate our component with the correct generator concept, in this case a UnaryGenerator<\/a><\/span>.<\/p>\n
\r\n\/\/ That's the actual columns generator\r\ntemplate <typename Subject>\r\nstruct simple_columns_generator\r\n  : boost::spirit::karma::primitive_generator&lt;\r\n        simple_columns_generator<Subject> >\r\n{\r\n    \/\/ more members go here as explained below\r\n    Subject subject;\r\n};\r\n<\/pre>\n
The embedded type properties<\/span> is specific for Karma<\/em> generators (and not needed for Qi<\/em> parser components). Here is a piece of background information: Karma internally wraps the user provided output iterator into its own. This is done to allow for certain advanced features as buffering, character counting, and output position tracking. Usually only a part of those features is required for the concrete generator expression. The embedded type properties<\/span> describes the required properties of all generator components. Karma<\/em> uses this information to optimize its wrapping output iterator to support only the required features. In our case the columns generator does not add any requirements, which makes it sufficient to expose the properties of the embedded generator.<\/p>\n
\r\n\/\/ Define required output iterator properties\r\ntypedef typename Subject::properties properties;\r\n<\/pre>\n
The embedded meta function attribute<\/span> is a template which will be instantiated with a Context<\/span> type (we can ignore this for our small example) and with the type of the OutputIterator<\/span> the generator component is being used with. It needs to have defined an embedded type definition type<\/span>. This will be interpreted as the attribute type exposed by the generator component. As the columns<\/span> generator will have to expose the attribute of its embedded expression we use another of Spirits customization points, attribute_of<\/span>, to retrieve the embedded attribute.<\/p>\n
\r\n\/\/ Define the attribute type exposed by this parser component\r\ntemplate <typename Context, typename Iterator>\r\nstruct attribute\r\n  : boost::spirit::traits::attribute_of<Subject, Context, Iterator>\r\n{};\r\n<\/pre>\n
The constructor of the simple_columns_generator<\/span> component will be called with a reference to the embedded generator, we will see the corresponding code later.<\/p>\n
\r\nsimple_columns_generator(Subject const& s)\r\n  : subject(s) {}\r\n<\/pre>\n
The member function generate()<\/span> is called in order to emit the output for this component. It will be invoked with the output iterator to be used (sink<\/span>), an instance of the generator context (ctx<\/span>), the reference to the used delimiter instance (delimit)<\/span>, and the reference to the attribute instance the component generates output from (attr)<\/span>. Most of the arguments are just forwarded to the embedded component\u2019s generate()<\/span> function. As mentioned above, our columns<\/span> generator wraps the outer delimiter into a new column delimiter (which is a generator itself) and passes this new column delimiter while executing generate()<\/span> of its embedded component. We will look at the implementation in a minute.<\/p>\n
\r\n\/\/ This function is called during the actual output generation.\r\n\/\/ It dispatches to the embedded generator while supplying a new\r\n\/\/ delimiter to use, wrapping the outer delimiter.\r\ntemplate <typename OutputIterator, typename Context\r\n  , typename Delimiter, typename Attribute>\r\nbool generate(OutputIterator& sink, Context& ctx\r\n  , Delimiter const& delimiter, Attribute const& attr) const\r\n{\r\n    columns_delimiter<Delimiter> d(delimiter);\r\n    return subject.generate(sink, ctx, d, attr) &&\r\n           d.final_delimit(sink);\r\n}\r\n<\/pre>\n
The function what()<\/span> is invoked by the library whenever a human readable string is needed identifying this generator. This is most notably used for error handling, allowing to generate a nicely formatted description about the error context. Our implementation again mostly forwards to the embedded generator.<\/p>\n
\r\n\/\/ This function is called during error handling to create\r\n\/\/ a human readable string for the error context.\r\ntemplate <typename Context>\r\nboost::spirit::info what(Context& ctx) const\r\n{\r\n    return boost::spirit::info("columns", subject.what(ctx));\r\n}\r\n<\/pre>\n
We have all desired parts of the generator in place now. What’s still missing is the column delimiter. It has two data members: the reference to the outer delimiter and the column counter. The outer delimiter is passed to the constructor from the generator()<\/span> function we saw above. As this class is used as a generator (all delimiters are<\/strong> generators) it needs to expose a generate()<\/span> function as well, with exactly the same prototype as described. The column delimiter will be invoked by the components constituting the embedded generator. For this reason we first invoke the wrapped (outer) delimiter and increment the invocation counter. After each 5th invocation we insert an additional newline into the output stream.<\/p>\n
The last missing piece is the function final_delimit()<\/span> which will be invoked once after the embedded generator returned (see the function generate()<\/span> described above). Its sole purpose is to append a column generator at the very end if the last invocation didn\u2019t already append one.<\/p>\n
\r\n\/\/ Special delimiter wrapping the original one while additionally emitting\r\n\/\/ the column delimiter after each 5th invocation\r\ntemplate <typename Delimiter>\r\nstruct columns_delimiter\r\n{\r\n    columns_delimiter(Delimiter const& delim)\r\n      : delimiter(delim), count(0) {}\r\n\r\n    \/\/ This function is called during the actual delimiter output\r\n    template <typename OutputIterator, typename Context\r\n      , typename Delimiter_, typename Attribute>\r\n    bool generate(OutputIterator& sink, Context&, Delimiter_ const&\r\n      , Attribute const&) const\r\n    {\r\n        \/\/ first invoke the wrapped delimiter\r\n        if (!karma::delimit_out(sink, delimiter))\r\n            return false;\r\n\r\n        \/\/ now we count the number of invocations and emit the column\r\n        \/\/ delimiter after each 5th column\r\n        if ((++count % 5) == 0)\r\n            *sink++ = '\\n';\r\n        return true;\r\n    }\r\n\r\n    \/\/ Generate a final column delimiter if the last invocation didn't\r\n    \/\/ emit one\r\n    template <typename OutputIterator>\r\n    bool final_delimit(OutputIterator& sink) const\r\n    {\r\n        if (count % 5)\r\n            *sink++ = '\\n';\r\n        return true;\r\n    }\r\n\r\n    Delimiter const& delimiter;   \/\/ wrapped delimiter\r\n    mutable unsigned int count;   \/\/ invocation counter\r\n};\r\n<\/pre>\n
Overall, this step was a bit more involved, mainly as we needed to write more code. Nevertheless,\u00a0 I believe it highlights the important parts of the interface your own components with the libraries infrastructure.<\/p>\n
Instantiating the Generator<\/h4>\n
The next required piece of code is a generator function object which will be used by the library to instantiate a new instance of our generator component. Non-surprisingly, the name of the function object we need to specialize is make_directive<\/span>, and since we were using karma::domain<\/span> in the first step above, we now need to place this specialization into the namespace boost::spirit::karma<\/span>.<\/p>\n
\r\nnamespace boost { namespace spirit { namespace karma\r\n{\r\n    \/\/ This is the factory function object invoked in order to create\r\n    \/\/ an instance of our simple_columns_generator.\r\n    template <typename Subject, typename Modifiers>\r\n    struct make_directive<\r\n        custom_generator::tag::columns, Subject, Modifiers>\r\n    {\r\n        typedef custom_parser::simple_columns_generator<Subject>\r\n            result_type;\r\n\r\n        result_type operator()(\r\n            unused_type, Subject const& s, unused_type) const\r\n        {\r\n            return result_type(s);\r\n        }\r\n    };\r\n}}}\r\n<\/pre>\n
You can think of this function object as of a factory for our generator object. Our specialization is again based on the tag::columns<\/span> as defined above. This identifies our generator component. The function object make_directive<\/span> has to expose the type of the component it creates as its embedded type definition result_type<\/span>. Additionally it exposes a function operator as the actual factory function. The unused_type<\/span> is Spirits<\/em> way of saying: \u2018I don\u2019t care\u2019, and as we don\u2019t care about two of the three (required) parameters we use unused_type<\/span> instead (here are the details anyway: the first parameter is a reference to the columns<\/span> placeholder instance resulting in this factory being invoked, and the third parameter is a reference to an object instance of the type Modifiers<\/span>, which is needed only for directives like lower[]<\/a>). The second parameter refers to the embedded generator being used with the columns<\/span> directive. We pass this to the constructor of\u00a0 the new instance of the simple_columns_generator<\/span> we create to return from the factory function. Earlier we saw how this instance is stored and used by the columns<\/span> generator.<\/p>\n
Enabling proper Auto-Rule Behavior<\/h4>\n
The last step required\u00a0for making a new directive for Spirit<\/em> is to specialize yet another customization point. In order to understand this you need some more background information. Spirit\u2019s<\/em> rule<><\/span> non-terminals have special built-in capabilities to do proper attribute propagation to\/from the right hand side expression they are initialized from (we call this auto-rule behavior). This attribute propagation is guaranteed to work only as long as no semantic actions are attached to any of the right hand side’s sub-expressions. The specialization of the template has_semantic_action<\/span> allows detecting its presence. Our code just uses built-in facilities to forward the detection to the embedded (subject) generator.<\/p>\n
\r\nnamespace boost { namespace spirit { namespace traits\r\n{\r\n    template <typename Subject>\r\n    struct has_semantic_action<\r\n            custom_generator::simple_columns_generator<Subject> >\r\n      : unary_has_semantic_action<Subject> {};\r\n}}}\r\n<\/pre>\n
Using the New columns<\/span> Component<\/h4>\n
Last but not least we will have a look at a small example demonstrating the usage of the new columns<\/span> component. The following code generates exactly the output shown in the very beginning of this article:<\/p>\n
\r\nstd::vector<int> v;\r\nfor (int i = 1; i < 10; ++i)\r\n    v.push_back(i);\r\n\r\nstd::string generated;\r\nstd::back_insert_iterator<std::string> sink(generated);\r\n\r\nkarma::generate_delimited(sink\r\n  , custom_generator::columns[*karma::int_]    \/\/ formatting directive\r\n  , karma::space                               \/\/ outer delimiter\r\n  , v);                                        \/\/ data to output\r\n<\/pre>\n
Conclusion<\/h4>\n
There is not much to add. A general observation perhaps: Spirit<\/em> has been built based on customizations points. These are templates to be specialized in order to add functionality. Usually the core library provides the main template for those specializations, exposing some default behavior. As all predefined Spirit<\/em> components use the customization points of the core library as well, all users are free to extend the library to adapt it to their needs. All this got possible only after the available C++ compilers started to properly support partial template specialization, so you won\u2019t be able to use Spirit with older or non-conforming compilers (see here<\/a> for a list of supported compilers).<\/p>\n
All of this code shown above can be downloaded from the Boost<\/a> SVN here<\/a> (and the example code is here<\/a>). As already mentioned, starting with the next release Karma<\/em> will provide a more complex columns<\/span> directive, but this small example will stay around as well.<\/p>\n
Share this:<\/h3>