Continue reading »

All three of Spirit’s sub-libraries – Qi, Karma, and Lex – support semantic actions. In each case they are different and have some specifics. Today I will highlight semantic actions in Qi. But I will dedicate later Tips of the Day to semantic actions in Karma and  Lex.

Semantic actions are functions or function objects attached to some specific part of a grammar. In Qi they are invoked after the corresponding parser successfully recognizes a portion of the input. Here the semantic action receives the attribute value of the matching parser.

Semantic Actions – a General View

A semantic action f are attached to a Qi parser p by simply writing:

p[f]

The function (or function object) f has to expose a certain interface allowing Spirit to pass the proper argument types. In the simplest case this can be a global function taking no arguments at all.

void func()
{
    std::cout << "Matched an integer!\n";
}

std::string input("1234");
std::string::const_iterator begin = input.begin();
std::string::const_iterator end = input.end();
qi::parse(begin, end, qi::int_[func]);     // this will call func

Most of the time this is not sufficient as a semantic action is expected to receive the matched attribute value. This is possible by writing:

void func(int attribute)
{
    std::cout << "Matched integer: " << attribute << "\n";
}

The type of the expected parameter (in this case the int) depends on the parser the semantic action is attached to. The attribute type exposed by the parser has to be convertible to the argument type.

There are actually 2 more arguments being passed: the parser context and a reference to a boolean ‘hit’ parameter. The parser context is meaningful only if the semantic action is attached somewhere to the right hand side of a rule. We will see more information about this shortly. The boolean value can be set to false inside the semantic action invalidates the match in retrospective, making the parser fail. Qi allows us to bind a nullary or a single argument function, like above. The other arguments are simply ignored.

It is feasible to bind any function object (such as generated by Boost.Bind or Boost.Lambda) as an semantic action. Even if the documentation shows a couple of examples (see here), I would not recommend using those libraries in this context. For me the preferred method of writing semantic actions is to employ Boost.Phoenix – a companion library bundled with Spirit. It is like Boost.Lambda on steroids, with special custom features that make it easy to integrate semantic actions with Spirit. If your requirements go beyond simple parsing, I suggest that you use this library. All the following examples in this article will use Boost.Phoenix for semantic actions. But whatever method you use, please let me highlight the following:

The three libraries allow you to utilize special placeholders to control parameter placement (_1, _2, etc.). Unfortunately, each of those libraries has it’s own implementation of the placeholders, all in different namespaces. You have to make sure not to mix placeholders with a library they don’t belong to and not to use different libraries while writing a semantic action.

Generally, for Boost.Bind, use ::_1, ::_2, etc. (yes, these placeholders are defined in the global namespace).

For Boost.Lambda use the placeholders defined in the namespace boost::lambda.

For semantic actions written using Boost.Phoenix use the placeholders defined in the namespace boost::spirit. Please note that all existing placeholders for your convenience are also available from the namespace boost::spirit::qi.

The current version of Spirit (V2.2) does not yet support binding a native C++0x lambda function as a semantic action, but this is something we are currently working on. You can expect this to be possible in the near future.

Writing Phoenix based Semantic Actions

Writing a semantic action with Phoenix is beneficial as Spirit  ‘knows’ about Phoenix. If you write them with the help of Phoenix you can utilize special placeholders Spirit provides you with. Those placeholders refer to elements in the context of the current parser execution such as attributes, local variables and inherited attributes of rules, etc. None of the other means of writing semantic actions (using Bind, Lambda, or had written function objects) gives you direct access to those elements. The following table lists all available placeholders exposed by Spirit (as mentioned earlier, all are defined in the namespace boost::spirit::qi). Again, please note, these are only available inside a semantic action and only if the semantic action is written utilizing Phoenix.

Obviously, the placeholders listed in the last three rows of the table are meaningful only if used in a rule definition. As an example, let us rewrite the semantic action from above with Phoenix:

std::string input("1234");
std::string::const_iterator begin = input.begin();
std::string::const_iterator end = input.end();
qi::parse(begin, end,
    qi::int_
    [
        std::cout << "Matched integer: " << qi::_1 << "\n";
    ]
);

One problem with earlier versions of Spirit (i.e. Spirit.Classic) was that while parsing sequences of things it was difficult to avoid calling a semantic action prematurely. For instance, in a parser sequence of two integer parsers (int_[f1] >> ‘,’ >> int_[f2]) the function f1 got called immediately after the first integer matched, and even if the second integer parser would fail later on. In the current version of Spirit this is not an issue anymore as it is possible to attach a semantic action to the whole sequence while still referring to the single attributes of the different sequence elements:

std::string input("1234,2345");
std::string::const_iterator begin = input.begin();
std::string::const_iterator end = input.end();
qi::parse(begin, end,
    (qi::int_ >> ',' >> qi::int_)
    [
        std::cout << "Matched integers: "
              << qi::_1 << " and " << qi::_2 << "\n";
    ]
);

Here, qi::_1 refers to the attribute matched by the first integer parser, and qi::_2 to the second one.

Initially I was planning to additionally describe the internal interface of a semantic action. Utilizing this interface allows you to write your own function objects and still to get access to the elements of the parser context mentioned above (attributes, the rule’s local variables and inherited attributes, etc.). But this post already got longer as anticipated, which is why I defer this discussion to a second Tip of the Day. Stay tuned!

While parsing some formatted data stream it is very often desirable to ignore some parts of the input. A common example would be the need to skip whitespace and comments while parsing some computer language. Certainly it is possible to explicitly account for the tokens to skip (such as the whitespace or the comments) while writing the grammar. But this can get very tedious as those tokens are valid to appear at any point in the input.

For the sake of simplicity, let us assume we want to parse a simple key/value expression: key=value, where we want to allow for any number of space characters before, in between, or after the key or the value. A naive grammar matching the plain key/value pair without whitespace skipping would look like (see Parsing a List of Key-Value Pairs Using Spirit.Qi for more details):

pair  =  key >> '=' >> value;
key   =  qi::char_("a-zA-Z_") >> *qi::char_("a-zA-Z_0-9");
value = +qi::char_("a-zA-Z_0-9");

If we want to explicitly accommodate the rule pair to match any interspersed space characters we get:

pair  = *space >> key >> *space >> '=' *space >> value >> *space;

which, while it produces the desired result, is not only error prone, but additionally difficult to write, to understand, and to maintain. If we look closer we see, that the process of skipping the whitespace tokens is easily automated. It seems to be sufficient to insert a repeated invocation of the space parser (or generally, any skip parser) in between the elements of the user defined parser expression sequences.

In fact, that is exactly what Spirit can do for you! The library invokes any supplied skip parser upon entry to the parse member function of any parser conforming to the PrimitiveParser concept. The skip parser has to be supplied by calling a special API function: phrase_parse:

namespace qi = boost::spirit::qi;
typedef std::string::const_iterator iterator;

qi::rule<iterator, qi::space_type> pair = key >> '=' >> value;
qi::rule<iterator> key = qi::char_("a-zA-Z_") >> *qi::char_("a-zA-Z_0-9");
qi::rule<iterator> value = +qi::char_("a-zA-Z_0-9");

std::string input(" key = value ");
iterator_type begin = input.begin();
iterator_type end = input.end();
qi::phrase_parse(begin, end, pair, qi::space);

This code snippet illustrates several important things:

• The function qi::phrase_parse is equivalent to the API function qi::parse except for its additional parameter, the skip parser. Our example utilizes qi::space, but it is possible to use any other, even more complex parser expression as the skipper instead.
• All rules which we want to perform the skip parsing need to be declared with the type of the skip parser they are going to be used with. Our example specifies the type of the qi::space parser expression, which is qi::space_type. For more complex parser expressions you might want to use a (mini) grammar or take advantage of BOOST_TYPEOF to let the compiler deduce the actual type.
• All rules which should not perform skip parsing have to be declared without an additional skip parser type. These rules behave like an implicit lexeme[] directive (for more information about lexeme[], see below), they inhibit the invocation of the skip parser even if they are executed as part of a rule with an associated skipper.

In the example above we suppressed skipping while matching either the key or the value, otherwise our grammar would match any additional space character inside the key or value as well. Remember, the expression char_ conforms to the PrimitiveParser concept, it will execute the skip parser for each of its invocations. In this case any skip parser would be executed in between any two of the matched characters.

Sometimes it is necessary to turn of skipping for a smaller part of the grammar only. For this purpose Spirit implements the lexeme[] directive. This directive inhibits skipping during the execution of the embedded parser. For instance, parsing a quoted string of alphanumeric characters would look like this:

string = lexeme['"' >> *alnum >> '"'];

Here the lexeme directive disables skipping while matching the string, which avoids ‘loosing’ characters otherwise matched by the skipper. Please note: lexeme[] performs a pre-skip step, even if it is not a PrimitiveParser itself (it is essentially considered to be a logical primitive by design). If this is undesired, you can utilize the no_skip[] directive instead:

string = '"' >> no_skip[*alnum] >> '"';

This parser will match all the characters in between the quotes, even if the string starts with a character sequence matched by the applied skip parser. The no_skip[] directive is semantically equivalent to lexeme[] except it does not perform a pre-skip before executing the embedded parser. Note: the no_skip[] directive has been added only recently. It will be available starting with the next release (Boost V1.43).

This short article would not be complete without mentioning the skip[] directive. This directive is the counterpart to lexeme[]. It enables skipping for the embedded parser. Without any argument it can be used inside a lexeme or no_skip directive only. In this case it just re-enables the outer skipper:

string = lexeme['"' >> *(alpha | skip[digit]) >> '"'];

This (purely hypothetical) parser would enable skipping inside a string as long as it matches digits. But the skip directive can do more. It may take an additional argument allowing to specify a new skipper, for instance:

skip(qi::space)[*alnum]

which will skip spaces while executing the embedded *alnum parser. This form of the directive can be applied for two purposes. It can be used either for changing the current skip parser or to establish skipping inside a context otherwise not doing skipping at all (even if invoked with the qi::parse() API function).

For more detailed information about all the mentioned directives please see the corresponding documentation.

Spirit’s permutation parser a ^ b matches either a, b, a >> b, or b >> a, where a and b can be arbitrary parser expressions. Just like normal sequences this operator can be utilized to combine more than two operands. For instance, the expression a ^ b ^ c will match a or b or c (or an combination thereof) in any sequence. The attribute propagation rule for the permutation parser is

a: A, b: B --> (a ^ b): tuple<optional<A>, optional<B> >

As usual, if one or more operand of the expression do not expose any attribute (expose unused_type as their attribute, which is equivalent), this operand disappears from attribute handling:

a: A, b: Unused --> (a ^ b): optional<A>;

The permutation parser works out of the box whenever you do not require to match all of the elements in the input. But what if you want strict permutation (operands get matched exactly once)? You have two possibilities, as often, one simple and less versatile and one more complex but universally applicable solution. The simple solution is to parse the input and to check afterward whether all optionals in the resulting attribute have been filled. I will leave that solution as an exercise for the reader.

If we assume the attribute to be a (Fusion) tuple of optionals, containing one optional for each of the parser components in the permutation parser we can write the following code (thanks to Carl Barron for the initial idea).

This code defines a Phoenix function (a lazy function encapsulating some custom functionality) checking whether one or more of the optionals in a given Fusion sequence are empty. The Fusion algorithm find_if iterates over the given sequence of optionals, invoking the option_empty::operator() for each of the elements. fusion::find_if stops iterating on the first invocation returning true and returns the iterator to the element it stopped on. This is very similar to the well known std::find_if algorithm.

namespace phoenix = boost::phoenix;
namespace fusion = boost::fusion;
namespace qi = boost::spirit::qi;

class no_empties_impl
{
    // helper function object to be invoked by fusion::find_if
    struct optional_empty
    {
        template <typename T>
        bool operator ()(T const& val) const
        {
            return !val;  // return true if 'val' is empty.
        }
    };

public:
    template <typename T>
    struct result { typedef bool type; };

    // This operator will get called from the semantic action attached
    // to the permutation parser. The parameter refers to its overall
    // attribute: the fusion tuple of optionals.
    template <typename T>
    bool operator ()(T const& t) const
    {
        // look for an empty optional, if any return false.
        return fusion::find_if<optional_empty>(t) ==
               fusion::end(t);
    }
};

// define the Phoenix function
phoenix::function<no_empties_impl> const no_empties = no_empties_impl();

The overall Phoenix function no_empties will return false if we found at least one non-initialized optional in the passed sequence. The following code snippet illustrates how everything fits together:

std::string input ("BCA");
std::string::const_iterator begin = input.begin();
std::string::const_iterator end = input.end();
qi::parse(begin, end,
    (qi::char_('A') ^ 'B' ^ 'C')[qi::_pass = no_empties(qi::_0)]);

We assign the result of the invocation of no_empties to Qi’s predefined placeholder _pass. If we assign false, then the parser the semantic action is attached to will be forced to fail in retrospective (even if it matched the input successfully before). As a result the overall parser expression will succeed as long as a) the permutation parser matches its input and b) the Phoenix function inside the semantic action returns true.

For more information about the permutation parser please consult its documentation here. Overall, this example is a bit more complex than the average parser you might usually write. It utilizes three libraries: Spirit, Phoenix, and Fusion in a seamless manner. But for sure, once you understand the idea, it will be easier for you to come up with similar solutions. Spirit has been designed with Phoenix and Fusion in mind, and in fact it relies on Fusion heavily itself. As a result, the integration of those libraries is almost perfect.

The one commonality of the two operators is the same as in Qi: both negate whether the component they are being used with succeeds generating. If the component ‘c’ succeeds, both compound constructs, ‘!c’ and ‘~c’ will fail, and similarly, if ‘c’ fails, the execution of the components ‘!c’ and ‘~c’ will succeed.

Similar to its counterpart in Qi, the unary Karma operator ‘~’ is applicable to character and character class generators only. It negates the set of characters a generator will be allowed to emit. Let me explain. The Karma character set generators, such as char_(“a-z”) or digit will emit their attribute only if the attribute value belongs to the character set described. Consequently, applying the operator ‘~’ will negate the character set the generator it is attached to. Here are some examples:

If a generator can’t emit its attribute it will fail. This is very similar to a failing parser component. The possibility for a generator component to fail is very useful. This can be utilized for alternatives, predicates and other constructs. But this is beyond today’s topic and will be discussed in a different installment of the ‘Tip of the Day’.

The generators created by the operator ‘~’ do not wrap the underlying generator. The operator rather modifies the behavior of the component it is attached to. This means there is no performance difference if compared to the plain character generators.

The unary operator ‘!’ creates a not-predicate generator. Again, similar to its counterpart in Qi, it can be attached to any (arbitrarily complex) generator. The not-predicate generator will succeed if the associated generator fails, and it will fail if its generator succeeds. It invokes the generator it is attached to but the emitted output will not show up in the overall output stream. So effectively, the not-predicate generator will never emit any output. The following example will succeed emitting a floating point number if the first attribute is false (which will make the true_ generator fail), otherwise it will not emit anything and will fail all together:

namespace karma = boost::spirit::karma;
std::string output;
std::back_insert_iterator<std::string> sink(output);
karma::generate(sink, !true_ << double_, false, 1.0); // will emit: 1.0

This example highlights another difference if compared to Qi’s not-predicate. The Karma not-predicate will always consume an attribute. More accurately, it will expose the attribute of the generator it is associated with (while in Qi the not-predicate never exposes any attribute). As mentioned earlier, we utilize the true_ generator’s ability to fail to control what output is generated (or if any output is generated at all as in our case).

We know the documentation of Spirit.Lex is not complete yet. So I will write  more about it here from now on to fill in the missing pieces and to show a couple of tricks demonstrating its best usage.

Lexical analysis is done in a separate module from the parser, feeding the parser with a stream of input tokens only. The following picture visualizes this process.

Instead of directly getting the next character from the input the parser asks the lexer for the next input token. The lexer in turn looks at the next input characters to decide what token patterns match best. Only after matching the next token from the input it returns it to the parser.

Theoretically it is not necessary to implement this separation as in the end the language is defined only by exactly one set of syntactical rules. So we could write the whole parser in one module. In fact, Spirit.Qi allows you to write parsers without using a lexer, while parsing the input character stream directly, and for the most part this is the way Spirit has been used since its invention.

So the question is: “When does it make sense to invest the additional time and effort to create a separate lexer for your grammar and when is it sufficient to exclusively utilize Spirit.Qì?” Unfortunately, there is no single answer to this question, and all I can try is to highlight some of the advantages and disadvantages of a separate lexer module. The concrete decision whether to utilize a lexer or to parse the grammar in one step has to be made on a case by case basis.

Spirit.Lex gives you the ability to create lexical analyzers based on patterns. These patterns are regular expression used to define the different tokens to be recognized in the character input sequence. The lexer generates internal tables from all token definitions – the so called deterministic finite automata (DFA’s). It is well know that matching an input sequence using a DFA is as efficient as it can get. Spirit.Lex is fast! Measurements prove that lexers generated with Spirit.Lex are faster than comparable lexers generated with flex, the most widely used lexical generator used today.

The input sequence is expected to be provided to the lexical analyzer as an arbitrary standard forward iterator. The lexical analyzer itself exposes a standard forward iterator as well. The difference here is that the exposed iterator provides access to the token sequence instead of to the character sequence. The tokens in this sequence are constructed on the fly by analyzing the underlying character sequence and matching this to the patterns as defined by the application.

Here is a list of additional advantages Spirit.Lex provides you with:

• The definition of tokens is done using regular expressions, where the token definitions can refer to special substitution strings (pattern macros) simplifying the token definitions.
• The generated lexer may have multiple start states (we will talk about lexer states in a separate post).
• It is possible to attach code to any of the token definitions; this code gets executed whenever the corresponding token pattern has been matched.
• The iterator exposed by the lexer buffers the last emitted tokens. This significantly speeds up parsing of grammars which require backtracking.
• The tokens created at runtime can carry arbitrary token specific data items which are available from the parser as attributes. The input character sequence is converted to the token data items exactly once, regardless of how often the value is accessed.
• Token based parsing simplifies the Qi grammar necessary to describe the matching rules. This lessens the runtime overhead introduced by the parser itself.

But at the same time this feature list hints at the disadvantages of a separate lexer module:

• The definition of tokens is done using regular expressions. To some, regular expressions are more difficult to understand compared to EBNF or PEG rules.
• Additional effort is required to develop and debug the lexer and its token descriptions.
• The lexer definition may add measurable compile time overhead, depending on what features are employed.
• Additional runtime overhead is required in order to generate the lexer tables and construct the tokens.

The last bullet point touches on the most interesting characteristic: the speed of parsing. As already mentioned, generally it is not possible to predict whether the overhead introduced by the lexer is amortized by savings on grammar specific things like required backtracking, repeated attribute conversion, required symbol lookup, etc. But the experience shows that lexers tend to pay off for moderately sized or large grammars, such as full language implementations or data format descriptions with a lot of dynamic structural alternatives. But when in doubt, you will have to do measurements based on your concrete grammar and input data to make a sound decision. You know the drill…

I chose this topic even if I got some comments asking me to avoid writing about things already covered in Spirit’s documentation. At the same time, I believe it’s valuable for some of you to repeat the facts from the docs but presented in a more pointed and descriptive way. Additionally, I think the articles as published here lay the ground for more detailed things not yet covered in the docs, but the basics need to be explained properly before we can touch the more advanced stuff. But please leave your comments below telling me what you would like to see highlighted. I am always interested in getting feedback on what works for you and what not.

Ok, here we go – let us talk about rule bound semantic actions.

Any semantic action attached to the right hand side of a rule (or a part of it) is special. We will call those ‘rule bound semantic actions’. Spirit provides you with extra placeholder variables (in addition to _1, _2, etc.) usable to access values related to the enclosing rule (the rule, the expression has been assigned to). For the purpose of this post I assume, that the semantic actions are written using Boost.Phoenix. In fact, all placeholders described below are implemented using Phoenix and do not integrate very well with other, similar facilities available in Boost (such as Boost.Bind or Boost.Lambda). You should avoid mixing Phoenix placeholder with expressions built from those libraries. This is no limitation, as from my experience, using Phoenix is just the simplest way to write semantic actions.

Let us assume further, that we have a component c with an attached semantic action f, which is assigned as the right hand side of a rule exposing a synthesized attribute of type Attr and providing local variables of types L1, L2, etc. (I will explain local variables shortly):

rule<Iterator, Attr(), locals<L1, L2, ...> > r = c[f];

Here is a list of all predefined placeholders available in the rule bound semantic action F (see also the related documentation here and here):

We already talked about _val, _1, _2, etc. last time, and I will not describe inherited attributes today, that is for another day. I would like to concentrate on local variables here.

Local variables allow to associate arbitrary typed data instances with each rule invocation. These variables are very similar to local variables in a function. They are valid only during the invocation of the rule they are defined in and they automatically go out of scope after the right hand side of the rule has finished executing. Further, each invocation of a rule creates a new set of (default constructed) local variables, even if the same rule is invoked recursively.

Here is a small Karma example generating a list of items prefixed with a sequence number (note: _a, eps, string, lit, rule, and locals are imported from namespace boost::spirit::karma).

rule<OutIter, locals<int>, std::vector<std::string>()> r =
    eps[_a = 1] << (lit(_a) << eps[++_a] << " " << string) % '\n';

Let us analyze, what this rule definition does. 

• The attribute of this rule is std::vector<std::string>, for the sake of simplicity we will pass in all items stored in this type of container. The items are emitted using the list operator (%), causing them to be interleaved with newlines.
• One of the rule’s template parameters is locals<int>. That is the way we declare the types of the local variables. The template locals<> is a type container you can use to list the types for all local variables to instantiate for each of the rule’s invocations.
• In our example, we have a single local variable of type int  and we reference it from the right hand side of the rule definition using the placeholder _a.
• The generator eps is utilized to inject invocations of semantic actions initializing and incrementing the local variable. The generator eps is perfect for doing this as it succeeds always while emitting nothing.
• The most notable construct is lit(_a) showing that it is possible to directly create a generator component from the local variable. Many of Spirit’s construct are enabled for lazy evaluation, and lit() is one of them.

The only additional thing I want to add is that everything shown above works equally well in Qi, but you probably already guessed this. For your convenience, the corresponding symbols are available from the namespace boost::spirit::qi as well. If you are interested in seeing locals in action in a more complete Qi example you should have a look at the mini-xml series as described in the docs here.

This ‘Tip of the Day’ got a little longer than usual, but I hope it is worth reading anyways. Spirit is a large library and there are uncounted hidden gems waiting not only to be described, but some of them are still waiting to be discovered. Happy hacking and experimenting!

Spirit has some new features related to semantics actions. That’s reason enough to talk about how attributes can be accessed from inside semantic actions.

First of all, let us revisit the concept of semantic actions.

Semantic actions may be attached to any point in the grammar specification. These actions are C++ functions or function objects that are called at certain points during the process of parsing or output generation. In Qi a semantic action is called whenever a part of the parser successfully recognizes a portion of the input. In Karma they are called whenever a part of the generator is about to be invoked.

Let us assume you have a component C, and a C++ function or function object F, then you can make the component call F by attaching it to the component:

C[F]

The expression above links F to the component C.

Even if it possible to utilize almost any separate C++ function as a semantic action, it is a lot simpler to write semantic actions using Boost.Phoenix. Phoenix enables to write semantic actions in a very straight forward way. The code to be executed is placed inline directly into the square brackets. Here is a very simple Qi example:

qi::int_[std::cout << qi::_1]

Here qi::_1 is a predefined (Phoenix) placeholder referencing the attribute of the parser the semantic actions has been attached to (the qi::int_). As the semantic action is invoked after the parser succeeded, the attribute will be set to the matched value. We can do something very similar in Karma as well:

int i = 3;
karma::int_[karma::_1 = phoenix::ref(i)] // will emit: 3

Here we assign the value we want to be emitted to the (Phoenix) placeholder karma::_1 (phoenix::ref() is almost identical to boost::ref(), except that it integrates nicely with any Phoenix expression). This works as semantic actions in Karma are invoked before the generator does its job, allowing to pass the current value of the variable ‘i’ as the attribute for the generator karma::int_.

A lesser known feature of Spirit’s semantic actions is that they can be attached to arbitrary complex parser (or generator) expressions, which is particularly useful if the expression is a sequence. In this case predefined placeholders enable to access the attributes of the sequence elements separately:

(qi::int_ >> qi::double_)[std::cout << (qi::_1 + qi::_2)]

Here qi::_1 refers to the attribute of the component qi::int_, and qi::_2 refers to the attribute of qi::double_. This feature is very handy as the semantic action will be invoked only after both parser components succeeded making sure the results will be evaluated only if the input has been matched completely. This is something which did not work in Spirit.Classic. Many developers have been complaining about the lack of any related functionality.

Needless to say, similar constructs are available for Karma as well, even if this functionality is there not as important as in Qi.

Today’s question has been asked by @psicode: “What is the difference between the components created by the unary operators ‘!’ and ‘~’?”. As the semantics of those operators are slightly dissimilar in Qi and Karma, I will talk about them separately. I will write about the Qi operators today and about the corresponding Karma operators in one of the next installments.

Let us start with the commonality between the two operators: both negate whether the component they are being used with succeeds parsing. If the component ‘c’ succeeds, both compound constructs, ‘!c’ and ‘~c’ will fail, and similarly, if ‘c’ fails, the execution of components ‘!c’ and ‘~c’ will succeed.

The differences are more interesting. The unary Qi operator ‘~’ is applicable to character and character class parsers only. It negates the set of characters matched by the parser component it is attached to. Here are some examples:

The parsers created by the operator ‘~’ do not wrap the underlying parser. It rather modifies the behavior of the component it is attached to. This means there is no performance difference if compared to the plain character parsers.

The unary operator ‘!’ is creating a not-predicate parser. It can be attached to any (arbitrarily complex) parser expression. The not-predicate is a look-ahead matching parser trying to match the expression it is attached to. This is done without moving the current input position forward. In other words, the not-predicate does not consume any input . If the attached parser fails matching the overall not-predicate will succeed. In this case the parsing resumes at the same point where the not-predicate started matching. The following (slightly artificial) example will succeed matching a floating point number after making sure it is not a Boolean expression:

namespace qi = boost::spirit::qi;
std::string input("1.0");
std::string::const_iterator b = input.begin();
double result = 0;
qi::parse(b, input.end(), !qi::bool_ >> qi::double_, result);

Any parser created by the not-predicate is neutral in terms of attribute handling because it exposes unused_type as its attribute. As a consequence, parser components augmented with ‘!’ will never expose their attribute, and never will participate in any attribute propagation.

Spirit’s non-terminals can expose any attribute type, and the required type needs to be explicitly specified while declaring them. This is true for Qi parsers and Karma generators. The following example declares a rule exposing an (synthesized) attribute of type double (if you wonder why it uses the unusual function declaration syntax, i.e. double(), please read the article mentioned above):

namespace qi = boost::spirit::qi;
std::string input("1.0");
std::string::const_iterator b = input.begin();
double result = 0;
qi::rule<std::string::const_iterator, double()> r = qi::double_;
qi::parse(b, input.end(), r, result);

The left hand side’s attribute (the result passed in by the user) is directly handed over to the right hand side of the rule. The parser created by qi::double_ will put its result into the very same double instance as passed in from the outside. No additional copies are created. We call this behavior auto attribute propagation. For rules it is enabled by default as long as no semantic actions are attached to the right hand side’s expression.

If you want to enforce auto attribute propagation even if the right hand side has semantic actions, you need to employ the ‘%=’ syntax as shown in the next Karma example:

namespace karma = boost::spirit::karma;
typedef std::back_insert_iterator<std::string> output_iterator;
std::string output;
output_iterator sink(output);
karma::rule<output_iterator, double()> r;
r %= karma::double_[++karma::_1];
karma::generate(sink, r, 1.0);    // will emit: 2.0

This code will increment the attribute (the 1.0)  before emitting the result to the output iterator (remember, semantic actions in Karma are called before invoking the related generator). Note, we incremented the attribute of the generator karma::double_, not the left hand side’s attribute (which would have been ++karma::_val). This works as we enforced the auto attribute propagation using the ‘%=’.

There is one more important thing to remember: regardless of how the auto attribute propagation is enabled, either based on the default behavior of ‘=’ or by enforcing it using ‘%=’, the attribute types of the rule and the right hand side must be compatible. It is not possible to define Spirit’s ‘attribute compatibility’ in a short sentence, so we leave the topic for another day. But in the simplest case it means the attributes have to be convertible. In Qi the right hand side’s attribute must at least be convertible to the rule’s attribute, while in Karma the opposite needs to be true: the rule’s attribute should at least be convertible to the right hand side’s attribute.