Welcome to Spirit

Spirit is a set of C++ libraries for parsing and output generation implemented as Domain Specific Embedded Languages (DSEL) using Expression templates and Template Meta-Programming. The Spirit libraries enable a target grammar to be written exclusively in C++. Inline grammar specifications can mix freely with other C++ code and, thanks to the generative power of C++ templates, are immediately executable.

Spirit is part of Boost Libraries, a peer-reviewed, open collaborative development effort.

Feb ’13 23

Oh, in case anyone wants to follow the development of X3, here are the Github links:

https://github.com/djowel/spirit_x3
https://github.com/djowel/spirit_x3.git
git@github.com:djowel/spirit_x3.git

The full attribute mechanism works + basic C++lambda support. I’ve ported one example (calc4.cpp) which parses to an AST. The AST remains the same. Only the way the grammars are written had to change.

For a teaser: here’s GCC times:

SpiritX3: TOTAL :   4.27 secs
Spirit2:  TOTAL :  10.00 secs

Even faster at CT than the first Spirit3 attempt. Runtime speed? I expect it to be faster than Spirit-2. The rules have changed (pun intentional). Now, there’s no longer the need for virtual functions and auto is used extensively. I expect code size to be smaller too because the compiler can generate more efficient code.

Here’s the calculator grammar (based on calc3.cpp):

///////////////////////////////////////////////////////////////////////////////
//  The calculator grammar
///////////////////////////////////////////////////////////////////////////////
namespace calculator_grammar
{
    using x3::uint_;
    using x3::char_;

    x3::rule<class expression, ast::program> const expression;
    x3::rule<class term, ast::program> const term;
    x3::rule<class factor, ast::operand> const factor;

    auto const expression_def =
        term
        >> *(   (char_('+') >> term)
            |   (char_('-') >> term)
            )
        ;

    auto const term_def =
        factor
        >> *(   (char_('*') >> factor)
            |   (char_('/') >> factor)
            )
        ;

    auto const factor_def =
            uint_
        |   '(' >> expression >> ')'
        |   (char_('-') >> factor)
        |   (char_('+') >> factor)
        ;

    auto const calculator = x3::grammar(
            expression = expression_def,
            term = term_def,
            factor = factor_def
        );
}

using calculator_grammar::calculator;
Feb ’13 23

My proposal “Inside Spirit X3. Redesigning Boost.Spirit for C++11″ has been accepted for presentation at the C++ Now! 2013 in Aspen Colorado. So after a 2 year hiatus, I’m back to Aspen.

Here’s the abstract:

Inside Spirit X3.Redesigning Boost.Spirit for C++11
Joel de Guzman, 2013

The hugely successful BoostCon ’07 ’08, ’09 and ’10 Spirit talks provided walk-through presentations and tutorials on how to use Spirit. This, time, I propose a presentation that will focus on the design and implementation of Spirit. But to add more to the thrill, I will present a major redesign of Spirit from the ground up, taking advantage of the new C++11 features. One important goal of this experimental version of Spirit (named X3) is to bring back the elegant simplicity of “Classic” Spirit, which was somehow lost with the complexity of Spirit-2 primarily due to the lack of important language features that are just starting to appear in C++ compilers. In this 90-minute presentation, I would like to get down and dirty with Modern C++11 code, and along the way, share my experience as well as expose some of C++11′s shortcomings and my wishes for C++1y.

Hope to see you there!

Aug ’11 10

We’ve had a Spirit IRC channel for a few months now. It started out as a semi-private channel. Well, we made it officially public now. You can get Spirit support via the ##spirit or #boost IRC channels at freenode, where you will often find knowledgeable people willing to help with your Spirit questions. The #boost channel is for anything Boost related, including Spirit. The ##spirit channel is solely for Spirit and Spirit related libraries (Fusion, Phoenix and Wave).

See you there!

Jul ’11 24

Spirit is 10 years old!

It’s hard to pinpoint exactly the birthday of Spirit. Looking back, Spirit 1.0 which was uploaded to SourceForge in July 27, 2001 contains this comment in its main header file:

8/28/1999    Original Implementation [ run time polymorphic version ] (JDG)
4/30/2001    Template meta-programming implementation (JDG)
5/13/2001    Major redesign using iterators (JDG)
5/26/2001    Port to G++3.0 and BCC 5.5.1 thanks to Vladimir Prus
5/27/2001    Bug fixes in Difference and Xor classes (JDG)
5/30/2001    Added Iterators (JDG)

Continue reading »

Jul ’11 23

Mike Lewis posted a marvelous experience report dubbed ‘Optimizing Boost Spirit – Blazing fast AST generation using boost::spirit’. He describes how he took an old compiler for the Epoch programming language (which was based on Spirit.Classic) and tuned it for performance using Spirit.Qi and Spirit.Lex. His results are exceptional, he got roughly a thousand fold speedup compared to the old version. The complete code for his compiler can be downloaded from here.

Continue reading »

Tagged with:
Jul ’11 12

Spirit V2.5 is now available as part of the recently released Boost V1.47. I suggest you look at the What’s New documentation page for a list of things changed. This is a very important release for Spirit, mostly in the area of feature consolidation and less so by adding new functionality. It brings a lot of unification and quite some speedup in the area of attribute handling, both for parsers (Qi) and generators (Karma). Most of the newly added features are in the area of unification of the overall user experience as well. The things added to the Lexer should finally resolve some long standing requests. Most importantly, we are very excited about having added full compatibility with the newly released Phoenix V3 library.

Generally, all changes are supposed to be fully backwards compatible. If you run into problems with your existing code, please tell us by leaving a comment or by sending a message to the Spirit mailing list (as described on our Support page).

Tagged with:
Jun ’11 12

Here’s another cool Spirit talk. This time from Bryce Alexander Lelbach:

http://blip.tv/boostcon/ast-construction-with-the-universal-tree-5266608

utree, a recent addition to the Boost.Spirit codebase, is a generic data structure designed to represent abstract syntax trees. Bindings to Qi and Karma make utree a powerful tool for parser and generator development with Boost.Spirit. This presentation would demonstrate the usage of utree and Spirit to build and manipulate an abstract syntax tree for four parsing/generating use cases: XML, symbolic expressions, JSON, and C-like source code. The details of utree’s integration with Spirit and their implications for writing utree-centric Spirit parsers and generators would be discussed. Additionally, design patterns for compiling a utree AST to other internal representations (DOM tree for XML, function tree for a Scheme-like language for symbolic expressions, associative arrays for JSON objects, a simple VM bytecode for mini-C source code) would be covered.

Here are the slides:

https://github.com/boostcon/2011_presentations/raw/master/fri/utree_talk.pdf

 

Jun ’11 08

Here’s another BoostCon video uploaded by Marshall Clow. This one is about Phoenix V3, by Hartmut Kaiser:

http://blip.tv/boostcon/phoenix-v3-an-overview-5250984

The slides for this talk can be found here: https://github.com/boostcon/2011_presentations/blob/master/mon/phoenix_v3.pdf?raw=true.

Phoenix will be the next generation of creating unnamed, inlined polymorphic function objects. With V3 we combine the functionality of Boost.Bind and Boost.Lambda, and arranges it into a new library. By writing this new library, we were able to fix some limitations of the aforementioned libraries without breaking backwardscompatibility. The purpose of the talk will be to outline the importance and elegance of functional programming (FP) in C++. The first part of the talk will give an introduction into the Domain Specific Embedded Language (DSEL) we defined with Phoenix. A DSEL is built with the help of regular C++ function and operator overloads. For Phoenix we defined such a language that emulates C++, to give potential users a low entry into the world of FP. While a lot of existing C++ code relies on higher order functions (better known as function objects), e.g. the C++ standard library use them as a way to let users customize operations in certain algorithms. We focus the second part of the talk on examples on how to use Phoenix instead of writing regular function objects and how to enable your legacy code to be used inside Phoenix expressions. However, Phoenix is more. Phoenix is equipped with a unique (in C++) mechanism to handle the expressions discussed in the previous sections as data. This allows us to handle Phoenix not in the C++ standard way but in any way you like. An overview of these mechanisms will be given in the last part of the talk to give potential users an insight on possible future applications that might evolve around Phoenix.

Jun ’11 08

This is the first time I missed attending BoostCon (May 15-20, 2011 – Aspen, Colorado). Fortunately, for us who were not able to attend, Marshall Clow uploaded some videos. Here’s one one that’s relevant to Spirit: “Spirit.Qi in the Real World”, by Robert Stewart. Watch the presentation here:

http://blip.tv/boostcon/spirit-qi-in-the-real-world-5254335

You can find the slides here: https://github.com/boostcon/2011_presentations/raw/master/tue/spirit_qi_in_the_real_world.pdf

Past sessions on Spirit have focused on introducing Spirit or showing extracts of real use, intermingled with tutorial highlights. Upon writing real Spirit.Qi parsers, however, one quickly discovers that “the devil is in the details.” There are special cases, tricks, and idioms that one must discover by trial and error or, perhaps, by following the Spirit mailing list, all of which take time and may not be convenient. In this session, we’ll walk through the development of a Spirit.Qi parser for printf()-style format strings. The result will be a replacement for printf() that is typesafe and efficient.

Tagged with:
Apr ’11 16

The keyword parser construct has recently been added to spirit’s repository (available in 1.47 or from svn) . Here’s a small introduction to help you get started using the keyword parsers.

Those of you familiar with the Nabialek trick will recognize it’s working under the hood. What you can achieve with the keywords parser can also be achieved with the Nabialek trick but not always as elegantly or as efficiently.

Continue reading »

Mar ’11 18

Somewhere over the past days we crossed a major milestone with Spirit’s website. Granted, it went by unnoticed, but we celebrated our 300,000 visitor since the launch of the new website in November 2009! As you can see from the Google analytics plot below, the average number of daily visitors stayed pretty constant over the last year, showing a slight increase lately.

I find it to be very encouraging that the average time people spend on the Spirit site is over 6 minutes. Compared to other websites (even purely technical ones), this number is huge. It is an indication that the contents we try to provide are useful and interesting to our visitors.

webstats

Continue reading »

Mar ’11 06

We (the spirit developers) are obviously advocates of FP (functional programming). We use as much FP as we can in the design of Spirit. Spirit is essentially a parser-combinator library in the likes of Haskell parser libraries such as Parsec. The similarities end however in how we deal with imperative constructs that are taboos in the FP world.

Read more here: Attributes: Functional vs Side-effects

Frank Dellaert
Feb ’11 28

When using expectation points, a parsing failure results in an exception that generically indicates the failure, but probably doesn’t explain the problem in the most meaningful way. It is possible to attach an error handler to react to the failed match in a more specialized way:

Continue reading »

Feb ’11 26

Frank Dellaert was asking a valid question on the Spirit mailing list where he pointed out some things he was not able to understand. His question clearly uncovered an inconsistency in Spirit’s API. This lead us to implement some minor additional feature, which in the end turned out to make more uniform the way semantic actions are handled.

Continue reading »

Tagged with:
Feb ’11 12

Narinder Claire asked a seemingly innocent question on the Spirit mailing list the other day. After starting to write an answer I realized that this question is not innocent at all as it touches the very fabric of Spirit: the rules of attribute handling. Many people have a hard time to properly understand what is going on in the nether regions of Spirit. More importantly, they have a hard time to understand why is Spirit implemented the way it is.

The new article Attribute Propagation and Attribute Compatibility not only answers Narinders questions but tries to explain those important concepts in more detail.

Tagged with:
preload preload preload