{"id":1183,"date":"2010-11-13T12:59:30","date_gmt":"2010-11-13T20:59:30","guid":{"rendered":"http:\/\/boost-spirit.com\/home\/"},"modified":"2010-11-13T13:25:38","modified_gmt":"2010-11-13T21:25:38","slug":"parsing-escaped-string-input-using-spirit-qi","status":"publish","type":"page","link":"http:\/\/boost-spirit.com\/home\/articles\/qi-example\/parsing-escaped-string-input-using-spirit-qi\/","title":{"rendered":"Parsing Escaped String Input Using Spirit.Qi"},"content":{"rendered":"

In a previous article<\/a> you’ve learned how to generated escaped strings, today we’re going to do the reverse. We’re going to describe a Qi<\/em> grammar you can use to parse quoted strings in which special characters are escaped.<\/p>\n

The purpose of the unescaped_string<\/span> grammar is to removed the enclosing quotes and un-escape the characters which were previously escaped (i.e. convert “\\\\n”<\/span> back to ‘\\n’<\/span> again). Unsurprisingly, the Parsing Expression Grammar (PEG) is quite like the one in the Karma article:<\/p>\n

\r\nunesc_str = '"' (unesc_char \/ . \/ "\\\\x" hex)* '"'\r\nunesc_char = &"\\\\a" '\\a' \/ &"\\\\b" '\\b' \/ &"\\\\f" '\\f' \/\r\n             &"\\\\n" '\\n' \/ &"\\\\r" '\\r' \/ &"\\\\t" '\\t' \/\r\n             &"\\\\v" '\\v' \/ &"\\\\\\\\" '\\\\' \/\r\n             &"\\\\\\'" '\\'' \/ &"\\\\\\"" '"'\r\n<\/pre>\n
An escaped strings starts and ends with the quoting character (in this case ‘”‘<\/span>) and all characters of the sequence are read either as an escaped character (unesc_char<\/span>), a printable character or a “\\\\x”<\/span> followed by the hexadecimal representation of a character code. The unesc_char will handle any of the listed character codes by parsing a backslash followed by the corresponding C-style encoding. Like in the Karma example, each of the listed alternatives (such as &”\\\\a” ‘\\a’<\/span>) reads as: if the input being parsed starts with “\\\\a”<\/span>, return ‘\\a’<\/span>.<\/p>\n
Now let’s convert this to a Spirit<\/em> grammar:<\/p>\n
\r\nunesc_str = '"' >> *(unesc_char | qi::alnum | "\\\\x" >> qi::hex) >> '"';\r\nunesc_char.add("\\\\a", '\\a')("\\\\b", '\\b')("\\\\f", '\\f')("\\\\n", '\\n')\r\n              ("\\\\r", '\\r')("\\\\t", '\\t')("\\\\v", '\\v')("\\\\\\\\", '\\\\')\r\n              ("\\\\\\'", '\\'')("\\\\\\"", '\\"');\r\n<\/pre>\n
Just like in the Karma<\/em> example we’re using predefined facilities, but in this case Qi<\/em> facilities. Where we were previously using karma::symbols<><\/span> to map special characters to their C-style representation, we’re doing the reverse now: mapping C-style representations to the special characters using qi::symbols<><\/span>. This too will conveniently fail if the representation is not in the symbol table. The next parser that will be tried is qi::alnum<\/span>, which successfully accepts an alphanumeric character and fails on all other input. At this point you might be wondering why I’m not using qi::print, like we were using karma::print<\/span>, I’ll come back to this later. Last but not least we’ll try converting a hexadecimal character representation to the actual character code if it’s prefixed with a “\\\\x”<\/span>.<\/p>\n
Generally, what is true for Karma<\/em> generators holds for their Qi<\/em> parser counterparts. They too have the ability to fail parsing if their preconditions are not met, allowing them to be used in alternatives just like the Karma<\/em> generators. When one fails, the next alternative is tried until we’re out of alternatives and have to fail overall. The next alternative here is qi::alnum<\/span>, which as we said accepts an alphanumeric character and fails otherwise.<\/p>\n
Since we now have the grammar, the next step is to figure out which attributes to use for the rules. The attributes in Qi<\/em> are the types and values we gat as the result of converting the input, thus it’s quite straightforward:<\/p>\n
\r\nqi::rule<InputIterator, std::string()> unesc_str;\r\nqi::symbols<char const, char const> unesc_char;\r\n<\/pre>\n
Note that we’re assuming narrow character representation here, for wide character representation one would have used std::wstring<\/span> and wchar_t const<\/span>.<\/p>\n
Akin to the Karma<\/em> example, we use the Qi<\/em> counterpart qi::rule<><\/span> as the non-terminal for storing the parsed data for the unesc_str<\/span>, for which we again assume std::string<\/span> to be its attribute. Again two of the parser alternatives inside the Kleene Star expose a single character as their attribute, the other (“\\\\x” >> qi::hex<\/span>) exposing an unsigned int<\/span> which can be implicitly casted to a char. If we strictly followed the rules we’ll see the rule actually exposes a vector<char><\/span>, but this is compatible with a std::string<\/span>. Secondly the C-style representations with their character matches are stored in a qi::symbols<><\/span> instance. Now you might wonder why it’s a qi::symbols<char const, char const><\/span> and not qi::symbols<char const *, char const><\/span> since the Karma<\/em> example was using karma<char const, char const *><\/span> the first parameter is the character type of the strings stored, not the type of the strings.<\/p>\n
That’s all there’s to it!<\/p>\n
Since the Karma<\/em> example allows you to specify the quoting character, we’re going to do so too, and I’ll use this opportunity to show just how similar Karma<\/em> and Qi<\/em> really are. Now here’s the Karma<\/em> code presented in the previous article:<\/p>\n
\r\nkarma::rule<OutputIterator, std::string(char const*)> esc_str =\r\n        karma::lit(karma::_r1)\r\n    << *(esc_char | karma::print | "\\\\x" << karma::hex)\r\n    <<  karma::lit(karma::_r1);\r\n<\/pre>\n
Since we’re parsing data instead of generating it, we should use >><\/span> instead of <<<\/span>, and obviously we’re using Qi<\/em> thus we need to use that namespace, which gives us:<\/p>\n
\r\nqi::rule<InputIterator, std::string(char const*)> unesc_str =\r\n        qi::lit(qi::_r1)\r\n    >> *(unesc_char | qi::alnum | "\\\\x" >> qi::hex)\r\n    >>  qi::lit(qi::_r1);\r\n<\/pre>\n
Easy huh?<\/p>\n
This leads us back to the reason we’re using qi::alnum<\/span> instead of qi::print<\/span>, there are two reasons:<\/p>\n
    \n
  1. The alternatives are are tried in the order they are in, thus if we had *(unesc_char | qi::print | “\\\\x” >> qi::hex)<\/span> the “\\\\x” >> qi::hex<\/span> option would never be tried since “\\\\x”<\/span> and hexadecimal characters are printable, and thus parsed using qi::print. Simply re-arranging the order solves this problem, thus we could use *(unesc_char | “\\\\x” >> qi::hex | qi::print)<\/span>.<\/li>\n
  2. The quoted character used in the example is “”'”<\/span>, which means we’re actually parsing as follows<\/li>\n<\/ol>\n
    \r\nunesc_str = qi::lit("'''")\r\n    >> *(unesc_char | "\\\\x" >> qi::hex | qi::print)\r\n    >>  qi::lit("'''");\r\n<\/pre>\n
    but the Kleene Star is greedy (an so is the unary +<\/span>, which means “one or more”), thus it would parse the closing “”'”<\/span> too! The solution here is not to use qi::print<\/span>, but (qi::print – “‘”<\/span>) which means “all printable characters, except “‘”<\/span>.<\/p>\n
    Putting this all together and wrapping it in a qi::grammar we get:<\/p>\n
    \r\ntemplate <typename InputIterator>\r\nstruct unescaped_string\r\n  : qi::grammar<InputIterator, std::string(char const*)>\r\n{\r\n    unescaped_string()\r\n      : unescaped_string::base_type(unesc_str)\r\n    {\r\n        unesc_char.add("\\\\a", '\\a')("\\\\b", '\\b')("\\\\f", '\\f')("\\\\n", '\\n')\r\n                      ("\\\\r", '\\r')("\\\\t", '\\t')("\\\\v", '\\v')\r\n                      ("\\\\\\\\", '\\\\')("\\\\\\'", '\\'')("\\\\\\"", '\\"')\r\n        ;\r\n\r\n        unesc_str = qi::lit(qi::_r1)\r\n            >> *(unesc_char | qi::alnum | "\\\\x" >> qi::hex)\r\n            >>  qi::lit(qi::_r1)\r\n        ;\r\n    }\r\n\r\n    qi::rule<InputIterator, std::string(char const*)> unesc_str;\r\n    qi::symbols<char const, char const> unesc_char;\r\n};\r\n<\/pre>\n
    which can be called as such:<\/p>\n
    \r\ntypedef std::string::const_iterator iterator_type;\r\n\r\nstd::string parsed;\r\nstd::string str("'''string to unescape:\\\\x20\\\\n\\\\r\\\\t\\\\\\"\\\\'\\\\x41'''");\r\nchar const* quote = "'''";\r\niterator_type iter = str.begin();\r\niterator_type end = str.end();\r\nclient::unescaped_string<iterator_type> p;\r\n\r\nqi::parse(iter, end, p(quote), parsed);\r\n\r\n    \/\/ this will result in: parsed == "string to unescape: \\n\\r\\t\\"\\'A"\r\n<\/pre>\n
    Which uses qi::parse<\/span>, one of Qi<\/em> its main API functions, similar to karma::generate<\/span> for generation. It takes the beginning and the end of the string to parse, an instance of the parser (p<\/span>) and an attribute instance to which the parsed data is saved (parsed<\/span>). We pass the quoting character sequence (the “”'”<\/span> in quote) as an inherited attribute when invoking the grammar. When successful qi::parse<\/span> returns true<\/span>, while it returns false<\/span> otherwise and one can compare iter<\/span> to end<\/span> to check if the complete input string has been parsed.<\/p>\n
    This leaves one minor enhancement, one can use parsed.reserve(str.length())<\/span> to reserve enough space in parsed<\/span> to fit the un-escaped string, which encompasses a small speed enhancement.<\/p>\n
    If you want to try out this example for yourself, the complete source code is available from the Boost<\/a> SVN here<\/a>.<\/p>\n
    Share this:<\/h3>