{"id":626,"date":"2009-11-27T10:31:58","date_gmt":"2009-11-27T18:31:58","guid":{"rendered":"http:\/\/boost-spirit.com\/home\/?page_id=626"},"modified":"2010-11-13T06:24:15","modified_gmt":"2010-11-13T14:24:15","slug":"generate-escaped-string-output-using-spirit-karma","status":"publish","type":"page","link":"http:\/\/boost-spirit.com\/home\/articles\/karma-examples\/generate-escaped-string-output-using-spirit-karma\/","title":{"rendered":"Generate Escaped String Output Using Spirit.Karma"},"content":{"rendered":"

This is another article in the series of \u201cHow To\u2019s\u201d providing you with shrink wrapped grammars directly usable in any project. I\u2019m going to describe a Karma<\/em> grammar you can use to generate output for quoted strings, where all contained special characters are properly escaped.<\/p>\n

The previous installment of this series (see here<\/a>) is already giving a high level overview of Karma<\/em>. This allows us to skip any related introductions and to start right away. The purpose of the escaped_string<\/span> grammar is to generate output for any given character sequence while enclosing it in quotes (i.e. \u2018\\\u201d\u2019<\/span> or \u2018\\\u2019\u2019<\/span>) and making sure all relevant characters get escaped by pre-pending a backslash (i.e. \u2018\\n\u2019<\/span> will be generated as \u201c\\\\n\u201d<\/span>, etc.). Let\u2019s start with the reverse Parsing Expression Grammar (PEG) for this:<\/span><\/p>\n

esc_str \u2192 '"' (esc_char \/ . \/ "\\\\x" hex)* '"'\r\nesc_char \u2192 &'\\a' "\\\\a" \/ &'\\b' "\\\\b" \/ &'\\f' "\\\\f" \/\r\n           &'\\n' "\\\\n" \/ &'\\r' "\\\\r" \/ &'\\t' "\\\\t" \/\r\n           &'\\v' "\\\\v" \/ &'\\\\' "\\\\\\\\" \/\r\n           &'\\'' "\\\\\\'" \/ &'"' "\\\\\\""\r\n<\/pre>\n
Any escaped string (esc_str<\/span>) starts and ends with the quoting character (in this case \u2018\u201d\u2019<\/span>) and all characters of the sequence are printed either as an escaped character (esc_char<\/span>) or a printable character or a \u201c\\\\x\u201d<\/span> followed by the hexadecimal representation of the corresponding character code. The esc_char will handle any of the listed character codes by generating a backslash followed by the corresponding C-style encoding. Each of the listed alternatives (such as &\u2018\\a\u2019 \u201c\\\\a\u201d<\/span>) reads as: if the character has the code \u2018\\a\u2019<\/span> print it as \u201c\\\\a\u201d<\/span> (a backslash followed by \u2018a\u2019<\/span>).<\/p>\n
We have seen this before: converting the PEG into a Spirit grammar is a simple and formal step (we discussed that briefly here<\/a>):<\/p>\n
\r\nesc_str = '"' << *(esc_char | karma::print | "\\\\x" << karma::hex) << '"';\r\nesc_char.add('\\a', "\\\\a")('\\b', "\\\\b")('\\f', "\\\\f")('\\n', "\\\\n")\r\n            ('\\r', "\\\\r")('\\t', "\\\\t")('\\v', "\\\\v")('\\\\', "\\\\\\\\")\r\n            ('\\'', "\\\\\\'")('"', "\\\\\\"");\r\n<\/pre>\n
Note we use predefined Karma<\/em> facilities here, replacing a straight reverse PEG translation. We could have defined the rule esc_char<\/span> in a very similar way as done in the reverse PEG notation above, but decided to use a predefined Karma<\/em> component, the karma::symbols<><\/a><\/span>, instead. This gives us a nice way of mapping the special characters to their C-style representation. Conveniently, the symbols generator fails to generate anything if the supplied character is not contained in the symbols table. The karma::print<\/a><\/span> is a primitive generator which succeeds generating output for characters satisfying std::isprint()<\/span><\/a> while failing for all others. The karma::hex<\/a> is a predefined numeric generator emitting its attribute in hexadecimal integer representation.\u00a0 <\/span><\/p>\n
Generally, Karma<\/em> generators have the ability to fail generating if some of their preconditions are not met. This allows to use them in alternatives (as shown above), forcing to try the next alternative if the current generator doesn\u2019t handle the output. In the example above a specific character is first tried to be handled by the symbols generator. If the character is not in the list of those to be handled by the symbols generator, the next alternative (karma::print<\/a><\/span>) is tried. If the character does not satisfy std::isprint()<\/a><\/span> (resulting in the karma::print<\/span> primitive to fail) we use the last alternative as a catch all fallback and emit the character\u2019s hexadecimal representation.<\/p>\n
The overall rule esc_str<\/span> reads as: \u2018generate any supplied string embedded inside quotes (\u2018\u201d\u2019<\/span>), convert all special characters into a sequence of a backslash and the corresponding C-style encoding and represent all non-printable characters in C-style hexadecimal character encoding\u2019.<\/p>\n
Now, as we have the grammar, the next step is to figure out the proper attribute types for the rules. As written before<\/a>, attributes in Qi<\/em> are the types and values we get as the result of converting the input. Attributes in Karma<\/em> are the types and values we want to generate output from. For our example the choice is straightforward and already implied by the goal. If we assume narrow character representation we get:<\/p>\n
\r\nkarma::rule<OutputIterator, std::string()> esc_str;\r\nkarma::symbol<char, char const*> esc_char;\r\n<\/pre>\n
We use karma::rule<><\/a><\/span> as the non-terminal for storing the output format for the esc_str<\/span>. As this is the top level rule we assume std::string<\/span> to be its attribute. As alluded to earlier, esc_char<\/span> is a karma::symbols<><\/a><\/span> instance using one character in the string as the key while storing the C-style representation of that character as its value. Two of the generator alternatives inside the Kleene Star expose a single char as their attribute. The third alternative, the (“\\\\x” << karma::hex<\/span>) exposes an int<\/span> as its attribute, but this is perfectly compatible with a single character. This allows the std::string<\/span> to be naturally split into single characters, one at a time.<\/p>\n
Essentially we are done. That was not too hard, was it?<\/p>\n
But before we write the (reusable) grammar and demonstrate how it needs to be called I thought to make things a bit more interesting. So far our output format is bound to generate strings quoted using the double quote character (\u2018\u201d\u2019<\/span>). Wouldn\u2019t it be nice to be able to customize the quoting character as well? This is interesting for instance if you need to generate Python style strings which come in 3 flavors: quoted with \u2018\u201d\u2019<\/span>, \u201c\u2019\u201d<\/span>, or \u201c\u2019\u2019\u2019\u201d<\/span>. So I decided to introduce another features provided by Spirit\u2019s<\/em> non-terminals: inherited attributes. Non-terminals in recursive descent parsers and generators can be seen as being very similar to functions. Parsers return a value, their synthesized<\/em> attribute, while generators require a special consumed<\/em> attribute. Both optionally may take arguments, their inherited<\/em> attributes. Spirit<\/em> uses the function declaration syntax in order to specify all attributes in a very compact form. So we make the grammar customizable by \u2018passing\u2019 the quoting character sequence as an inherited attribute. We modify the rule esc_str<\/span> to expect a single inherited attribute (the quoting sequence) as a plain char const*<\/span>, and change the rule definition to use this attribute:<\/p>\n
\r\nkarma::rule<OutputIterator, std::string(char const*)> esc_str =\r\n        karma::lit(karma::_r1)\r\n    << *(esc_char | karma::print | "\\\\x" << karma::hex)\r\n    <<  karma::lit(karma::_r1);\r\n<\/pre>\n
The predefined primitive karma::lit<\/span> emits its argument as a literal and karma::_r1<\/span> is a special predefined placeholder expression referring to the first inherited attribute of the rule on the left hand side of the expression.<\/p>\n
Now, as we have all the required pieces in place I\u2019ll show you how to wrap everything into a karma::grammar<\/span> to make it reusable:<\/p>\n
\r\ntemplate <typename OutputIterator>\r\nstruct escaped_string\r\n  : karma::grammar<OutputIterator, std::string(char const*)>\r\n{\r\n    escaped_string()\r\n      : escaped_string::base_type(esc_str)\r\n    {\r\n        esc_char.add('\\a', "\\\\a")('\\b', "\\\\b")('\\f', "\\\\f")('\\n', "\\\\n")\r\n                    ('\\r', "\\\\r")('\\t', "\\\\t")('\\v', "\\\\v")('\\\\', "\\\\\\\\")\r\n                    ('\\'', "\\\\\\'")('"', "\\\\\\"")\r\n            ;\r\n        esc_str =   karma::lit(karma::_r1)\r\n                << *(esc_char | karma::print | "\\\\x" << karma::hex)\r\n                <<  karma::lit(karma::_r1)\r\n            ;\r\n    }\r\n    karma::rule<OutputIterator, std::string(char const*)> esc_str;\r\n    karma::symbols<char, char const*> esc_char;\r\n};\r\n<\/pre>\n
The derivation from Karma\u2019s<\/em> grammar type converts the escaped_string<\/span> type into a generator. Its member rules define a grammar which makes it usable for emitting quoted strings using arbitrary quoting characters. The base class constructor gets passed the esc_str rule, which is the top most rule of the grammar to be executed when the grammar is invoked. The type escaped_string<\/span> has a template parameter allowing to utilize this grammar with arbitrary output iterator types.<\/p>\n
The last missing code piece shows how to invoke the newly created generator.<\/p>\n
\r\ntypedef std::back_insert_iterator<std::string> sink_type;\r\n\r\nstd::string generated;\r\nsink_type sink(generated);\r\n\r\nstd::string str("string to escape: \\n\\r\\t\\"'\\x19");\r\nchar const* quote = "\u2019\u2019\u2019";\r\n\r\nclient::escaped_string<sink_type> g;\r\nkarma::generate(sink, g(quote), str);\r\n    \/\/ this will emit: \u2019\u2019\u2019string to escape: \\n\\r\\t\\"\\'\\x19\u2019\u2019\u2019<\/a>\r\n<\/pre>\n
The function karma::generate()<\/span> is another of Spirit\u2019s main API functions. In the simplest case it takes an output iterator representing the output target to send the output to (sink)<\/span>, an instance of the generator to invoke (g)<\/span>, and the attribute instance holding the data (v)<\/span>. We pass the quoting character sequence (the \u201c\\\u201d\u201d<\/span>) as a inherited attribute while invoking the grammar. This function executes the actual generator operation and returns true<\/span> if it was successful.<\/p>\n
If you want to try out this example for yourself, the complete source code is available from the Boost<\/a> SVN here<\/a>. In the future this example will be distributed as part of the Spirit distribution, but for now it lives in the SVN only. Additionally, the karma::symbols<><\/a><\/span> generator is not yet part of the latest released code (Spirit V2.1). You either need to checkout the current version of Spirit from the SVN or download the related file here<\/a>. In this case you need to explicitly include it into the example in order to be able to compile the code.<\/p>\n
Share this:<\/h3>