{"id":1018,"date":"2010-03-05T08:03:47","date_gmt":"2010-03-05T16:03:47","guid":{"rendered":"http:\/\/boost-spirit.com\/home\/?page_id=1018"},"modified":"2010-03-06T09:29:04","modified_gmt":"2010-03-06T17:29:04","slug":"tracking-the-input-position-while-parsing","status":"publish","type":"page","link":"http:\/\/boost-spirit.com\/home\/articles\/qi-example\/tracking-the-input-position-while-parsing\/","title":{"rendered":"Tracking the Input Position While Parsing"},"content":{"rendered":"

The following example is about tracking the parsing position with Spirit<\/em> V2. This is useful for generating error messages which tell the user exactly where an error has occurred. We also show how to use Spirit<\/em> V2 to parse from an input stream without first reading the whole stream into a std::string<\/span>.<\/p>\n

In our example, we want to parse a list of real numbers and return them in a std::vector<\/span> and throw an exception if an error occurs. We will start with a naive way of achieving our goal and then improve the program in two steps. This naive way could be the following small program. It provides basic and not very helpful error handling and scales bad for large inputs:<\/p>\n

\r\n#include <vector>\r\n#include <istream>\r\n#include <sstream>\r\n#include <iostream>\r\n#include <boost\/include\/spirit\/qi.hpp>\r\n\r\nnamespace qi = boost::spirit::qi;\r\nnamespace ascii = boost::spirit::ascii;\r\n\r\n\/\/ parse list of doubles from input stream\r\n\/\/ throw exception (perhaps including filename) on errors\r\nstd::vector<double>\r\n  parse(std::istream& input, const std::string& filename);\r\n\r\n\/\/ main function\r\nint main(int, char**)\r\n{\r\n  try\r\n  {\r\n    parse(std::cin, "STDIN");\r\n  }\r\n  catch(const std::exception& e)\r\n  {\r\n    std::cerr << "Exception: " << e.what() << std::endl;\r\n    return -1;\r\n  }\r\n  return 0;\r\n}\r\n\r\n\/\/ implementation\r\nstd::vector<double>\r\n  parse(std::istream& input, const std::string& filename)\r\n{\r\n  \/\/ get input into string\r\n  std::ostringstream buf;\r\n  buf << input.rdbuf();\r\n  std::string str = buf.str();\r\n\r\n  \/\/ prepare iterators\r\n  std::string::iterator first = str.begin();\r\n  std::string::iterator last = str.end();\r\n\r\n  \/\/ parse\r\n  std::vector<double> output;\r\n  bool r = qi::phrase_parse(\r\n    \/\/ iterators over input\r\n    first, last,\r\n    \/\/ recognize list of doubles\r\n    qi::double_ >> *(',' >> qi::double_) >> qi::eoi,\r\n    \/\/ comment skipper\r\n    ascii::space | '#' >> *(ascii::char_ - qi::eol) >> qi::eol,\r\n    \/\/ store into this object\r\n    output);\r\n\r\n  \/\/ error detection\r\n  if(!r || first != last)\r\n    throw std::runtime_error("parse error in " + filename);\r\n\r\n  \/\/ return result\r\n  return output;\r\n}\r\n<\/pre>\n
On the following input, parsing succeeds (and there is no output):<\/p>\n
\r\n# good input\r\n1.3,2.3, 5\r\n<\/pre>\n
With the following faulty input<\/p>\n
\r\n# bad input\r\n123,42.0,a,1.4\r\n<\/pre>\n
we get an exception:<\/p>\n
\r\nException: parse error in STDIN\r\n<\/pre>\n
If you want to try out this version of the program you may download it here: naive_parsing.cpp<\/a>, the two input files are goodinput.txt<\/a> and badinput.txt<\/a>.<\/p>\n
To make the above scalable for large files, we now change the example to parse from an STL stream, and not buffer everything into a std::string<\/span>. For that, we need the boost::spirit::multi_pass<\/span> class which wraps a stream iterator and exposes an iterator usable by Qi<\/em> parsers. The reason we need this adaptor is, that a std::istream_iterator<\/span> is an input iterator, and Qi<\/em> requires a forward iterator which allows certain backtracking on the input.<\/p>\n
So we include the following file<\/p>\n
\r\n#include <boost\/include\/spirit\/support_multi_pass.hpp>\r\n<\/pre>\n
and replace<\/p>\n
\r\n   \/\/ get input into string\r\n   std::ostringstream buf;\r\n   buf << input.rdbuf();\r\n   std::string str = buf.str();\r\n\r\n   \/\/ prepare iterators\r\n   std::string::iterator first = str.begin();\r\n   std::string::iterator last = str.end();\r\n<\/pre>\n
by<\/p>\n
\r\n   \/\/ iterate over stream input\r\n   typedef std::istreambuf_iterator<char> base_iterator_type;\r\n   base_iterator_type in_begin(input);\r\n\r\n   \/\/ convert input iterator to forward iterator, usable by spirit parser\r\n   typedef boost::spirit::multi_pass<base_iterator_type> forward_iterator_type;\r\n   forward_iterator_type fwd_begin =\r\n       boost::spirit::make_default_multi_pass(in_begin);\r\n   forward_iterator_type fwd_end;\r\n<\/pre>\n
Now we only need to rename all occurrences of first<\/span> to fwd_begin<\/span> and all occurrences of last<\/span> to fwd_end<\/span> and we are done! (The complete second program can be downloaded from here: stream_iterator_parsing.cpp<\/a>.)<\/p>\n
This new program has the same behavior as the first one, but it does not need to buffer the whole input.<\/p>\n
The final task is to get more useful error messages if a parsing problem occurs. To that end we wrap the iterator yet another time, using the boost::spirit::classic::position_iterator2<\/span> class which records line number, column number, and filename of the parsed input.<\/p>\n
First we need one of additional include file:<\/p>\n
\r\n#include <boost\/include\/spirit\/classic_position_iterator.hpp>\r\nnamespace classic = boost::spirit::classic;\r\n<\/pre>\n
This file in fact provides several possibilities for position iterators: boost::spirit::classic::position_iterator<\/span> is a bit more efficient than boost::spirit::classic::position_iterator2<\/span>, but the latter allows to extract the whole line of input from an iterator in that line (and we will use this feature). Additionally, one can specify a ‘position iterator policy’ to only store line numbers and not column numbers. This increases efficiency but might decrease usefulness of the position information.<\/p>\n
Wrapping the forward iterator with the position iterator is not difficult, we create another typedef and initialize the iterators as follows:<\/p>\n
\r\n   \/\/ wrap forward iterator with position iterator, to record the position\r\n   typedef classic::position_iterator2<forward_iterator_type>\r\n     pos_iterator_type;\r\n   pos_iterator_type position_begin(fwd_begin, fwd_end, filename);\r\n   pos_iterator_type position_end;\r\n<\/pre>\n
Hint: make sure there are no brackets after position_end<\/span> – this often happens and creates compile errors which are difficult to find: with brackets you do not declare a variable of type pos_iterator_type<\/span>, but a function returning pos_iterator_type<\/span>!<\/p>\n
We now use the new position iterator for parsing, but the grammar needs to be adjusted a bit. We will replace the following fragment:<\/p>\n
\r\n  \/\/ parse\r\n  bool r = qi::phrase_parse(\r\n    \/\/ iterators over input\r\n    fwd_begin, fwd_end,\r\n    \/\/ recognize list of doubles\r\n    qi::double_ >> *(',' >> qi::double_) >> qi::eoi,\r\n    \/\/ comment skipper\r\n    ascii::space | '#' >> *(ascii::char_ - qi::eol) >> qi::eol,\r\n    \/\/ doubles are stored into this object\r\n    output);\r\n\r\n  \/\/ error detection\r\n  if (!r || fwd_begin != fwd_end)\r\n    throw std::runtime_error("parse error in " + filename);\r\n<\/pre>\n
In our new code, we first call qi::phrase_parse<\/span> as before, but this time we disallow backtracking by using ><\/span> instead of >><\/span> for certain parser sequence operators in the grammar: if the parser would backtrack over such a sequence, it throws a qi::expectation_failure<pos_iterator_type><\/span> exception.<\/p>\n
\r\n   try\r\n   {\r\n     qi::phrase_parse(\r\n       \/\/ iterators over input\r\n       position_begin, position_end,\r\n       \/\/ recognize list of doubles\r\n       qi::double_ > *(',' > qi::double_) >> qi::eoi,\r\n       \/\/ comment skipper\r\n       ascii::space | '#' >> *(ascii::char_ - qi::eol) >> qi::eol,\r\n       \/\/ store into this object\r\n       output);\r\n   }\r\n   catch(const qi::expectation_failure<pos_iterator_type>&amp; e)\r\n   {\r\n   }\r\n<\/pre>\n
To process the caught exception we add the following code:<\/p>\n
\r\n     const classic::file_position_base<std::string>& pos =\r\n         e.first.get_position();\r\n     std::stringstream msg;\r\n     msg <<\r\n         "parse error at file " << pos.file <<\r\n         " line " << pos.line << " column " << pos.column << std::endl <<\r\n         "'" << e.first.get_currentline() << "'" << std::endl <<\r\n         std::setw(pos.column) << " " << "^- here";\r\n     throw std::runtime_error(msg.str());\r\n<\/pre>\n
(The complete final program can be downloaded from here: stream_iterator_errorposition_parsing.cpp<\/a>.)<\/p>\n
This code creates an error message from the filename, the line and column number and the currently parsed line.<\/p>\n
The faulty input now creates the following useful error message:<\/p>\n
\r\nException: parse error at file STDIN line 1 column 10\r\n'123,42.0,a,1.4'\r\n          ^- here\r\n<\/pre>\n
The programs of this example can be downloaded here: naive_parsing.cpp<\/a>, stream_iterator_parsing.cpp<\/a>, and stream_iterator_errorposition_parsing.cpp<\/a>. The two input files are goodinput.txt<\/a> and badinput.txt<\/a>.<\/p>\n
Share this:<\/h3>