We got many questions how to parse data which has to be accessed using a standard input stream (usually a std::istream) without having to read all of the data into memory first.

The standard way of iterating over a stream would be to wrap it into a std::istream_iterator, but unfortunately this is not possible. Qi requires iterators to be at least forward iterators, while the std::istream_iterator is an input iterator only. To overcome this limitation Spirit V2.2 (the next version of Spirit, to be released with Boost V1.42) will implement a new iterator.

namespace boost { namespace spirit
{
    // this is functionally equivalent to std::istream_iterator
    template <typename Char, typename Traits = std::char_traits<Char> >
    struct basic_istream_iterator;

    // predefine specialization for 'char'
    typedef basic_istream_iterator<char> istream_iterator;
}}

This iterator is functionally equivalent to the std::istream_iterator except that it is a forward iterator you can utilize for your Qi parsing needs. Here is an example:

#include <boost/spirit/include/support_istream_iterator.hpp>

namespace spirit = boost::spirit;

// open file, disable skipping of whitespace
std::ifstream in("some_data_file");
in.unsetf(std::ios::skipws);

// wrap istream into iterator
spirit::istream_iterator begin(in);
spirit::istream_iterator end;

// use iterator to parse file data
spirit::qi::parse(begin, end, <...your grammar here...>);

This iterator is implemented on top of the multi_pass iterator framework, which is documented here. For those curious enough to have a peek: you can access the new iterator from Boost SVN here. But in order to use it with Boost prior to V1.42 you will need to download the whole iterators directory from SVN.