Re: prevent multipass iterators from consuming on incomplete parse

In the meantime have solved this issue using a workaround by parsing
directly from the streambuf rather than an istream upon it:

const char *original_begin = reinterpret_cast<const char
*>(n_streambuf.data().data());
const char *begin_ptr = original_begin;
const char *end_ptr = begin_ptr + n_streambuf.size();

const bool retval = qi::parse(begin_ptr, end_ptr, s_response_parser,
n_response);

if (retval) {
const std::size_t bytes_to_consume = begin_ptr - original_begin;
MOOSE_ASSERT(bytes_to_consume)

// We have parsed one message successfully. This means, begin_ptr
// should now point to where the msg ended. We consume the buffer to remove
it
n_streambuf.consume(bytes_to_consume);
}

Semantics is the same, parser is unchanged as well but the behavior is now
better. Performance seems much better as well, so I'm guessing I'm gonna
stick with that. Just wanted to let you know in case anybody else stumbles
upon this.

Thanks,
Stephan


On Wed, May 22, 2019 at 11:06 AM Stephan Menzel <[email protected]>
wrote:

> Hello all,
>
> I'm working on a spirit based redis client (
> https://github.com/MrMoose/mredis/blob/master/RESP.cpp) and make heavy
> use of multi-pass iterators, which I have now discovered a problem with. I
> seek advice on how to solve this.
>
> Basically, I have a number of grammars that parse specific message types
> and then an alternative message type:
>
>     template <typename InputIterator>
>     struct message_parser : qi::grammar<InputIterator, RedisMessage()> {
>
>     message_parser() : message_parser::base_type(m_start, "message") {
>          m_start %= m_simple_string | m_integer | m_array | m_null_result
> | m_bulk_string | m_error;
>     }
>
>     integer_parser<InputIterator>        m_integer;
>     simple_string_parser<InputIterator>  m_simple_string;
>     bulk_string_parser<InputIterator>    m_bulk_string;
>     null_parser<InputIterator>           m_null_result;
>     array_parser<InputIterator>          m_array;
>     error_parser<InputIterator>          m_error;
>     qi::rule<InputIterator, RedisMessage()>  m_start;
>     };
>
> Each parser parses one message. I parse those messages directly from an
> asio streambuf object and use multipass iterators to do this:
>
>     boost::spirit::multi_pass<stream_iterator_type> first =
>
> boost::spirit::make_default_multi_pass(stream_iterator_type(n_input_stream));
>     const boost::spirit::multi_pass<stream_iterator_type> last =
>         boost::spirit::make_default_multi_pass(stream_iterator_type());
>
>     const bool retval = qi::parse(first, last, s_response_parser,
> n_response);
>
> Now I ran into problems with large messages and after extensive debugging
> I found that the multipass iterator will consume the input when it parses
> part of a message but not the entire message.
>
> Like when the stream contains the beginning of a large message but it is
> incomplete and I need to continue reading. parse() will fail and I see
> remaining bytes in the buffers but those parts of the message that were
> parsed are already missing.
>
> I suppose this page explains the background:
> https://www.boost.org/doc/libs/1_70_0/libs/spirit/doc/html/spirit/support/multi_pass.html
>
> Sadly, I don't really understand what this means to me and my use case.
> How can I prevent this from happening? I have tried adding flush_multi_pass
> at the end of each seperate message but that didn't help.
>
> What I need is for the stream to be "unconsumed", untouched, when the
> overall parse command fails and no complete message could be parsed. Is
> this possible?
>
> Thanks for any input!
>
> Stephan
>
>
>

_______________________________________________
Spirit-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/spirit-general