Re: Design/structure X3 parser more like Qi parser

Sandro Pirkwieser <[email protected]>
Newsgroups gmane.comp.parsers.spirit.general
Message-ID <4A73F31802B9FB46A4428BA93C0A3DEB050330B98AED@hermes.ith-icoserve.com> 
Hi,

the new structure certainly also has its pros, and was perhaps necessary to achieve the fast compile times. Still, as highlighted in the previous mail, reusing parsers when one does not like/want having them in a namespace seems tricky at the least, if even possible in a meaningful way.
But before we get lost in this topic, I want to follow up on another one: a runtime comparison between an equivalent Qi and X3 parser.
The CSV parser mentioned in the last mail is attached in the Qi and X3 variant, which you can find in main_qi.cpp and main_x3.cpp, respectively, along with some hardcoded input strings in input.hpp.
Initially I tried to offer it via Coliru (with input.hpp inlined), but the Qi variant seems too much to handle (execution expired).
The examples are compiled with
g++ -std=c++14 -Wall -pedantic -o main_qi main_qi.cpp -O2
and
g++ -std=c++14 -Wall -pedantic -o main_x3 main_x3.cpp -O2

The binaries are executed with the number of iterations and output the runtime and the success, e.g.:

./main_qi 100000
Total time: 8798ms
#parsed: 100000, success: 100000

./main_x3 100000
Total time: 17005ms
#parsed: 100000, success: 100000

The Qi version is parameterized by giving the field separator to the constructor, for X3 there are several versions: one where `x3::with<>` is used, closest resembling the Qi variant, one where the field separator is fixed, and one where also no semantic actions (via lambdas) are used. Each parser lies in its namespace.

Since for each of the given input strings the Qi parser peforms better, I'm wondering whether the "translation" from Qi to X3 is correct and what is causing this rather unexpected result.

Best regards,
Sandro

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/xeonphi

_______________________________________________
Spirit-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/spirit-general
input.hpp (application/octet-stream, 47.2 KB) - not displayed
main_qi.cpp (text/plain, 4.1 KB) 
#include <iostream>
#include <list>
#include <chrono>

#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix.hpp>

#include <boost/exception/diagnostic_information.hpp>
#include <boost/exception_ptr.hpp>

#include "input.hpp"

using CsvRecordInformation=std::list<std::string>;
using CsvFileInformation=std::list<CsvRecordInformation>;

class CsvParser : public boost::spirit::qi::grammar<std::string::const_iterator, CsvFileInformation ()>
{
public:
    CsvParser(const std::string& m_fieldSeparator) :
        CsvParser::base_type(m_file, "file")
    {
        m_unescapedId %= boost::spirit::qi::lexeme[*(boost::spirit::qi::char_ - (boost::spirit::qi::lit(m_fieldSeparator) | boost::spirit::qi::lit(s_quoteCharacter) | boost::spirit::qi::lit(s_crCharacter) | boost::spirit::qi::lit(s_lfCharacter)))];
        m_escapedId =
            boost::spirit::qi::lexeme[*((boost::spirit::qi::char_ - boost::spirit::qi::lit(s_quoteCharacter))[boost::spirit::qi::_val += boost::spirit::qi::_1]
            | (boost::spirit::qi::lit(s_quoteCharacter) >> boost::spirit::qi::lit(s_quoteCharacter))[boost::spirit::qi::_val += s_quoteCharacter])];
        m_lineBreak %=
            boost::spirit::qi::lexeme[boost::spirit::qi::lit(s_lfCharacter)
            | boost::spirit::qi::lit(s_crCharacter) >> (boost::spirit::qi::lit(s_lfCharacter) | boost::spirit::qi::eps)];

        m_field %=
            (boost::spirit::qi::repeat[boost::spirit::qi::lit(s_whitespaceCharacter)] >> boost::spirit::qi::lit(s_quoteCharacter) > m_escapedId > boost::spirit::qi::lit(s_quoteCharacter) > boost::spirit::qi::repeat[boost::spirit::qi::lit(s_whitespaceCharacter)]) // the order is important because unescaped ID can consume zero characters
            | m_unescapedId;
        m_record %= m_field % boost::spirit::qi::lit(m_fieldSeparator);
        m_file.name("file");
        m_file %= ((!boost::spirit::qi::eoi > m_record) % m_lineBreak) > (m_lineBreak | boost::spirit::qi::eps) > boost::spirit::qi::eoi;

        // error handling removed
    }

    virtual ~CsvParser() = default;

private:
    boost::spirit::qi::rule<std::string::const_iterator, std::string ()> m_unescapedId;
    boost::spirit::qi::rule<std::string::const_iterator, std::string ()> m_escapedId;
    boost::spirit::qi::rule<std::string::const_iterator, void ()> m_lineBreak;
    boost::spirit::qi::rule<std::string::const_iterator, std::string ()> m_field;
    boost::spirit::qi::rule<std::string::const_iterator, CsvRecordInformation ()> m_record;
    boost::spirit::qi::rule<std::string::const_iterator, CsvFileInformation ()> m_file;
    static const char s_quoteCharacter = '"';
    static const char s_crCharacter = '\x0d';
    static const char s_lfCharacter = '\x0a';
    static const char s_whitespaceCharacter = '\x20';
};


int main(int argc, char** argv)
{

    try
    {
        unsigned int parsed=0;
        unsigned int parsedSuccess=0;
        const unsigned int iterations = std::atoi(argv[1]);

        std::chrono::steady_clock::time_point startTime = std::chrono::steady_clock::now();

        const CsvParser csvParser(",");
        for (unsigned int i=0; i<iterations; ++i)
        {
            CsvFileInformation result;
            std::string::const_iterator start = csvStrSingleLine.begin();
            std::string::const_iterator end = csvStrSingleLine.end();

            const bool success = boost::spirit::qi::parse(start, end, csvParser, result);
            ++parsed;

            if (success && start == end)
                ++parsedSuccess;
        }

        std::chrono::steady_clock::time_point endTime = std::chrono::steady_clock::now();
        std::cout << "Total time: " << std::chrono::duration_cast<std::chrono::milliseconds>(endTime - startTime).count() << "ms" << std::endl;
        std::cout << "#parsed: " << parsed << ", success: " << parsedSuccess << std::endl;
    }
    catch (boost::exception& e) {
        std::cout << "boost exception: " << boost::diagnostic_information(e) << std::endl;
    }
    catch (std::exception& e) {
        std::cout << "exception: " << e.what() << std::endl;
    }
    return EXIT_SUCCESS;
}
main_x3.cpp (text/plain, 9.2 KB) 
#include <iostream>
#include <list>
#include <chrono>

#include <boost/spirit/home/x3.hpp>
#include <boost/exception/diagnostic_information.hpp>
#include <boost/exception_ptr.hpp>

#include "input.hpp"

using CsvX3RecordInformation=std::list<std::string>;
using CsvX3FileInformation=std::list<CsvX3RecordInformation>;

namespace csv
{
    const char quoteCharacter('"');
    const char crCharacter('\x0d');
    const char lfCharacter('\x0a');
    const char whitespaceCharacter('\x20');
    struct fieldSeparator {};

    auto addChar = [&](auto& ctx){ boost::spirit::x3::_val(ctx) += boost::spirit::x3::_attr(ctx); };
    auto addQuoteChar = [&](auto& ctx){ boost::spirit::x3::_val(ctx) += quoteCharacter; };

    auto isFieldSeparator = [&](auto& ctx){ boost::spirit::x3::_pass(ctx) = (boost::spirit::x3::_attr(ctx) == boost::spirit::x3::get<fieldSeparator>(ctx)); };

    const boost::spirit::x3::rule<class CsvX3Unescaped, std::string, true> unescaped = "CsvX3Unescaped";
    const auto unescaped_def = boost::spirit::x3::lexeme[*(boost::spirit::x3::char_ - (boost::spirit::x3::char_[isFieldSeparator] | boost::spirit::x3::lit(quoteCharacter) | boost::spirit::x3::lit(crCharacter) | boost::spirit::x3::lit(lfCharacter)))]; // char_ represents 8 bit value

    const boost::spirit::x3::rule<class CsvX3Escaped, std::string> escaped = "CsvX3Escaped";
    const auto escaped_def =
        boost::spirit::x3::lexeme[*((boost::spirit::x3::char_ - boost::spirit::x3::lit(quoteCharacter))[addChar]
        | (boost::spirit::x3::lit(quoteCharacter) >> boost::spirit::x3::lit(quoteCharacter))[addQuoteChar]
        )];

    const boost::spirit::x3::rule<class CsvX3LineBreak> lineBreak = "CsvX3LineBreak";
    const auto lineBreak_def =
        boost::spirit::x3::lexeme[boost::spirit::x3::lit(lfCharacter)
        | boost::spirit::x3::lit(crCharacter) >> (boost::spirit::x3::lit(lfCharacter) | boost::spirit::x3::eps)];

    const boost::spirit::x3::rule<class CsvX3Field, std::string> field = "CsvX3Field";
    const auto field_def =
        (boost::spirit::x3::repeat[boost::spirit::x3::lit(whitespaceCharacter)] >> boost::spirit::x3::lit(quoteCharacter) > escaped > boost::spirit::x3::lit(quoteCharacter) > boost::spirit::x3::repeat[boost::spirit::x3::lit(whitespaceCharacter)]) // the order is important because unescaped can consume zero characters
        | unescaped;

    const boost::spirit::x3::rule<class CsvX3Record, CsvX3RecordInformation, true> record = "CsvX3Record";
    const auto record_def = field % boost::spirit::x3::char_[isFieldSeparator];

    const boost::spirit::x3::rule<class CsvX3File, CsvX3FileInformation> file = "CsvX3File";
    const auto file_def = ((!boost::spirit::x3::eoi > record) % lineBreak) > (lineBreak | boost::spirit::x3::eps) > boost::spirit::x3::eoi;

    BOOST_SPIRIT_DEFINE(unescaped, escaped, lineBreak, field, record, file)
}

namespace csv_fixed_separator
{
    const char quoteCharacter('"');
    const char crCharacter('\x0d');
    const char lfCharacter('\x0a');
    const char whitespaceCharacter('\x20');
    const char fieldSeparator(',');

    auto addChar = [&](auto& ctx){ boost::spirit::x3::_val(ctx) += boost::spirit::x3::_attr(ctx); };
    auto addQuoteChar = [&](auto& ctx){ boost::spirit::x3::_val(ctx) += quoteCharacter; };

    const boost::spirit::x3::rule<class CsvX3Unescaped, std::string> unescaped = "CsvX3Unescaped";
    const auto unescaped_def = boost::spirit::x3::lexeme[*(boost::spirit::x3::char_ - (boost::spirit::x3::lit(fieldSeparator) | boost::spirit::x3::lit(quoteCharacter) | boost::spirit::x3::lit(crCharacter) | boost::spirit::x3::lit(lfCharacter)))]; // char_ represents 8 bit value

    const boost::spirit::x3::rule<class CsvX3Escaped, std::string> escaped = "CsvX3Escaped";
    const auto escaped_def =
        boost::spirit::x3::lexeme[*((boost::spirit::x3::char_ - boost::spirit::x3::lit(quoteCharacter))[addChar]
        //boost::spirit::x3::lexeme[*((~boost::spirit::x3::char_(quoteCharacter))//[addChar] // possible alternative
        | (boost::spirit::x3::lit(quoteCharacter) >> boost::spirit::x3::lit(quoteCharacter))[addQuoteChar]
        )];

    const boost::spirit::x3::rule<class CsvX3LineBreak> lineBreak = "CsvX3LineBreak";
    const auto lineBreak_def =
        boost::spirit::x3::lexeme[boost::spirit::x3::lit(lfCharacter)
        | boost::spirit::x3::lit(crCharacter) >> (boost::spirit::x3::lit(lfCharacter) | boost::spirit::x3::eps)];

    const boost::spirit::x3::rule<class CsvX3Field, std::string> field = "CsvX3Field";
    const auto field_def =
        (boost::spirit::x3::repeat[boost::spirit::x3::lit(whitespaceCharacter)] >> boost::spirit::x3::lit(quoteCharacter) > escaped > boost::spirit::x3::lit(quoteCharacter) > boost::spirit::x3::repeat[boost::spirit::x3::lit(whitespaceCharacter)]) // the order is important because unescaped can consume zero characters
        | unescaped;

    const boost::spirit::x3::rule<class CsvX3Record, CsvX3RecordInformation> record = "CsvX3Record";
    const auto record_def = field % boost::spirit::x3::lit(fieldSeparator);

    const boost::spirit::x3::rule<class CsvX3File, CsvX3FileInformation> file = "CsvX3File";
    const auto file_def = ((!boost::spirit::x3::eoi > record) % lineBreak) > (lineBreak | boost::spirit::x3::eps) > boost::spirit::x3::eoi;

    BOOST_SPIRIT_DEFINE(unescaped, escaped, lineBreak, field, record, file)
}

namespace csv_fixed_separator_no_semantic_actions
{
    const char quoteCharacter('"');
    const char crCharacter('\x0d');
    const char lfCharacter('\x0a');
    const char whitespaceCharacter('\x20');
    const char fieldSeparator(',');

    const boost::spirit::x3::rule<class CsvX3Unescaped, std::string> unescaped = "CsvX3Unescaped";
    const auto unescaped_def = boost::spirit::x3::lexeme[*(boost::spirit::x3::char_ - (boost::spirit::x3::lit(fieldSeparator) | boost::spirit::x3::lit(quoteCharacter) | boost::spirit::x3::lit(crCharacter) | boost::spirit::x3::lit(lfCharacter)))];

    const boost::spirit::x3::rule<class CsvX3Escaped, std::string> escaped = "CsvX3Escaped";
    const auto escaped_def =
        boost::spirit::x3::lexeme[*((boost::spirit::x3::char_ - boost::spirit::x3::lit(quoteCharacter))
        | (boost::spirit::x3::lit(quoteCharacter) >> boost::spirit::x3::char_(quoteCharacter))
        )];

    const boost::spirit::x3::rule<class CsvX3LineBreak> lineBreak = "CsvX3LineBreak";
    const auto lineBreak_def =
        boost::spirit::x3::lexeme[boost::spirit::x3::lit(lfCharacter)
        | boost::spirit::x3::lit(crCharacter) >> (boost::spirit::x3::lit(lfCharacter) | boost::spirit::x3::eps)];

    const boost::spirit::x3::rule<class CsvX3Field, std::string> field = "CsvX3Field";
    const auto field_def =
        (boost::spirit::x3::repeat[boost::spirit::x3::lit(whitespaceCharacter)] >> boost::spirit::x3::lit(quoteCharacter) > escaped > boost::spirit::x3::lit(quoteCharacter) > boost::spirit::x3::repeat[boost::spirit::x3::lit(whitespaceCharacter)]) // the order is important because unescaped can consume zero characters
        | unescaped;

    const boost::spirit::x3::rule<class CsvX3Record, CsvX3RecordInformation> record = "CsvX3Record";
    const auto record_def = field % boost::spirit::x3::lit(fieldSeparator);

    const boost::spirit::x3::rule<class CsvX3File, CsvX3FileInformation> file = "CsvX3File";
    const auto file_def = ((!boost::spirit::x3::eoi > record) % lineBreak) > (lineBreak | boost::spirit::x3::eps) > boost::spirit::x3::eoi;

    BOOST_SPIRIT_DEFINE(unescaped, escaped, lineBreak, field, record, file)
}

int main(int argc, char** argv)
{

    try
    {
        unsigned int parsed=0;
        unsigned int parsedSuccess=0;
        const unsigned int iterations = std::atoi(argv[1]);

        std::chrono::steady_clock::time_point startTime = std::chrono::steady_clock::now();

        // "dynamic" variant
        //const char fieldSeparator(',');
        //auto parser = boost::spirit::x3::with<csv::fieldSeparator>(std::ref(fieldSeparator))[csv::file];

        // variant with fixed separator
        //auto parser = csv_fixed::file;

        // variant with fixed separator and no semantic actions
        auto parser = csv_fixed_separator_no_semantic_actions::file;

        for (unsigned int i=0; i<iterations; ++i)
        {
            CsvX3FileInformation result;
            std::string::const_iterator start = csvStrSingleLine.begin();
            std::string::const_iterator end = csvStrSingleLine.end();

            const bool success = boost::spirit::x3::parse(start, end, parser, result);
            ++parsed;

            if (success && start == end)
                ++parsedSuccess;
        }

        std::chrono::steady_clock::time_point endTime = std::chrono::steady_clock::now();
        std::cout << "Total time: " << std::chrono::duration_cast<std::chrono::milliseconds>(endTime - startTime).count() << "ms" << std::endl;
        std::cout << "#parsed: " << parsed << ", success: " << parsedSuccess << std::endl;
    }
    catch (boost::exception& e) {
        std::cout << "boost exception: " << boost::diagnostic_information(e) << std::endl;
    }
    catch (std::exception& e) {
        std::cout << "exception: " << e.what() << std::endl;
    }
    return EXIT_SUCCESS;
}
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.