Re: Horrible compiletimes and memory usage while compiling a parser with X3

Hi!

Yes, X3 is very nice in many ways, but template depth really gets to be a problem when you create a more complicated grammar.

I have some experience implementing expressions too, but I have to make Visual Studio compile my code, and it doesn't even seem to have an option like "-ftemplate-depth", so I'm stuck with what the default there allows me. I haven't seen a problem like yours, but I DID run into the template limit.

If you have to increase the setting to over 4000, then clearly you have a LOT of depth, and that's probably creating some exponential situation which kills your compile time. I have not really seen an issue with compile time, but my limit came from VS, and is below 400, I think.

When I ran into the template-depth problem I solved it using a "template firewall" using custom parsers (very few) to solve small tricky common cases, like an identifier parser (plus, I parse tokens, not characters, so I save some depth by the help of the lexer and the custom basic token parser I use for everything). I also have a custom parser for parsing a whole expression (that also cleans up the ast a bit before returning it), so that my grammars that contain an expression doesn't add to the template depth. The real "firewalling" comes, of course, from calling a non-template function to do the parsing (notice the call to parse_expression in the templated parser), like so:

In expression.h:

  struct expression_class;
  using expression_type = x3::rule<expression_class, ast::Expression>;
  BOOST_SPIRIT_DECLARE(expression_type);
  const expression_type& expression_rule();

  bool parse_expression(iterator_type& iIter, const iterator_type& iEnd, error_handler_type& iErrHandler, ast::Expression& oAst);

  struct OptimizedExpressionParser : x3::parser<OptimizedExpressionParser> {  // Parse an expression and return an optimized ast
    using attribute_type = ast::Expression;
    static bool const has_attribute = true;

    template <typename Ctx, typename Attribute>
    bool parse(iterator_type& iFirst, const iterator_type& iLast, const Ctx& iCtx, x3::unused_type, Attribute& oAttr) const {
      ast::Expression a;
      if (parse(iFirst, iLast, iCtx, x3::unused, a)) {
        x3::traits::move_to(a, oAttr);
        return true;
      }
      return false;
    }

    template <typename Ctx>
    bool parse(iterator_type& iFirst, const iterator_type& iLast, const Ctx& iCtx, x3::unused_type, ast::Expression& oAttr) const {
      if (iFirst != iLast) {
        auto& e = x3::get<error_handler_tag>(iCtx).get();
        if (parse_expression(iFirst, iLast, e, oAttr)) {
          oAttr.optimize();
          return true;
        }
      }
      return false;
    }
  };

In expression.cpp:

  // template firewall function
  bool parse_expression(iterator_type& iIter, const iterator_type& iEnd, error_handler_type& iErrHandler, ast::Expression& oAst) {
    const auto expr = expression_rule();
    auto const exprParser = x3::with<error_handler_tag>(std::ref(iErrHandler))[expr];
    return x3::parse(iIter, iEnd, exprParser, oAst);
  }

In expression_def.h:

  // full expression grammar, plus this:
  const expression_type             expression = "expression";
  const expression_type& expression_rule() {
    return expression;
  }

And used in other _def files like this (TL calls my custom token literal parser):

    const auto expression = OptimizedExpressionParser{};
    const auto assignment = expression >> TL(tokenAssign) >> expression;


I hope I have given you some ideas that may help.... :)

  /Mikael


-----Original Message-----
From: Exagon [mailto:[email protected]] 
Sent: Sunday, October 23, 2016 21:40
To: [email protected]
Subject: [Spirit-general] Horrible compiletimes and memory usage while compiling a parser with X3

I am currently implementing expressions and the operator hierarchy for my DSL, using boost spirit X3. 
I think my parser is semanticaly correct, but when I try to compile it, while compiling, gcc and clang having HUGE memory footprint, compiles for an infinite amount of time, and then quits with "g++: internal compiler error:
Killed (program cc1plus)". 
I tried to minimize the code arround the expression parser, but its somehow not that trivial. 
Here <http://melpon.org/wandbox/permlink/8J3DNGd8xDy2v6Eo>   is an example
of what causes the compiler error. I dont know if this is a bug, or if I have a semantic bug in my code, or if this is a bug. 

I think the problem is somwhere there: 

    auto const idx = as<ast::Operation>(helper::idxaccess_op > expression > ']'); 
    auto const func = as<ast::Operation>(helper::func_call_op > expression%',' > ')'); 
    auto const data = as<ast::Operation>(helper::access_op > expression); 

    auto const func_call_expr_def = 
        primary_expr >> *(idx|func|data); 

if I change (idx|func|data) to (idx|func), it also compiles forever and uses up to 16GB of ram, but gcc is able to compile it, and the parser works how it should work.

Thanks
Exagon



--
View this message in context: http://boost.2283326.n4.nabble.com/Horrible-compiletimes-and-memory-usage-while-compiling-a-parser-with-X3-tp4689104.html
Sent from the spirit-general mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Spirit-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/spirit-general

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! http://sdm.link/slashdot