Re: DejaGnu structured logs

Jacob Bachmeyer <[email protected]> Sat, 30 Aug 2025 21:46:57 -0500
Newsgroups	gmane.comp.sysutils.dejagnu.general,gmane.comp.gcc.devel
Message-ID	<[email protected]>
This is a multi-part message in MIME format.
--------------rGRwQDBA4k6RnwRzF4liqwJC
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

On 8/30/25 12:22, David Malcolm wrote:
> On Sat, 2025-08-30 at 10:49 -0600, Rob Savoye via Gcc wrote:
>> On 8/29/25 10:47 PM, Jacob Bachmeyer wrote:
>>
>>> There is an option for XML logs that is currently a bit "half-
>>> baked"
>>> and, as far as I can tell, what it
>>     Sorry about that, I'm probably the only one that ever used it. The
>> entire idea of the .sum files was so the output could be easily
>> diff'd,
>> XML potentially being somewhat better cause at least it was
>> structured.
>> The idea of using a JSON file is probably better than fixing the
>> minimal
>> XML support. Or support both.
> I mentioned this in a different branch of this discussion, but perhaps
> the JSON format to use for test results could be SARIF:
>    https://sarif.info/
> in that it has already has an extensive "vocabulary" for describing
> results, related source locations, comparison against a baseline, etc.
> For example, the 3.27.9 "kind" property maps quite nicely to PASS,
> FAIL, UNSUPPORTED (and perhaps XPASS and XFAIL using the
> "baselineState" property).

SARIF seems at first glance to be a very bad match to DejaGnu's data 
model, or to testsuite results generally.

The problem, as I gather from a quick reading, is that the 3.27.9 "kind" 
property is attached to a 3.27 "result" object, which is specific to the 
evaluation of a rule.  Further, it appears that those rules must be 
declared elsewhere in the SARIF log and it seems that "rules" would 
correspond to tests in DejaGnu.

Since the DejaGnu framework does not know the full set of tests in a 
testsuite, nor impose the stability demanded in the SARIF spec, there 
seems to be no reasonable way to map DejaGnu results to SARIF that would 
not severely "twist" SARIF and badly confuse any unsuspecting SARIF 
processors presented with the result.

> Disclosure: I'm on the SARIF Technical Committee, and maintain GCC's
> support for SARIF output for its diagnostics and some other SARIF
> tooling, so I clearly have an interest here!
>
> Dave


-- Jacob

--------------rGRwQDBA4k6RnwRzF4liqwJC
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <div class="moz-cite-prefix">On 8/30/25 12:22, David Malcolm wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:[email protected]">
      <pre wrap="" class="moz-quote-pre">On Sat, 2025-08-30 at 10:49 -0600, Rob Savoye via Gcc wrote:
</pre>
      <blockquote type="cite">
        <pre wrap="" class="moz-quote-pre">On 8/29/25 10:47 PM, Jacob Bachmeyer wrote:

</pre>
        <blockquote type="cite">
          <pre wrap="" class="moz-quote-pre">There is an option for XML logs that is currently a bit "half-
baked" 
and, as far as I can tell, what it 
</pre>
        </blockquote>
        <pre wrap="" class="moz-quote-pre">   Sorry about that, I'm probably the only one that ever used it. The
entire idea of the .sum files was so the output could be easily
diff'd, 
XML potentially being somewhat better cause at least it was
structured. 
The idea of using a JSON file is probably better than fixing the
minimal 
XML support. Or support both.
</pre>
      </blockquote>
      <pre wrap="" class="moz-quote-pre">
I mentioned this in a different branch of this discussion, but perhaps
the JSON format to use for test results could be SARIF:
  <a class="moz-txt-link-freetext" href="https://sarif.info/">https://sarif.info/</a>
in that it has already has an extensive "vocabulary" for describing
results, related source locations, comparison against a baseline, etc.
For example, the 3.27.9 "kind" property maps quite nicely to PASS,
FAIL, UNSUPPORTED (and perhaps XPASS and XFAIL using the
"baselineState" property).</pre>
    </blockquote>
    <p>SARIF seems at first glance to be a very bad match to DejaGnu's
      data model, or to testsuite results generally.</p>
    <p>The problem, as I gather from a quick reading, is that the 3.27.9
      "kind" property is attached to a 3.27 "result" object, which is
      specific to the evaluation of a rule.  Further, it appears that
      those rules must be declared elsewhere in the SARIF log and it
      seems that "rules" would correspond to tests in DejaGnu.</p>
    <p>Since the DejaGnu framework does not know the full set of tests
      in a testsuite, nor impose the stability demanded in the SARIF
      spec, there seems to be no reasonable way to map DejaGnu results
      to SARIF that would not severely "twist" SARIF and badly confuse
      any unsuspecting SARIF processors presented with the result.<span
      style="white-space: pre-wrap">
</span></p>
    <blockquote type="cite"
cite="mid:[email protected]">
      <pre wrap="" class="moz-quote-pre">Disclosure: I'm on the SARIF Technical Committee, and maintain GCC's
support for SARIF output for its diagnostics and some other SARIF
tooling, so I clearly have an interest here!

Dave</pre>
    </blockquote>
    <p><br>
    </p>
    <p>-- Jacob<br>
    </p>
  </body>
</html>

--------------rGRwQDBA4k6RnwRzF4liqwJC--