Re: DejaGnu structured logs (was: Providing more precise "excess errors" message in DejaGnu)

On 8/29/25 04:40, Richard Earnshaw wrote:
> On 29/08/2025 10:36, Iain Sandoe wrote:
>> On 29 Aug 2025, at 10:32, Richard Earnshaw <[email protected]> 
>> wrote:
>>> On 29/08/2025 04:08, Jacob Bachmeyer wrote:
>>>>> [...]
>>>> The problem with detecting duplicate names in the DejaGnu framework 
>>>> is that it would add memory overhead that scales with the number of 
>>>> tests and DejaGnu tries to avoid that kind of unbounded space 
>>>> requirement. (OK, it *is* bounded for any finite testsuite, but the 
>>>> idea of a steadily growing memory footprint during a test run still 
>>>> bothers me.)
>>>> I suggest that the comparison script GCC uses is probably the best 
>>>> place to check for duplicate test names, since that seems to also 
>>>> be the script that can be confused by them.
>>>
>>> That's exactly what I was suggesting.  Trying to do it in dejagnu 
>>> would be a nightmare given that we run multiple instances of it to 
>>> get parallel testing.
>>
>> For the record, I’ve now proposed a BoF for the cauldron on 
>> ‘improving the raw output’ from the testsuite, since that seems at 
>> least one place we can make progress - by making the input to the 
>> post-processing tools more machine-friendly.  Hopefully the concerns 
>> and ideas from this thread can contribute there.
>
> I don't know how much maintenance DejaGNU is getting these days, but 
> certainly something like a --json option to runtest that caused the 
> .sum files to be written as structured data would make some 
> post-processing significantly easier

There is an option for XML logs that is currently a bit "half-baked" 
and, as far as I can tell, what it currently tries to do is not actually 
possible with Expect.  (There seems to be no good way to reliably 
distinguish text to and from child processes in the log.  The fatal flaw 
is that text "to" a child is typically echoed by the pty driver and 
therefore is actually logged as "from" the child.)

The "least ridiculous" option may be to divide the log into per-test 
segments with result tags as segment delimiters---XML syntactic sugar 
atop the plain text log.  Removing XML log support entirely and only 
using XML for a structured summary might be a better option.  Annotating 
text sent by DejaGnu before it appears as echoed by the pty driver is 
another option that could be a legitimate use of XML.

There are plans to radically overhaul XML support for the DejaGnu 1.7 
release.  There is some preliminary code on the "psql" branch but I am 
unsure if that code will ultimately be merged or prove to be a dead 
end.  The revised XML support might simply provide an XML summary file 
alongside the plain log and summary files.

The DejaGnu 1.6.3 release introduced the first elements of support for 
test groups.  In 1.6.3, all that the {testcase group ...} commands do is 
validate group nesting and report the current group if [testcase group] 
is used.  The framework will eventually implicitly group tests; explicit 
groups will nest among the implicit groups.  In 1.7, the new XML output 
will record groups as XML container tags in the XML log and/or summary.

All that said, once the infrastructure for the revised XML support is 
complete, reusing it to also produce .sum.json files would be relatively 
simple, really only a matter of output syntax.  (Note that that would 
mean the resulting JSON structure would mirror the XML tree structure.)


DejaGnu maintenance in general is:  bugs get fixed quickly in Git 
master, feature advances occur as I collect the right "round tuits", and 
there is a roadmap for features to land in at least the next few 
releases but no timeline planned for when those releases will be made.

For example, the currently planned major highlight for 1.6.4 will be a 
rewritten table-driven default_target_compile because that procedure is 
teetering on the edge of "BigBallOfMud" unmaintainability.

There are also plans for (eventual) native parallel testing support, but 
these are far off at the moment and will require major refactoring of 
how DejaGnu runs tests.  Expect limitations mean while that parallel 
testing support is possible, implementing it reliably will not be easy.  
(A first experimental attempt has already foundered on limitations of 
Expect.)

The refactoring required for parallel testing will not affect 
well-behaved testsuites, and any testsuites it does affect will be 
unavoidably broken if run in parallel anyway.  Using slave interpreters 
to isolate test cases has been in the TODO file for a very long time.  
(I guess probably since Tcl introduced the feature...)

For now, the best option for parallel testing is to run parallel 
instances of DejaGnu on different subsets of the testsuite.


I should probably also give advance notice that if you are interested in 
this, you should be subscribed to the DejaGnu mailing list 
(<[email protected]>).  It is generally low traffic, and there are some 
currently undocumented dubious behaviors that will be changed in the 
future.  Such changes will be announced on that list to check if anyone 
is actually depending on the current behavior.

For example, the (undocumented) procedure findfile can return a 
directory, despite its name.  (Yes, directories *are* technically files 
in POSIX...)  This is likely to change in the future.


-- Jacob