Re: Possible typo in contrib/sum2junit.sh

Rob Savoye wrote:
> On 10/31/18 7:34 AM, Jacob Bachmeyer wrote:
>   
>> While browsing the DejaGnu sources, I noticed that sum2junit.sh appears
>> to miscount failing tests due to a possible typo on line 49.  I am not
>> sufficiently familiar with JUnit to say with certainty or to be sure of
>> a correct patch, but this patch at least makes the code and comment
>> consistent.
>>     
>
>   I'm traveling right now, but as the author of that contrib script, it
> looks ok. I can test it when I get home in a few days. That script is
> for importing test results into Jenkins for CI, and not core DejaGnu
> infrastructure.
>   

The apparent typo fixed in that patch would only affect the summary 
counts generated on the "testsuite" tag.  Passes and expected failures 
are counted as passed tests, while expected failures and unexpected 
passes are counted as failures.  Unexpected failures (DejaGnu result 
"FAIL") do not appear to be counted at all.  Expected failures (DejaGnu 
result "XFAIL") are counted twice.  This patch counts FAIL as failure 
instead of counting XFAIL as both pass and failure.

Further examination finds another suspicious bit of code on line 93:  
testcases are reported using only either "skipped" or "failure" tags, 
even for tests that produced a PASS result, which just looks "fishy" to me.

I have not been able to find a DTD or even an actual specification for 
this format.  I found what appears to be an attempt to document the 
format at <URL:http://llg.cubic.org/docs/junit/> but that page is 
clearly not authoritative, as it asks for any additional information 
anyone might have.

Apparently, JUnit XML does not allow any message to be recorded for a 
test that passes?  (This would explain the suspicious use of "failure" 
tags in the sum2junit script.)

Perhaps Jenkins ignores these details in its XML input?  It seems that 
the sum2junit script groups test results into faked classes, such that a 
failing test appears to be in class "${tool}-FAIL" and a passing test in 
class "${tool}-PASS".  This will probably work until someone tries to do 
aggregation of test results across multiple runs or testing systems in 
Jenkins where every test from a DejaGnu testsuite appears to fail if not 
skipped.  The information is displayed, but any statistics generated in 
Jenkins will be wrong.

The best answer would probably be to add support for DejaGnu's own XML 
format (which seems to have a far more sensible data model) to Jenkins, 
but that is outside the scope of the DejaGnu project and I am not 
currently using Jenkins, so I will leave that to someone else.


-- Jacob