Re: testing

On Friday 30 January 2015, Felix Salfelder wrote:
> there's something odd with the new tests from the 'testing'
> branch.
.......
> 
> apparently, the "resistor" line eats the probes on X1.q1. is
> this a bug or a feature?

It's a bug, that I forgot about but really knew about for a long 
time.

It has to do with subcircuit expansion.  If you make a 
structural change, it is expanded again so the probes inside a 
subcircuit are lost.  

=====================

The purpose of the testing branch is to catch things like this.  
I invite everyone to participate.  Eventually, it will be merged 
to master, but I like to do that in bigger chunks making sure 
that master always works, kind of a perpetual "release 
candidate".

Compared to master, the testing branch adds more test cases, and 
also changes the script so the non-spice tests are included.  
Some of them have been there a long time, but not included  in 
the script because they didn't run in the "-b" spice-batch mode.

It got me going when I saw the qucs project was looking at their 
test coverage, I decided to take another look at gnucap's.

If you want a number ...  the number I come up with is 86% of 
the "branches".  The way I got that number is to grep for 
untested(), grep for {, grep for case, and compare.  It's a 
little off because there are other uses of {, but this error is 
less than 1%.

By function, by a similar grep approximation, I get 91% 
coverage.  In this case, there are things like blocked 
constructors that must exist but don't get called.  Also, there 
are the device functions like "tr_unload", which are used only 
in mixed-mode simulations.


Although I thought it was better, and hoped it would be, 86% is 
really not bad.  Most of the "untested" blocks have to do with 
abnormal situations which are difficult or tedius to test in a 
deliberate way.  Many of the "untested" blocks were really 
tested at the time of development, but do not have scripted 
tests.  Also, #ifdef'd out blocks are included in the count, as 
are lines commented out.

The gnucap project has for a long time used a procedure 
described here:
http://gnucap.org/dokuwiki/doku.php?id=gnucap:manual:tech:testing
It works very well.  I highly recommend the procedure to anyone 
on any project.