Re: NUnit XML Output

Hi Mark, 

> Hmm, I think there are two different definitions here.
> 
> By 'log' I believe I was referring simply to the XML output 
> file for test results (TestResults.xml).  I'm not requesting 
> any other additions or changes to it, but merely to keep it 
> up to date after each fixture.
> 
> Any problems with concurrency would surely affect the current 
> implementation anyway - if test A finishes and writes, and 
> then test B does too, wouldn't you lose the original output?

No, because we don't write the file at all as the tests execute,
which is what I meant when I said the file is not a log.

What happens is that a tree of results is built up independent 
of the order in which the tests execute and that tree is used 
at the end of the run to create the xml document.

Note that currently this happens slightly differently in the
gui and the console, which is a problem I plan to resolve soon.

> In our setup we have the outputs going to different locations 
> which would seem a logical implementation - but to have it 
> written intermittently would make things much better.
>
> *LOGS* on the other hand are separate and should in theory be 
> done by the application under test itself, as NUnit is merely 
> the framework, not a diagnosis tool.

Except that only NUnit can easily log which tests it has initiated
and which are complete. 

> However while the discussion is ongoing I'm going to take a 
> look at the code form of the listeners for monitoring 
> progress of the tests.  Any brief summary or class 
> documentation around Charlie or is it more of a case of "dive 
> on in"? :)

Take a look at the general docs on writing extensions. You want
to write an EventListener extension, which will implement the
EventListener interface.

However, as I asked in an earlier note, why not just use
NUnit's console output, which can be set to list each test
as it executes? That would certainly tell you which test
caused a hang.

Charlie

> Cheers,
> 
> Mark
> 
> 
> On Mon, Aug 18, 2008 at 9:09 PM, Charlie Poole 
> <[email protected]> wrote:
> 
> 
> 
> 	Hi Aiste,
> 	
> 
> 	> >> On the logging side - presumably at the end of the test, a
> 	> XML write
> 	> >> is called which includes validating the XML by 
> closing tags at the
> 	> >> end etc.  Can this not be called after each fixture, just
> 	> overwriting
> 	> >> the previous file with the 'latest' results?  To 
> anyone that just
> 	> >> uses the xml file created at the end, the 
> behaviour would not have
> 	> >> changed from their original usage, and for those 
> that like the
> 	> >> incremental results, the benefit is substantial (well for
> 	> me it would
> 	> >> be :))
> 	>
> 	> > We could write the entire file at the end of each 
> fixture, but that
> 	> > seems a bit of a kluge, with potential bad side effects
> 	> down the road.
> 	> > For example, when we are running two fixtures in parallel.
> 	>
> 	> I would join those who would like incremental logging. I
> 	> actually wrote to this mailing list quite a while ago, asking
> 	> if it was possible to flush the log file after each test, but
> 	> it got lost or something.
> 	
> 	
> 	I do remember such a discussion, but maybe it was with someone
> 	else. I have to distinguish here, however. NUnit does *not*
> 	have a log file and I think one issue with this discussion is
> 	that some people want TestResult.xml to become a log file.
> 	
> 	If we need a log file, I believe we should create one, based
> 	on some stories about how it would be used. But many people
> 	have already built apps based on the test result file and
> 	it doesn't seem right to change it in order to serve an
> 	entirely different purpose, especially when NUnit already
> 	has a quite different mechanism for incremental progress
> 	reporting.
> 	
> 
> 	> So I ended up implementing my own
> 	> logging, with only the basic information I needed.
> 	
> 	
> 	That's what I believe should be done in NUnit, but the
> 	request was to specifically modify the creation of
> 	TestResult.xml so it could be used for a new purpose.
> 	That sounds a bit risky.
> 	
> 
> 	> It is
> 	> still XML with all its advantages. I can't remember exactly
> 	> now, but I think I overwrite it instead of appending.
> 	
> 	
> 	I'm wondering how this scales. Also, how well it serves the
> 	use case somebody mentioned where the test does not terminate.
> 	However, it's an implementation detail for whoever 
> implements it.
> 	
> 
> 	> No clue
> 	> about parallel running though - never used it or even thought
> 	> it was possible :)
> 	
> 	
> 	It's only just becoming possible.
> 	
> 
> 	> However IMHO running two fixtures in
> 	> parallel and need for incremental logging is equally
> 	> common/incommon. Well, but even if I am not really sure how
> 	> running tests in parallel works, I am quite sure that there's
> 	> not so much overhead for implementing logging having it in
> 	> mind. Maybe having separate log files, maybe merging them
> 	> afterwards or maybe just synchronizing writing to the file.
> 	
> 	
> 	My example was just one example, given as a way of saying
> 	"It's not as trivial as you see to think."
> 	
> 
> 	> My main reason for incremental logging was one kind of tests
> 	> crashing Nunit randomly, which I could not find a way to
> 	> resolve. So I would just rerun the tests from where they
> 	> crashed if it happened.
> 	
> 	
> 	Why not just look at the text output - after setting it to
> 	display the name of each test as it starts?
> 	
> 
> 	> Any other format for log files is acceptable for me, as long
> 	> as they can be viewed comfortably (as xml is with xls). NUnit
> 	> custom appendable format may be used with Nunit's own viewer,
> 	> for example.
> 	
> 	
> 	As I have always understood the term, "log" implies trivial
> 	viewing - possibly in a text editor. Log data is sequential
> 	and (usually) unstructured. I think of log4net, nlog or
> 	trace when I hear "log." I think some things work well with
> 	a log, while others require an object representation of some
> 	sort - as is the case of the test result format.
> 	
> 	I would be OK with a log having individual entries that used
> 	XML - It wouldn't be a valid XML document, but could contain
> 	all the correct data.
> 	
> 	IMO, we need to go back to what you - and others - want to
> 	accomplish and figure out how it needs to work rather than
> 	starting with an existing feature - intended for a different
> 	purpose - and trying to modify to do... what? I'm not really
> 	sure if I know the scope of what is being asked for.
> 	
> 	Charlie
> 	
> 
> 	> Best regards,
> 	> Aiste
> 	>
> 	>
> 	> __________ Information from ESET NOD32 Antivirus, version of
> 	> virus signature database 3365 (20080818) __________
> 	>
> 	> The message was checked by ESET NOD32 Antivirus.
> 	>
> 	> http://www.eset.com
> 	>
> 	>
> 	>
> 	> --------------------------------------------------------------
> 	> -----------
> 	> This SF.Net email is sponsored by the Moblin Your Move
> 	> Developer's challenge Build the coolest Linux based
> 	> applications with Moblin SDK & win great prizes Grand prize
> 	> is a trip for two to an Open Source event anywhere in the
> 	> world 
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> 	> _______________________________________________
> 	> Nunit-users mailing list
> 	> [email protected]
> 	> https://lists.sourceforge.net/lists/listinfo/nunit-users
> 	>
> 	
> 	
> 	
> 	
> --------------------------------------------------------------
> -----------
> 	This SF.Net email is sponsored by the Moblin Your Move 
> Developer's challenge
> 	Build the coolest Linux based applications with Moblin 
> SDK & win great prizes
> 	Grand prize is a trip for two to an Open Source event 
> anywhere in the world
> 	http://moblin-contest.org/redirect.php?banner_id=100&url=/
> 	_______________________________________________
> 	Nunit-users mailing list
> 	[email protected]
> 	https://lists.sourceforge.net/lists/listinfo/nunit-users
> 	
> 
> 
> 



-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/