Re: 2.5 console runner documenation

Hi Jay, 

> I'm not sure I see how using a documented feature is out of 
> the ordinary :)  
>
> Really, if it's the only way to customize the console output 
> to produce extra terse output, it'd be a shame to scrap it 
> entirely.  It's like the /b option for the dir command in 
> dos.  Sometimes you just need something that outputs the bear 
> minimal information.  The benefit of xslt is that you can go 
> beyond that and customize things however you like.  I 
> apologize that I wasn't around to voice a vote when you were 
> looking for opinions of the matter.  And I appreciate that 
> xslt is a pain to use for something as simple as this.  But I 
> guarantee if you don't have something to replace its 
> functionality you're going to get other users grumbling, too. 
>  I can't be the only one that works from the command line, 
> relying on stdout.
>
> Is it that you're not familiar with xslt, and another 
> contributor who isn't around anymore is the one who added it? 
>  I can appreciate that, too.  But really, it should be a 
> simple matter to test one way or another on console output.  
> If the transform file was specified, feed the current test's 
> output XML into it and print the result to stdout.  Otherwise 
> use whatever system you've built to replace it.  Just because 
> the system can use xslt doesn't mean YOU have to.  This just 
> seems like a case where you shouldn't be breaking backwards 
> compatibility for no reason.

In fact, the xml output is still there for you to use. And xslt
will be used to produce reports out of NUnit. It's just not
being done in the nunit-console program. 
> Actually, if NUnit didn't have the transform option on the 
> console, you and I wouldn't be having this conversation.  I 
> would have found another unit testing framework that did what 
> I needed.  At the very least you need to have a little error 
> message saying that -transform is no longer supported if you 
> try to run the console runner with that option.

I'm sorry that the change doesn't seem to work for you. I 
indicated that there was discussion about it because the 
upshot of that discussion was that the rearranging of the
different facilities - they have not been removed - still
allowed everything to be done that was being done and also
allowed some new things to be done that couldn't be done
before. I'm pretty sure that you would find you can use
the new arrangement in pretty much the same way as before
if you gave it a chance. 

> Okay, my little rant over, let's get to the matter at hand.  
> 1 and 2 won't work, because I need the output to go to the 
> console before the unit tests can return a return code.

That's what option 1 would do. Option 2 could be made to do
that if you made use of a wrapper script to give the
return code.
  
> That's really the way the whole chain is set up, and it'd be 
> a major pain to change things.
> 
> #3 sounds promising.  Looking through the source it doesn't 
> seem too bad.  What's a mile high overview of what ConsoleUi 
> does?  It's only summary statement is: "Summary description 
> for ConsoleUi.", which isn't terribly descriptive ;)

You're right that it's not mucy of a description. It should
say "Runs the NUnit tests and produces console and xml output. :-)

The code is fairly clear, I think. You'll find routines that
produce part of the console output. However, if you are 
already using xslt, I'd suggest you continue to do so. From
what you have described, it sounds like you only need to
make a small change.
 
> The other option is the non-action of just sticking with the 
> 2.4.X fork and lumping it.  Right now there's nothing in 2.5 
> that I need, but I'd hate to cut myself off from future updates.

That is an option. A further option, of course, would be to 
modify the 2.5 code to work as you think it should.

However, I would really suggest that you take another look at
the five approaches I listed for you. It seems that one of 
them ought to do the job. In particular, it seems as if you
could simply replace your invocation of nunit-console with
a script that could produce exactly what you now get, by
judicious use of command-line options and/or a separate
reformatting program. I can provide you with a utility
to do the reformatting if you don't have one handy.

Charlie
 
> -Jay Lemmon
> 
> --- On Sun, 9/7/08, Charlie Poole <[email protected]> wrote:
> 
> 
> 	From: Charlie Poole <[email protected]>
> 	Subject: RE: [Nunit-users] 2.5 console runner documenation
> 	To: [email protected]
> 	Date: Sunday, September 7, 2008, 5:32 PM
> 	
> 	
> 	Hi Jay,
> 	 
> 	I'll start with the last question: based on feedback 
> from users, most people
> 	assumed that the option caused the xml file to be 
> reformatted, not the
> 	console output. I got a lot of "bugs" about how using 
> /transform broke
> 	the console output and sent html to stdout instead of 
> to the file. :-)
> 	 
> 	The xml is the primary output from nunit and it seemed 
> that anyone
> 	wanting to reformat it could do so easily enough. My 
> own little command-line
> 	utility that reformats the xml using a stylesheet is 
> about 15 lines long.
> 	 
> 	But using something as sophisticated as xml and xslt 
> just to produce the
> 	command-line output, which is not intended for 
> reporting purposes seemed
> 	like a waste of effort - using a sledge hammer to crack 
> walnuts. More important,
> 	it coupled two different things together: console 
> output and reporting, making
> 	it very ard to change either one. With a reporting 
> subsystem on the drawing
> 	board, I didn't want to have to think about how changes 
> in the xml would impact
> 	the nunit-console display.
> 	 
> 	Back to your problem, then...
> 	 
> 	I see five options...
> 	 
> 	1. Use the /xmlconsole switch to send the xml to the 
> console and add a filter program
> 	to your toolchain to reformat it.
> 	 
> 	2. Use the same external filter program with 
> TestResult.xml as it's input.
> 	 
> 	3. Do the reformatting within your own runner, taking 
> the code from
> 	ConsoleUi.cs.
> 	 
> 	4. Replicate the nunit logic in your own runner - i.e. 
> copy it - and modify
> 	the runner to write output in the form you prefer.
> 	 
> 	5. Same as #4, but modify the event handlers in 
> EventCollector.cs to
> 	do the same thing.
> 	 
> 	 
> 	I don't think any of these are particularly harder than 
> what you have
> 	done - but of course, you've already done that work, so 
> I recognize 
> 	this is extra effort for you. I apologize for that, but 
> I think this change
> 	will get us farther in the long run.
> 	 
> 	Charlie
> 	 
> 	PS: The change was discussed before and at the time of 
> implementing.
> 	It's good to be involved in such list disussions, 
> particularly if you know
> 	you're doing something a little out of the ordinary.
> 	 
> 	 
> 
> 
> ________________________________
> 
> 		From: Jay Lemmon [mailto:[email protected]] 
> 		Sent: Sunday, September 07, 2008 3:04 PM
> 		To: Charlie Poole
> 		Subject: Re: [Nunit-users] 2.5 console runner 
> documenation
> 		
> 		
> 		Yes, I'm still pretty raw with NUnit, so any 
> help would be appreciated :)
> 		
> 		My tool chain is set to use the console pretty 
> exclusively.  That means stdout AND return codes.  Also, the 
> stdout needs to be kept very clean, because I don't want to 
> write a lot of special logic excising extra data.  So a 
> command line option would probably be too weird.
> 		
> 		So that leaves something done in code.  Right 
> now my test runner is pretty stupid.  It just adds some 
> command line switches automatically to the console options 
> and feeds it to the console runner library.  How deeply am I 
> going to have to dig in to the code to get what I need?
> 		
> 		Also, out of curiosity, why was the xslt option 
> removed?  It seems like a step backwards without the 
> corresponding two steps forward.  But then my scope is pretty 
> narrow :)
> 		
> 		-Jay Lemmon
> 		
> 		--- On Sun, 9/7/08, Charlie Poole 
> <[email protected]> wrote:
> 		
> 
> 			From: Charlie Poole 
> <[email protected]>
> 			Subject: Re: [Nunit-users] 2.5 console 
> runner documenation
> 			To: [email protected], 
> [email protected]
> 			Date: Sunday, September 7, 2008, 2:54 PM
> 			
> 			
> 			That sounds like a valid use case, 
> altough unusual. What's interesting is that it matches exactly
> 			the use cases I wrote about earlier: 
> You want a report in a particular format, but the fact that
> 			it is produced on stdout by NUnit is 
> just an implementation detail and not a requirement.
> 			 
> 			AFAICS, you can do everything you want 
> using any of the options I described in the last
> 			note. If it's not clear, I'll try to 
> supply more detail.
> 			 
> 			Charlie
> 
> 			...
> 			
> 			
> 			The output is created in code. You 
> could certainly do that yourself if you like. Essentially, 
> you have two choices: to look at the final tree of results, 
> as NUnit does, or to handle each test result as it comes 
> along - which is how NUnit creates it's test output.
> 			 
> 			Of course, you can also continue to use 
> xslt as you now do - you just can't have 
> 			NUnit apply the xslt for you. That's 
> trivial to do at the end of the test run by taking
> 			the xml output adn transforming it.
> 			 
> 			Charlie
> 
> 
> ________________________________
> 
> 				From: 
> [email protected] 
> [mailto:[email protected]] On Behalf 
> Of Jay Lemmon
> 				Sent: Sunday, September 07, 2008 2:29 PM
> 				To: [email protected]
> 				Subject: Re: [Nunit-users] 2.5 
> console runner documenation
> 				
> 				
> 				A quick follow up to my last 
> email.  This is the format I'm looking for the console runner 
> to output.  I need it this way because it mirrors the output 
> from another unit testing framework (UnitTest++), so it makes 
> everything nice and consistent for my tool chain.
> 				
> 				
> c:\glassworksstudio\lodestone\source\trunk\unittests\quarticte
> sts.cpp(56): error: Failure in Fixture1.Test_1: Expected 0 but was 2
> 				
> c:\glassworksstudio\lodestone\source\trunk\unittests\quarticte
> sts.cpp(63): error: Failure in Fixture1.Test_2: 20 < 10
> 				FAILURE: 2 out of 156 tests 
> failed (2 failures).
> 				Test time: 0.02 seconds.
> 				
> 				Note each test failure follows 
> this format:
> 				[file]([line]): error: Failure 
> in [TestFixture].[TestName]: [Error message]
> 				all inline, with a new line for 
> each failed test.  Then a summary report and the time to run 
> the tests comes last.
> 				
> 				-Jay Lemmon
> 				
> 				--- On Sun, 9/7/08, Jay Lemmon 
> <[email protected]> wrote:
> 				
> 
> 					From: Jay Lemmon 
> <[email protected]>
> 					Subject: Re: 
> [Nunit-users] 2.5 console runner documenation
> 					To: 
> [email protected]
> 					Date: Sunday, September 
> 7, 2008, 2:16 PM
> 					
> 					
> 					Well I can't speak for 
> anyone else, but customizing the (console) output of the 
> console runner is important to me.  If it's impossible in 
> 2.5, it'll mean I can't upgrade :(
> 					
> 					It doesn't have to be 
> xslt.  And it doesn't have to be a command line switch.  
> Right now I'm actually linking against the console runner 
> assemblies (DLLs) to build my own custom EXE for every test 
> project I build, so it could be something done purely in 
> code.  How is the format for the console runner decided right 
> now if it's not xslt?
> 					
> 					-Jay Lemmon
> 					
> 					--- On Sun, 9/7/08, 
> Charlie Poole <[email protected]> wrote:
> 					
> 
> 					From: Charlie Poole 
> <[email protected]>
> 					Subject: Re: 
> [Nunit-users] 2.5 console runner documenation
> 					To: 
> [email protected]
> 					Date: Sunday, September 
> 7, 2008, 1:10 PM
> 					
> 					
> 					Hi Jay,
> 					 
> 					There doesn't seem to 
> be a real need to customize console output, except possibly to have
> 					NUnit show more 
> detailed info - an option I'm working on. Most people using 
> the old feature
> 					seemed to be producing 
> html with it. That's still possible, but the xml would need to be 
> 					formatted after the NUnit run.
> 					 
> 					Charlie
> 
> 
> ________________________________
> 
> 					From: 
> [email protected] 
> [mailto:[email protected]] On Behalf 
> Of Jay Lemmon
> 					Sent: Sunday, September 
> 07, 2008 12:54 PM
> 					To: 
> [email protected]
> 					Subject: Re: 
> [Nunit-users] 2.5 console runner documenation
> 					
> 					
> 					I just copied the 2.5 
> binaries over my old binaries, so it wasn't a fool proof test, no :)
> 					
> 					So how do I, the end 
> user, customize console output now?  The release notes don't 
> mention anything in this regard.
> 					
> 					-Jay Lemmon
> 					
> 					--- On Sun, 9/7/08, 
> Charlie Poole <[email protected]> wrote:
> 					
> 
> 					From: Charlie Poole 
> <[email protected]>
> 					Subject: Re: 
> [Nunit-users] 2.5 console runner documenation
> 					To: 
> [email protected]
> 					Date: Sunday, September 
> 7, 2008, 6:39 AM
> 					
> 					
> 					Hi Jay,
> 					 
> 					There's mention in the 
> release notes, under 2.5 alpha 1, about the removal of the transform
> 					option. The console 
> output is now indepedent of the xml, so it no longer has any purpose.
> 					 
> 					Since the option is no 
> longer there, I'm pretty sure it can't be working. :-) Are you sure
> 					you were using 2.5 for 
> your test?
> 					 
> 					Charlie
> 
> 
> ________________________________
> 
> 					From: 
> [email protected] 
> [mailto:[email protected]] On Behalf 
> Of Jay Lemmon
> 					Sent: Saturday, 
> September 06, 2008 9:42 PM
> 					To: 
> [email protected]
> 					Subject: [Nunit-users] 
> 2.5 console runner documenation
> 					
> 					
> 					When comparing the 
> 2.4.8 docs and the 2.5 docs, I noticed that this section in 
> console runner: command-line was mostly deleted and partly 
> merged with the one before it:
> 					
> 					"Specifying the Transform file
> 					
> 					The console interface 
> uses XSLT to transform the test results from the XML file to 
> what is printed to the screen when the program executes. The 
> console interface has a default transformation that is part 
> of the executable. To specify your own transformation named 
> "myTransform.xslt" use the following command line option:
> 					
> 					nunit-console 
> /transform:myTransform.xslt nunit.tests.dll
> 					
> 					Note: For additional 
> information see the XML schema for the test results. This 
> file is in the same directory as the executable and is called 
> Results.xsd. The default transform Summary.xslt is located in 
> the core source directory."
> 					
> 					In the 2.5 docs, 
> there's no such section, and the mention of the Results.xsd 
> is merged in to the section before it on specifying an output 
> xml file.
> 					
> 					Is this an error in the 
> docs, or is this feature (specifying a transform) going to be 
> removed in 2.5? A quick test shows that it still works.
> 					
> 					-Jay Lemmon
> 						
> 
> 
> 					
> --------------------------------------------------------------
> -----------
> 					This SF.Net email is 
> sponsored by the Moblin Your Move Developer's
> 					challenge
> 					Build the coolest Linux 
> based applications with Moblin SDK & win great
> 					prizes
> 					Grand prize is a trip 
> for two to an Open Source event anywhere in the world
> 					
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> 					
> _______________________________________________
> 					Nunit-users mailing list
> 					
> [email protected]
> 					
> https://lists.sourceforge.net/lists/listinfo/nunit-users
> 					
> 
> 					
> 					
> --------------------------------------------------------------
> -----------
> 					This SF.Net email is 
> sponsored by the Moblin Your Move Developer's
> 					challenge
> 					Build the coolest Linux 
> based applications with Moblin SDK
> 					 &
> 					 win
> 					
> 					 great
> 					prizes
> 					Grand prize is a trip 
> for two to an Open Source event anywhere in the world
> 					
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> 					
> _______________________________________________
> 					Nunit-users mailing list
> 					
> [email protected]
> 					
> https://lists.sourceforge.net/lists/listinfo/nunit-users
> 					
> 
> 					
> 					
> --------------------------------------------------------------
> -----------
> 					This SF.Net email is 
> sponsored by the Moblin Your Move Developer's
> 					challenge
> 					Build the coolest Linux 
> based applications with Moblin SDK & win great
> 					prizes
> 					Grand prize is a trip 
> for two to an Open Source event anywhere in the world
> 					
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> 					
> _______________________________________________
> 					Nunit-users mailing
> 					
> 					 list
> 					
> [email protected]
> 					
> https://lists.sourceforge.net/lists/listinfo/nunit-users
> 					
> 
> 				
> 			
> --------------------------------------------------------------
> -----------
> 			This SF.Net email is sponsored by the 
> Moblin Your Move Developer's
> 			challenge
> 			Build the coolest Linux based 
> applications with Moblin SDK & win great
> 			prizes
> 			Grand prize is a trip for two to an 
> Open Source event anywhere in the world
> 			
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> 			_______________________________________________
> 			Nunit-users mailing list
> 			[email protected]
> 			
> https://lists.sourceforge.net/lists/listinfo/nunit-users
> 			
> 
> 		
> 
> 



-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/