Re: 2.5 console runner documenation

Hi Jay,

I haven't heard further from you on this. As noted in a separate
post, I floated the issue on the nunit-discuss list and it was
apparently a non-issue, since nobody responded to two posts there.

If you still have an issue on this, I suggest you jump on the
other thread I just started here. If you can show me why the
recommended approach of using a trivial reformatting app
right after nunit-console runs is not useful in some specific
circumstances, I'll be happy to deal with that. So far, you
only stated it won't work, which isn't making sense to me -
quite possibly a lack of imagination on my part, but I still
need you to help me out if you want anything to change.

Charlie

> -----Original Message-----
> From: [email protected] 
> [mailto:[email protected]] On Behalf 
> Of Charlie Poole
> Sent: Monday, September 08, 2008 2:03 AM
> To: [email protected]
> Subject: Re: [Nunit-users] 2.5 console runner documenation
> 
> Hi Jay, 
> 
> > I'm not sure I see how using a documented feature is out of the 
> > ordinary :)
> >
> > Really, if it's the only way to customize the console output to 
> > produce extra terse output, it'd be a shame to scrap it entirely.  
> > It's like the /b option for the dir command in dos.  Sometimes you 
> > just need something that outputs the bear minimal information.  The 
> > benefit of xslt is that you can go beyond that and customize things 
> > however you like.  I apologize that I wasn't around to voice a vote 
> > when you were looking for opinions of the matter.  And I appreciate 
> > that xslt is a pain to use for something as simple as this.  But I 
> > guarantee if you don't have something to replace its functionality 
> > you're going to get other users grumbling, too.
> >  I can't be the only one that works from the command line, 
> relying on 
> > stdout.
> >
> > Is it that you're not familiar with xslt, and another 
> contributor who 
> > isn't around anymore is the one who added it?
> >  I can appreciate that, too.  But really, it should be a 
> simple matter 
> > to test one way or another on console output.
> > If the transform file was specified, feed the current test's output 
> > XML into it and print the result to stdout.  Otherwise use whatever 
> > system you've built to replace it.  Just because the system can use 
> > xslt doesn't mean YOU have to.  This just seems like a case 
> where you 
> > shouldn't be breaking backwards compatibility for no reason.
> 
> In fact, the xml output is still there for you to use. And 
> xslt will be used to produce reports out of NUnit. It's just 
> not being done in the nunit-console program. 
> > Actually, if NUnit didn't have the transform option on the console, 
> > you and I wouldn't be having this conversation.  I would have found 
> > another unit testing framework that did what I needed.  At the very 
> > least you need to have a little error message saying that 
> -transform 
> > is no longer supported if you try to run the console runner 
> with that 
> > option.
> 
> I'm sorry that the change doesn't seem to work for you. I 
> indicated that there was discussion about it because the 
> upshot of that discussion was that the rearranging of the 
> different facilities - they have not been removed - still 
> allowed everything to be done that was being done and also 
> allowed some new things to be done that couldn't be done 
> before. I'm pretty sure that you would find you can use the 
> new arrangement in pretty much the same way as before if you 
> gave it a chance. 
> 
> > Okay, my little rant over, let's get to the matter at hand.  
> > 1 and 2 won't work, because I need the output to go to the console 
> > before the unit tests can return a return code.
> 
> That's what option 1 would do. Option 2 could be made to do 
> that if you made use of a wrapper script to give the return code.
>   
> > That's really the way the whole chain is set up, and it'd 
> be a major 
> > pain to change things.
> > 
> > #3 sounds promising.  Looking through the source it doesn't 
> seem too 
> > bad.  What's a mile high overview of what ConsoleUi does?  
> It's only 
> > summary statement is: "Summary description for ConsoleUi.", which 
> > isn't terribly descriptive ;)
> 
> You're right that it's not mucy of a description. It should 
> say "Runs the NUnit tests and produces console and xml output. :-)
> 
> The code is fairly clear, I think. You'll find routines that 
> produce part of the console output. However, if you are 
> already using xslt, I'd suggest you continue to do so. From 
> what you have described, it sounds like you only need to make 
> a small change.
>  
> > The other option is the non-action of just sticking with the 2.4.X 
> > fork and lumping it.  Right now there's nothing in 2.5 that I need, 
> > but I'd hate to cut myself off from future updates.
> 
> That is an option. A further option, of course, would be to 
> modify the 2.5 code to work as you think it should.
> 
> However, I would really suggest that you take another look at 
> the five approaches I listed for you. It seems that one of 
> them ought to do the job. In particular, it seems as if you 
> could simply replace your invocation of nunit-console with a 
> script that could produce exactly what you now get, by 
> judicious use of command-line options and/or a separate 
> reformatting program. I can provide you with a utility to do 
> the reformatting if you don't have one handy.
> 
> Charlie
>  
> > -Jay Lemmon
> > 
> > --- On Sun, 9/7/08, Charlie Poole 
> <[email protected]> wrote:
> > 
> > 
> > 	From: Charlie Poole <[email protected]>
> > 	Subject: RE: [Nunit-users] 2.5 console runner documenation
> > 	To: [email protected]
> > 	Date: Sunday, September 7, 2008, 5:32 PM
> > 	
> > 	
> > 	Hi Jay,
> > 	 
> > 	I'll start with the last question: based on feedback 
> from users, most 
> > people
> > 	assumed that the option caused the xml file to be 
> reformatted, not 
> > the
> > 	console output. I got a lot of "bugs" about how using 
> /transform 
> > broke
> > 	the console output and sent html to stdout instead of 
> to the file. 
> > :-)
> > 	 
> > 	The xml is the primary output from nunit and it seemed 
> that anyone
> > 	wanting to reformat it could do so easily enough. My own little 
> > command-line
> > 	utility that reformats the xml using a stylesheet is 
> about 15 lines 
> > long.
> > 	 
> > 	But using something as sophisticated as xml and xslt 
> just to produce 
> > the
> > 	command-line output, which is not intended for 
> reporting purposes 
> > seemed
> > 	like a waste of effort - using a sledge hammer to crack 
> walnuts. More 
> > important,
> > 	it coupled two different things together: console output and 
> > reporting, making
> > 	it very ard to change either one. With a reporting 
> subsystem on the 
> > drawing
> > 	board, I didn't want to have to think about how changes 
> in the xml 
> > would impact
> > 	the nunit-console display.
> > 	 
> > 	Back to your problem, then...
> > 	 
> > 	I see five options...
> > 	 
> > 	1. Use the /xmlconsole switch to send the xml to the 
> console and add 
> > a filter program
> > 	to your toolchain to reformat it.
> > 	 
> > 	2. Use the same external filter program with 
> TestResult.xml as it's 
> > input.
> > 	 
> > 	3. Do the reformatting within your own runner, taking 
> the code from
> > 	ConsoleUi.cs.
> > 	 
> > 	4. Replicate the nunit logic in your own runner - i.e. 
> > copy it - and modify
> > 	the runner to write output in the form you prefer.
> > 	 
> > 	5. Same as #4, but modify the event handlers in 
> EventCollector.cs to
> > 	do the same thing.
> > 	 
> > 	 
> > 	I don't think any of these are particularly harder than 
> what you have
> > 	done - but of course, you've already done that work, so 
> I recognize
> > 	this is extra effort for you. I apologize for that, but 
> I think this 
> > change
> > 	will get us farther in the long run.
> > 	 
> > 	Charlie
> > 	 
> > 	PS: The change was discussed before and at the time of 
> implementing.
> > 	It's good to be involved in such list disussions, 
> particularly if you 
> > know
> > 	you're doing something a little out of the ordinary.
> > 	 
> > 	 
> > 
> > 
> > ________________________________
> > 
> > 		From: Jay Lemmon [mailto:[email protected]] 
> > 		Sent: Sunday, September 07, 2008 3:04 PM
> > 		To: Charlie Poole
> > 		Subject: Re: [Nunit-users] 2.5 console runner 
> documenation
> > 		
> > 		
> > 		Yes, I'm still pretty raw with NUnit, so any 
> help would be 
> > appreciated :)
> > 		
> > 		My tool chain is set to use the console pretty 
> exclusively.  That 
> > means stdout AND return codes.  Also, the stdout needs to 
> be kept very 
> > clean, because I don't want to write a lot of special logic 
> excising 
> > extra data.  So a command line option would probably be too weird.
> > 		
> > 		So that leaves something done in code.  Right 
> now my test runner is 
> > pretty stupid.  It just adds some command line switches 
> automatically 
> > to the console options and feeds it to the console runner library.  
> > How deeply am I going to have to dig in to the code to get what I 
> > need?
> > 		
> > 		Also, out of curiosity, why was the xslt option 
> removed?  It seems 
> > like a step backwards without the corresponding two steps forward.  
> > But then my scope is pretty narrow :)
> > 		
> > 		-Jay Lemmon
> > 		
> > 		--- On Sun, 9/7/08, Charlie Poole
> > <[email protected]> wrote:
> > 		
> > 
> > 			From: Charlie Poole
> > <[email protected]>
> > 			Subject: Re: [Nunit-users] 2.5 console 
> runner documenation
> > 			To: [email protected],
> > [email protected]
> > 			Date: Sunday, September 7, 2008, 2:54 PM
> > 			
> > 			
> > 			That sounds like a valid use case, 
> altough unusual. What's 
> > interesting is that it matches exactly
> > 			the use cases I wrote about earlier: 
> > You want a report in a particular format, but the fact that
> > 			it is produced on stdout by NUnit is 
> just an implementation detail 
> > and not a requirement.
> > 			 
> > 			AFAICS, you can do everything you want 
> using any of the options I 
> > described in the last
> > 			note. If it's not clear, I'll try to 
> supply more detail.
> > 			 
> > 			Charlie
> > 
> > 			...
> > 			
> > 			
> > 			The output is created in code. You 
> could certainly do that yourself 
> > if you like. Essentially, you have two choices: to look at 
> the final 
> > tree of results, as NUnit does, or to handle each test result as it 
> > comes along - which is how NUnit creates it's test output.
> > 			 
> > 			Of course, you can also continue to use 
> xslt as you now do - you 
> > just can't have
> > 			NUnit apply the xslt for you. That's 
> trivial to do at the end of 
> > the test run by taking
> > 			the xml output adn transforming it.
> > 			 
> > 			Charlie
> > 
> > 
> > ________________________________
> > 
> > 				From: 
> > [email protected]
> > [mailto:[email protected]] On Behalf Of Jay 
> > Lemmon
> > 				Sent: Sunday, September 07, 2008 2:29 PM
> > 				To: [email protected]
> > 				Subject: Re: [Nunit-users] 2.5
> > console runner documenation
> > 				
> > 				
> > 				A quick follow up to my last
> > email.  This is the format I'm looking for the console runner to 
> > output.  I need it this way because it mirrors the output 
> from another 
> > unit testing framework (UnitTest++), so it makes everything 
> nice and 
> > consistent for my tool chain.
> > 				
> > 				
> > c:\glassworksstudio\lodestone\source\trunk\unittests\quarticte
> > sts.cpp(56): error: Failure in Fixture1.Test_1: Expected 0 but was 2
> > 				
> > c:\glassworksstudio\lodestone\source\trunk\unittests\quarticte
> > sts.cpp(63): error: Failure in Fixture1.Test_2: 20 < 10
> > 				FAILURE: 2 out of 156 tests
> > failed (2 failures).
> > 				Test time: 0.02 seconds.
> > 				
> > 				Note each test failure follows
> > this format:
> > 				[file]([line]): error: Failure
> > in [TestFixture].[TestName]: [Error message]
> > 				all inline, with a new line for
> > each failed test.  Then a summary report and the time to 
> run the tests 
> > comes last.
> > 				
> > 				-Jay Lemmon
> > 				
> > 				--- On Sun, 9/7/08, Jay Lemmon
> > <[email protected]> wrote:
> > 				
> > 
> > 					From: Jay Lemmon
> > <[email protected]>
> > 					Subject: Re: 
> > [Nunit-users] 2.5 console runner documenation
> > 					To: 
> > [email protected]
> > 					Date: Sunday, September
> > 7, 2008, 2:16 PM
> > 					
> > 					
> > 					Well I can't speak for
> > anyone else, but customizing the (console) output of the console 
> > runner is important to me.  If it's impossible in 2.5, it'll mean I 
> > can't upgrade :(
> > 					
> > 					It doesn't have to be
> > xslt.  And it doesn't have to be a command line switch.  
> > Right now I'm actually linking against the console runner 
> assemblies 
> > (DLLs) to build my own custom EXE for every test project I 
> build, so 
> > it could be something done purely in code.  How is the 
> format for the 
> > console runner decided right now if it's not xslt?
> > 					
> > 					-Jay Lemmon
> > 					
> > 					--- On Sun, 9/7/08,
> > Charlie Poole <[email protected]> wrote:
> > 					
> > 
> > 					From: Charlie Poole
> > <[email protected]>
> > 					Subject: Re: 
> > [Nunit-users] 2.5 console runner documenation
> > 					To: 
> > [email protected]
> > 					Date: Sunday, September
> > 7, 2008, 1:10 PM
> > 					
> > 					
> > 					Hi Jay,
> > 					 
> > 					There doesn't seem to
> > be a real need to customize console output, except possibly to have
> > 					NUnit show more
> > detailed info - an option I'm working on. Most people using the old 
> > feature
> > 					seemed to be producing
> > html with it. That's still possible, but the xml would need to be 
> > 					formatted after the NUnit run.
> > 					 
> > 					Charlie
> > 
> > 
> > ________________________________
> > 
> > 					From: 
> > [email protected]
> > [mailto:[email protected]] On Behalf Of Jay 
> > Lemmon
> > 					Sent: Sunday, September
> > 07, 2008 12:54 PM
> > 					To: 
> > [email protected]
> > 					Subject: Re: 
> > [Nunit-users] 2.5 console runner documenation
> > 					
> > 					
> > 					I just copied the 2.5
> > binaries over my old binaries, so it wasn't a fool proof test, no :)
> > 					
> > 					So how do I, the end
> > user, customize console output now?  The release notes 
> don't mention 
> > anything in this regard.
> > 					
> > 					-Jay Lemmon
> > 					
> > 					--- On Sun, 9/7/08,
> > Charlie Poole <[email protected]> wrote:
> > 					
> > 
> > 					From: Charlie Poole
> > <[email protected]>
> > 					Subject: Re: 
> > [Nunit-users] 2.5 console runner documenation
> > 					To: 
> > [email protected]
> > 					Date: Sunday, September
> > 7, 2008, 6:39 AM
> > 					
> > 					
> > 					Hi Jay,
> > 					 
> > 					There's mention in the
> > release notes, under 2.5 alpha 1, about the removal of the transform
> > 					option. The console
> > output is now indepedent of the xml, so it no longer has 
> any purpose.
> > 					 
> > 					Since the option is no
> > longer there, I'm pretty sure it can't be working. :-) Are you sure
> > 					you were using 2.5 for
> > your test?
> > 					 
> > 					Charlie
> > 
> > 
> > ________________________________
> > 
> > 					From: 
> > [email protected]
> > [mailto:[email protected]] On Behalf Of Jay 
> > Lemmon
> > 					Sent: Saturday,
> > September 06, 2008 9:42 PM
> > 					To: 
> > [email protected]
> > 					Subject: [Nunit-users]
> > 2.5 console runner documenation
> > 					
> > 					
> > 					When comparing the
> > 2.4.8 docs and the 2.5 docs, I noticed that this section in console 
> > runner: command-line was mostly deleted and partly merged 
> with the one 
> > before it:
> > 					
> > 					"Specifying the Transform file
> > 					
> > 					The console interface
> > uses XSLT to transform the test results from the XML file 
> to what is 
> > printed to the screen when the program executes. The 
> console interface 
> > has a default transformation that is part of the executable. To 
> > specify your own transformation named "myTransform.xslt" use the 
> > following command line option:
> > 					
> > 					nunit-console
> > /transform:myTransform.xslt nunit.tests.dll
> > 					
> > 					Note: For additional
> > information see the XML schema for the test results. This 
> file is in 
> > the same directory as the executable and is called Results.xsd. The 
> > default transform Summary.xslt is located in the core source 
> > directory."
> > 					
> > 					In the 2.5 docs,
> > there's no such section, and the mention of the Results.xsd 
> is merged 
> > in to the section before it on specifying an output xml file.
> > 					
> > 					Is this an error in the
> > docs, or is this feature (specifying a transform) going to 
> be removed 
> > in 2.5? A quick test shows that it still works.
> > 					
> > 					-Jay Lemmon
> > 						
> > 
> > 
> > 					
> > --------------------------------------------------------------
> > -----------
> > 					This SF.Net email is
> > sponsored by the Moblin Your Move Developer's
> > 					challenge
> > 					Build the coolest Linux
> > based applications with Moblin SDK & win great
> > 					prizes
> > 					Grand prize is a trip
> > for two to an Open Source event anywhere in the world
> > 					
> > http://moblin-contest.org/redirect.php?banner_id=100&url=/
> > 					
> > _______________________________________________
> > 					Nunit-users mailing list
> > 					
> > [email protected]
> > 					
> > https://lists.sourceforge.net/lists/listinfo/nunit-users
> > 					
> > 
> > 					
> > 					
> > --------------------------------------------------------------
> > -----------
> > 					This SF.Net email is
> > sponsored by the Moblin Your Move Developer's
> > 					challenge
> > 					Build the coolest Linux
> > based applications with Moblin SDK
> > 					 &
> > 					 win
> > 					
> > 					 great
> > 					prizes
> > 					Grand prize is a trip
> > for two to an Open Source event anywhere in the world
> > 					
> > http://moblin-contest.org/redirect.php?banner_id=100&url=/
> > 					
> > _______________________________________________
> > 					Nunit-users mailing list
> > 					
> > [email protected]
> > 					
> > https://lists.sourceforge.net/lists/listinfo/nunit-users
> > 					
> > 
> > 					
> > 					
> > --------------------------------------------------------------
> > -----------
> > 					This SF.Net email is
> > sponsored by the Moblin Your Move Developer's
> > 					challenge
> > 					Build the coolest Linux
> > based applications with Moblin SDK & win great
> > 					prizes
> > 					Grand prize is a trip
> > for two to an Open Source event anywhere in the world
> > 					
> > http://moblin-contest.org/redirect.php?banner_id=100&url=/
> > 					
> > _______________________________________________
> > 					Nunit-users mailing
> > 					
> > 					 list
> > 					
> > [email protected]
> > 					
> > https://lists.sourceforge.net/lists/listinfo/nunit-users
> > 					
> > 
> > 				
> > 			
> > --------------------------------------------------------------
> > -----------
> > 			This SF.Net email is sponsored by the 
> Moblin Your Move Developer's
> > 			challenge
> > 			Build the coolest Linux based
> > applications with Moblin SDK & win great
> > 			prizes
> > 			Grand prize is a trip for two to an 
> Open Source event anywhere in 
> > the world
> > 			
> > http://moblin-contest.org/redirect.php?banner_id=100&url=/
> > 			_______________________________________________
> > 			Nunit-users mailing list
> > 			[email protected]
> > 			
> > https://lists.sourceforge.net/lists/listinfo/nunit-users
> > 			
> > 
> > 		
> > 
> > 
> 
> 
> 
> --------------------------------------------------------------
> -----------
> This SF.Net email is sponsored by the Moblin Your Move 
> Developer's challenge Build the coolest Linux based 
> applications with Moblin SDK & win great prizes Grand prize 
> is a trip for two to an Open Source event anywhere in the 
> world http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Nunit-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/nunit-users
> 



-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/