Re: [perl #60642] [CAGE] add a codingstd test to ensure TODOed tests have an RT ticket number

James Keenan via RT wrote:
> On Wed Nov 19 23:13:27 2008, [email protected] wrote:
>> James Keenan via RT wrote:
>>> On Tue Nov 18 10:22:25 2008, [email protected] wrote:
>>>
>>> This will probably be quite challenging.  Let's assume that all tests
>>> are found in files with names ending in '.t'.  Those .t files can be
>>> written in any language, which probably have different ways of
>>> classifying a test as TODO.
>>>
>>> My count tonight is that 1384 .t files in the distribution.  Of these
>>> 524 are *not* found under ./languages/.
>>>
>>> I wonder if we could formulate the specification in this ticket a bit
>>> more precisely before someone embarks on coding.
>> Yes, absolutely.  I just added the basic ticket on the spur of the 
>> moment, to make sure I didn't forget about it.
>>
>> I've been thinking about this.  A few things come to mind, for instance 
>> detecting the language based on the hashbang (if any) or subdirectory 
>> it's in, and invoking a language-specific parser.  And detecting the 
>> cases we can't handle, and skipping those.
>>
>> But to me that sounds like way too much work.  It doesn't really matter 
>> to me whether the ticket number occurs within the TODO output string, a 
>> nearby comment is good enough for me.  So how about skipping all the 
>> above nonsense and just ignoring the test language entirely?  How about 
>> a simple regex-based test that tallies all instances of /TODO/ in the 
>> set of test files, skipping the lines that start with obvious comment 
>> characters, and for each instance, looks for a match of /#\d+/?  It can 
>> even expand the search to also look a couple lines above and below the 
>> TODO line, for additional flexibility.  I think that should be 
>> reasonable for most, if not all, possible test languages.
>>
>> Do you think that would catch all the cases?  
> 
> 
> Two thoughts:
> 
> 1.  We already have code that can detect the existence of TODO in
> certain kinds of files.  Cf. t/codingstd/fixme.t.  Paul Cochrane used
> that a couple of years back to generate hundreds of RTs -- most of which
> are probably still outstanding.  Can we leverage that
> Parrot::Distribution-based code?

I'm sure we can.  Though I'm not really sure why the test you mention is 
limited to only checking C sources.

In fact, PDD07 says:

     If a bug must be fixed soon, use "XXX" and put a
     ticket in the bug tracking system.  This means that each "XXX"
     should have an RT ticket number next to it.

As such, we might just extend fixme.t to look for a nearby ticket 
number, and fail if it can't find one.  And do this everywhere, not just 
C sources or tests.  (PDD07 doesn't seem to be talking about code 
written in a particular language, here.)  This sounds quite doable to 
me, and if it sounds reasonable to everyone else, I'll take a shot at it 
this evening just to see how badly it breaks.

It does also raise the question of how to disambiguate between RT and 
trac tickets mentioned within the parrot sources.  We could just use 
/#\d+/ for both, which makes the test happy, but then humans won't know 
where to go find the ticket.

Mark