Re: Running ACCESS TEST suite

Hi Arnaud,

Since you have been most active in both the test suite and the code, can 
you reproduce Geoff's results?  I understand you work mostly on Linux, 
perhaps there  are a couple platform dependencies we need to document.  Not 
sure.

Hi Geoff,

First off, thanks for writing and your interest in Tidy.  Although my own 
participation has tapered off to an occasional support role, Tidy continues 
to improve and evolve because we get participation from knowledgeable folks 
like yourself.  Arnaud has really been carrying the ball for the team 
lately (apologies for U.S. football metaphor), so he can best comment on 
the current state of the test suite.  That said, I can provide some 
background that you may find useful.

Be aware, for example, that the test suite has never been an utterly 
rigorous affair.  The simple reason being that no text-based test can 
verify how the markup is rendered on a number of different 
browsers!  Unfortunately, I know of no technique to automate that 
process.  We can have perfect test suite conformance, but still break on 
some new version of IE, Safari, or what have you.

Can you comment on what the correct response should be to fix the suite? 
Specifically, what should change: the expected results, the input markup, 
the result markup (if any) or, finally, the code - and how?

Also, as your 1st email indicates, Tidy has lingering browser-compatibility 
issues.  The stated goal of Tidy is to produce standards compliant markup 
that displays in browser the same as the original.  But, in general, Tidy 
is not great at resolving overlapped elements.  E.g.

<b>Bold Only <i>Bold and Italic</b> Only Italic</i>

Tidy treats this as:
<b>Bold Only <i>Bold and Italic</i> Only Italic</b>

Whereas IE and Firefox treat this as:
<b>Bold Only </b><i><b>Bold and Italic</b> Only Italic</i>

Although the issues are similar, Tidy rightly treats inline elements 
differently than block.  Block elements (div, form, table, tr, pre) each 
have unique properties wrt cleanup.  Within a block element, inline 
elements can be handled as a class.  So far, so good.   What tidy does not 
do, in either case, is identify the enclosing scope in the same way that 
most browsers do.

The above example is one of the oldest bugs in the bug tracker.  Likewise, 
there are some venerable tracker entries describing table and form 
issues.  You are encouraged to work on these issues and submit a patch.

take it easy,
Charlie


At 02:25 PM 12/27/2005 +1100, Geoff Air wrote:

>Oops, I thought I had sailed through the access tests,
>but I made an error in the use of FIND (in WIN32) ...
>
>When I corrected the error in the command script, I
>find that my version of Tidy, compiled from CVS source,
>FAILS in some 34 of the 120 tests, namely -
>
>FAILED [ 2-2-1-1 2-2-1-2 2-2-1-3 2-2-1-4 3-5-2-1 3-5-2-2 3-5-2-3 3-6-1-1 
>3-6-1-2
>3-6-1-4 4-1-1-1 4-3-1-1 4-3-1-2 5-1-2-3 5-3-1-1 5-4-1-1 5-5-1-1 5-5-1-2 
>5-5-1-3
>5-5-1-6 5-6-1-1 5-6-1-2 5-6-1-3 6-3-1-4 6-5-1-2 6-5-1-4 7-2-1-1 7-4-1-1 
>7-5-1-1
>10-1-1-1 10-1-1-2 11-2-1-1 11-2-1-4 11-2-1-7] ...
>
>AND, the downloaded binary has EXACTLY the SAME set of 'failures' ...
>
>As expressed, can anyone shed an light on these 'test failures' ...
>
>Regards,
>
>Geoff.
>
>PS: If others, in WIN32, want to try, here are my two command files.
>As mentioned previously, my accesscasesa.txt is exactly per the CVS
>accesscases.txt, with the FIRST line removed ... be careful to
>correct any 'email' line wrap ... especially the 'for', %TIDY%,
>and 'echo' lines ...
>
>At the end, the variable FAILEDACC contains the FAILED numbers, and
>ACCERR.TXT contains all the OUTPUT, in those cases ...
>
><acctest.cmd>
>@echo off
>REM acctest.cmd - execute all access cases
>echo Running ACCESS TEST suite > ACCERR.TXT
>set FAILEDACC=
>for /F "tokens=1,2*" %%i in (accesscasesa.txt) do call onetesta.cmd %%i 
>%%j %%k
>echo FAILED [%FAILEDACC%] ...
></acctest.cmd>
>
>Note, in the following, you can set various TIDY versions,
>you may have ...
>I copy the runtime to a FOLDER that exists within my PATH.
>In my case the downloaded binary, it is called tidy.exe, and
>my compiled version, tidycvs.exe ... just so I can run the
>tests using various versions ...
>
><ontesta.cmd>
>@echo off
>
>REM Set the version of Tidy you want to use
>REM set TIDY=..\bin\tidy
>REM set TIDY=tidycvs
>set TIDY=tidy
>
>echo Testing %1 %2 %3
>set TESTNO=%1
>set TESTEXPECTED=%2
>set ACCESSLEVEL=%3
>
>set INFILES=.\accessTest\%1.*ml
>set CFGFILE=.\accessTest\cfg_%1.txt
>
>set TIDYFILE=.\tmp\out_%1.html
>set MSGFILE=.\tmp\msg_%1.txt
>
>set HTML_TIDY=
>
>REM If no test specific config file, use default.
>if NOT exist %CFGFILE% set CFGFILE=.\accessTest\cfg_default.txt
>
>REM Get specific input file name
>for %%F in ( %INFILES% ) do set INFILE=%%F
>
>REM Remove any pre-existing test outputs
>if exist %MSGFILE%  del %MSGFILE%
>if exist %TIDYFILE% del %TIDYFILE%
>
>REM Make sure output directory exists
>if NOT exist .\tmp  md .\tmp
>
>REM this has to all one line ...
>%TIDY% -f %MSGFILE% --accessibility-check %ACCESSLEVEL% -config %CFGFILE% 
>--gnu-emacs yes --tidy-mark no -o %TIDYFILE% %INFILE%
>
>@REM output the FIND count to the a result file
>find /c "%TESTEXPECTED%" %MSGFILE% > tempres.txt
>@REM load the find count, token 3, into variable RESULT
>for /F "tokens=3" %%i in (tempres.txt) do set RESULT=%%i
>@REM test the RESULT variable ...
>if "%RESULT%." == "0." goto Err
>if "%RESULT%." == "1." goto done
>echo note - test '%TESTEXPECTED%' found %RESULT% times in file '%INFILE%'
>goto done
>
>:Err
>echo FAILED --- test '%TESTEXPECTED%' not detected in file '%INFILE%'
>type %MSGFILE%
>echo FAILED --- test '%TESTEXPECTED%' not detected in above
>set FAILEDACC=%FAILEDACC% %1
>REM append results to the ACCERR.TXT file
>echo ======================================= >> ACCERR.TXT
>echo %TIDY% -f %MSGFILE% --accessibility-check %ACCESSLEVEL% -config 
>%CFGFILE% --gnu-emacs yes --tidy-mark no -o %TIDYFILE% %INFILE% >> ACCERR.TXT
>echo FAILED --- test '%TESTEXPECTED%' not detected in file '%MSGFILE%', as 
>follows - >> ACCERR.TXT
>type %MSGFILE% >> ACCERR.TXT
>echo FAILED --- test '%TESTEXPECTED%' not detected in above >> ACCERR.TXT
>goto done
>
>:done
></ontesta.cmd>
>
>EOF
>
>_________________________________________________________________
>realestate.com.au: the biggest address in property
>http://ninemsn.realestate.com.au
>
>
>
>-------------------------------------------------------
>This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
>for problems?  Stop!  Download the new AJAX search engine that makes
>searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
>http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
>_______________________________________________
>Tidy-develop mailing list
>[email protected]
>https://lists.sourceforge.net/lists/listinfo/tidy-develop



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click