Re: 3.0.20 regression testing

Ian Silvester <[email protected]> wrote:

> Following Fabian's prompting I've explored using privoxy-regression-test.pl
> and thought I should start a thread where we discuss results.
> 
> First of all, Fabian you're a star for having written this thing - I feel
> very strongly that good testing is a prerequisite of a new release and this
> tool gives us confidence as to whether the new release at least does not break
> any existing filtering, and does so with great ease! Lee if it's not too much
> of a pain please run the test set on Windows (no doubt Fabian you do so on BSD!).

Thanks for the kind words.

One thing to keep in mind is that privoxy-regression-test.pl can
only test a small subset of things that could be theoretically
tested, though. As far as "coverage" is concerned we are probably
still at the "well it's better than nothing" stage.

For example we currently have no way to automatically test the
effect of filters (TODO #1) and another serious limitation is
that privoxy-regression-test.pl doesn't help with HTTP compliance
testing (with a few exceptions).

The latter issue is where the previously mentioned curl test suite
will shine, but it's still work in progress:
http://curl.haxx.se/mail/lib-2012-11/0145.html

By the way you were right about me mainly testing on FreeBSD,
and something privoxy-regression-test.pl wasn't explicitly designed
to detect but did anyway as a side effect was a recent regression
in FreeBSD's TCP/IP stack that has since been temporarily reverted:
http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/173309

> So then, I ran the standard test set on OS X using the following command line
> (the privoxy address is necessary since the OS X command line does not respect
> the System Preference proxy config and hence p.p is not routeable):
> 
> /privoxy-regression-test.pl --privoxy-address 127.0.0.1:8118 --show-skipped-tests --max-level 109
> 
> The results were pretty good - only one failure:
> 
> 2013-01-03 11:55:58: Asking Privoxy for the number of action files available ...
> 2013-01-03 11:55:58: Gathering regression tests from 3 action file(s) delivered by Privoxy.
> 2013-01-03 11:55:58: Executing regression tests ...
> 2013-01-03 11:55:59: Failure for test 72/72/0/72. Supposedly-blocked URL: 'http://www91.intel.com/b/ss/intelcorp,intelcorpdc/...ONDataProvider.aspx%3FDownloadType%3DDrivers&ot=A&AQE=1'
> 2013-01-03 11:56:02: Skipping test 345: Ignore flag is set.
> 2013-01-03 11:56:02: Skipping test 346: Ignore flag is set.
> 2013-01-03 11:56:02: Skipping test 347: Ignore flag is set.
> 2013-01-03 11:56:02: Skipping test 348: Ignore flag is set.
> 2013-01-03 11:56:02: Skipping test 349: Ignore flag is set.
> 2013-01-03 11:56:02: Executed 345 regression tests. Skipped 5. 344 successes, 1 failures.

Given the 3 action files and the 345 executed tests I assume that
you tested with default.action (which happens to contain 345 tests)
and two other action files that don't have any additional tests.

This is supposed to cause test failures if sections in default.action
whose effects are tested are overruled.

It's a valid use case, though, as that's helpful for users with a
modified configuration to verify that they don't unintentionally
overrule sections they might care about.

It also a good idea in general to try to write tests for custom
action file changes as well, to be able to quickly test that they
work as expected and keep working as expected in case of changes
in the future.

What's currently missing for this is proper support to signal
privoxy-regression-test.pl something like "I intentionally cause
earlier tests in other files to fail with these URLs, so don't
execute the tests".

The options in this situation are currently removing or disabling the
test, ignoring the test failure, or removing the modification that
intentionally causes the test failure.

> The thing is, this being the first time I've run the test I don't know if this
> is a regression that 3.0.20 introduced (maybe due to the new /download unblocker
> from actionsfile feedback #3498129?). I guess I'll downgrade to 3.0.19 and retest.

Assuming you are indeed using custom action files, I'd start
testing without them first.

You could also check:
http://config.privoxy.org/show-url-info?url=http%3A%2F%2Fwww91.intel.com%2Fb%2Fss%2Fintelcorp%2Cintelcorpdc%2F...ONDataProvider.aspx%253FDownloadType%253DDrivers%26ot%3DA%26AQE%3D1
to quickly see what Privoxy thinks about the URL.

It's working for me as expected:

fk@r500 ~/git/privoxy $privoxy-regression-test.pl --test-number 72 --verbose
2013-01-03 20:15:44: Asking Privoxy for the number of action files available ...
2013-01-03 20:15:44: Gathering regression tests from 9 action file(s) delivered by Privoxy.
2013-01-03 20:15:45: Executing regression tests ...
2013-01-03 20:15:45: Success for test 0/72/0/72. Supposedly-blocked URL: 'http://www91.intel.com/b/ss/intelcorp,intelcorpdc/...ONDataProvider.aspx%3FDownloadType%3DDrivers&ot=A&AQE=1'
2013-01-03 20:15:45: Executed 1 regression tests. Skipped 1027. 1 successes, 0 failures.

I usually configure Privoxy with FEATURE_EXTENDED_HOST_PATTERNS, though,
so we are probably not testing the same Privoxy code and theoretically
this could also explain test differences (in this case it probably
doesn't, though).

> Looking forward to others' results!

Using only the official files:

actionsfile match-all.action
actionsfile default.action
actionsfile regression-tests.action

I get:

fk@r500 ~/git/privoxy $privoxy-regression-test.pl --max-level 109 --show-skipped-tests
2013-01-03 20:26:44: Asking Privoxy for the number of action files available ...
2013-01-03 20:26:44: Gathering regression tests from 3 action file(s) delivered by Privoxy.
2013-01-03 20:26:44: Executing regression tests ...
2013-01-03 20:26:54: Skipping test 467: depends on 'FEATURE_CONNECTION_KEEP_ALIVE' being set to 'No'.
2013-01-03 20:26:54: Skipping test 468: depends on 'FEATURE_CONNECTION_KEEP_ALIVE' being set to 'No'.
2013-01-03 20:26:54: Skipping test 470: depends on config line matching: 'connection-sharing\s+1'.
2013-01-03 20:26:54: Skipping test 471: depends on config line matching: 'connection-sharing\s+1'.
2013-01-03 20:26:54: Skipping test 475: depends on 'FEATURE_CONNECTION_KEEP_ALIVE' being set to 'No'.
2013-01-03 20:26:54: Skipping test 484: depends on 'FEATURE_GRACEFUL_TERMINATION' being set to 'Yes'.
2013-01-03 20:26:54: Skipping test 509: depends on 'FEATURE_TOGGLE' being set to 'No'.
2013-01-03 20:26:57: Executed 658 regression tests. Skipped 7. 658 successes, 0 failures.

And with my standard configuration:

fk@r500 ~/git/privoxy $privoxy-regression-test.pl --max-level 200 --show-skipped-tests
2013-01-03 20:27:56: Asking Privoxy for the number of action files available ...
2013-01-03 20:27:56: Gathering regression tests from 9 action file(s) delivered by Privoxy.
2013-01-03 20:27:57: Executing regression tests ...
2013-01-03 20:27:59: Failure for test 104/104/0/104. Supposedly-blocked URL: 'http://rover.ebay.com/ar/1/2/3?mpt=123&adtype=1&size=728x90'
2013-01-03 20:28:02: Ooops. '-block' is not among the final results.
2013-01-03 20:28:02: Failure for test 217/217/0/217. Sticky Actions: '-block' and URL: 'http://apps.facebook.com/onthefarm/track.php?creative=&cat=friendvisit&subcat=weeds&key=a789a971dc687bee4c20c044834fabdd&next=index.php%3Fref%3Dnotif%26visitId%3D898835505'
2013-01-03 20:28:06: Skipping test 399: Ignore flag is set.
2013-01-03 20:28:06: Skipping test 409: Ignore flag is set.
2013-01-03 20:28:06: Skipping test 410: Ignore flag is set.
2013-01-03 20:28:08: Skipping test 482: depends on 'FEATURE_EXTERNAL_FILTERS' being set to 'Yes'.
2013-01-03 20:28:08: Skipping test 483: depends on 'FEATURE_EXTERNAL_FILTERS' being set to 'Yes'.
2013-01-03 20:28:08: Skipping test 484: depends on 'FEATURE_EXTERNAL_FILTERS' being set to 'Yes'.
2013-01-03 20:28:09: Skipping test 509: Ignore flag is set.
2013-01-03 20:28:09: Skipping test 510: Ignore flag is set.
2013-01-03 20:28:14: Skipping test 771: depends on 'FEATURE_CONNECTION_KEEP_ALIVE' being set to 'No'.
2013-01-03 20:28:14: Skipping test 772: depends on 'FEATURE_CONNECTION_KEEP_ALIVE' being set to 'No'.
2013-01-03 20:28:14: Skipping test 774: depends on config line matching: 'connection-sharing\s+1'.
2013-01-03 20:28:14: Skipping test 775: depends on config line matching: 'connection-sharing\s+1'.
2013-01-03 20:28:14: Skipping test 779: depends on 'FEATURE_CONNECTION_KEEP_ALIVE' being set to 'No'.
2013-01-03 20:28:14: Skipping test 788: depends on 'FEATURE_GRACEFUL_TERMINATION' being set to 'Yes'.
2013-01-03 20:28:15: Skipping test 813: depends on 'FEATURE_TOGGLE' being set to 'No'.
2013-01-03 20:28:19: Skipping test 1022: Ignore flag is set.
2013-01-03 20:28:19: Skipping test 1024: Ignore flag is set.
2013-01-03 20:28:19: Executed 1011 regression tests. Skipped 17. 1009 successes, 2 failures.

The two test failures here are the result of me intentionally overriding
sections in default.action in my own files without being able to explain
this to privoxy-regression-test.pl as described above.

I usually silence them by setting the ignore flag, but of course
each default.action update removes the flags again and I just
"updated" it before this test ...

Fabian

------------------------------------------------------------------------------
Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS,
MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current
with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
MVPs and experts. ON SALE this month only -- learn more at:
http://p.sf.net/sfu/learnmore_122712

_______________________________________________
Ijbswa-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ijbswa-developers

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (FreeBSD)

iEYEARECAAYFAlDl5DIACgkQSMVSH78upWOOsACeKsD1Bz/i1SbNcHonle5GeWv/
P+IAnjNXcKGwXtI+6CHv67C/b6ffVSCz
=4SVB
-----END PGP SIGNATURE-----