Re: 3.0.20 regression testing
Ian Silvester <[email protected]>
| Newsgroups | gmane.comp.web.privoxy.devel |
|---|---|
| Message-ID | <[email protected]> |
TL;DR - 3.0.20 passes regression test suite on OS X! For more detail please read on.. On 2013-01-03, at 3:03 PM, Fabian Keil wrote: > Ian Silvester <[email protected]> wrote: > >> Following Fabian's prompting I've explored using privoxy-regression-test.pl >> and thought I should start a thread where we discuss results. >> >> First of all, Fabian you're a star for having written this thing - I feel >> very strongly that good testing is a prerequisite of a new release and this >> tool gives us confidence as to whether the new release at least does not break >> any existing filtering, and does so with great ease! Lee if it's not too much >> of a pain please run the test set on Windows (no doubt Fabian you do so on BSD!). > > Thanks for the kind words. > > One thing to keep in mind is that privoxy-regression-test.pl can > only test a small subset of things that could be theoretically > tested, though. As far as "coverage" is concerned we are probably > still at the "well it's better than nothing" stage. > > For example we currently have no way to automatically test the > effect of filters (TODO #1) and another serious limitation is > that privoxy-regression-test.pl doesn't help with HTTP compliance > testing (with a few exceptions). > > The latter issue is where the previously mentioned curl test suite > will shine, but it's still work in progress: > http://curl.haxx.se/mail/lib-2012-11/0145.html Sterling work! Naturally this'll be extremely useful once available. > By the way you were right about me mainly testing on FreeBSD, > and something privoxy-regression-test.pl wasn't explicitly designed > to detect but did anyway as a side effect was a recent regression > in FreeBSD's TCP/IP stack that has since been temporarily reverted: > http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/173309 Nice!! >> So then, I ran the standard test set on OS X using the following command line >> (the privoxy address is necessary since the OS X command line does not respect >> the System Preference proxy config and hence p.p is not routeable): >> >> /privoxy-regression-test.pl --privoxy-address 127.0.0.1:8118 --show-skipped-tests --max-level 109 >> >> The results were pretty good - only one failure: >> >> 2013-01-03 11:55:58: Asking Privoxy for the number of action files available ... >> 2013-01-03 11:55:58: Gathering regression tests from 3 action file(s) delivered by Privoxy. >> 2013-01-03 11:55:58: Executing regression tests ... >> 2013-01-03 11:55:59: Failure for test 72/72/0/72. Supposedly-blocked URL: 'http://www91.intel.com/b/ss/intelcorp,intelcorpdc/...ONDataProvider.aspx%3FDownloadType%3DDrivers&ot=A&AQE=1' >> 2013-01-03 11:56:02: Skipping test 345: Ignore flag is set. >> 2013-01-03 11:56:02: Skipping test 346: Ignore flag is set. >> 2013-01-03 11:56:02: Skipping test 347: Ignore flag is set. >> 2013-01-03 11:56:02: Skipping test 348: Ignore flag is set. >> 2013-01-03 11:56:02: Skipping test 349: Ignore flag is set. >> 2013-01-03 11:56:02: Executed 345 regression tests. Skipped 5. 344 successes, 1 failures. > > Given the 3 action files and the 345 executed tests I assume that > you tested with default.action (which happens to contain 345 tests) > and two other action files that don't have any additional tests. > > This is supposed to cause test failures if sections in default.action > whose effects are tested are overruled. Ah yes, I see that now! I tested 3.0.19 vanilla and didn't get the failure, then added back into user.action the unblocker for actionsfile #3498129 and bingo, got the failure once more. I then tested 3.0.20 vanilla (indeed I'd done an upgrade the first time around and hence my custom user.action was in force at the time) and, with that actionsfile patch in default.action and no longer in user.action, got no test failures, so all is well in that regard. > It's a valid use case, though, as that's helpful for users with a > modified configuration to verify that they don't unintentionally > overrule sections they might care about. > > It also a good idea in general to try to write tests for custom > action file changes as well, to be able to quickly test that they > work as expected and keep working as expected in case of changes > in the future. > > What's currently missing for this is proper support to signal > privoxy-regression-test.pl something like "I intentionally cause > earlier tests in other files to fail with these URLs, so don't > execute the tests". > > The options in this situation are currently removing or disabling the > test, ignoring the test failure, or removing the modification that > intentionally causes the test failure. > >> The thing is, this being the first time I've run the test I don't know if this >> is a regression that 3.0.20 introduced (maybe due to the new /download unblocker >> from actionsfile feedback #3498129?). I guess I'll downgrade to 3.0.19 and retest. > > Assuming you are indeed using custom action files, I'd start > testing without them first. > > You could also check: > http://config.privoxy.org/show-url-info?url=http%3A%2F%2Fwww91.intel.com%2Fb%2Fss%2Fintelcorp%2Cintelcorpdc%2F...ONDataProvider.aspx%253FDownloadType%253DDrivers%26ot%3DA%26AQE%3D1 > to quickly see what Privoxy thinks about the URL. > > It's working for me as expected: > > fk@r500 ~/git/privoxy $privoxy-regression-test.pl --test-number 72 --verbose > 2013-01-03 20:15:44: Asking Privoxy for the number of action files available ... > 2013-01-03 20:15:44: Gathering regression tests from 9 action file(s) delivered by Privoxy. > 2013-01-03 20:15:45: Executing regression tests ... > 2013-01-03 20:15:45: Success for test 0/72/0/72. Supposedly-blocked URL: 'http://www91.intel.com/b/ss/intelcorp,intelcorpdc/...ONDataProvider.aspx%3FDownloadType%3DDrivers&ot=A&AQE=1' > 2013-01-03 20:15:45: Executed 1 regression tests. Skipped 1027. 1 successes, 0 failures. > > I usually configure Privoxy with FEATURE_EXTENDED_HOST_PATTERNS, though, > so we are probably not testing the same Privoxy code and theoretically > this could also explain test differences (in this case it probably > doesn't, though). > >> Looking forward to others' results! > > Using only the official files: > > actionsfile match-all.action > actionsfile default.action > actionsfile regression-tests.action > > I get: > > fk@r500 ~/git/privoxy $privoxy-regression-test.pl --max-level 109 --show-skipped-tests > 2013-01-03 20:26:44: Asking Privoxy for the number of action files available ... > 2013-01-03 20:26:44: Gathering regression tests from 3 action file(s) delivered by Privoxy. > 2013-01-03 20:26:44: Executing regression tests ... > 2013-01-03 20:26:54: Skipping test 467: depends on 'FEATURE_CONNECTION_KEEP_ALIVE' being set to 'No'. > 2013-01-03 20:26:54: Skipping test 468: depends on 'FEATURE_CONNECTION_KEEP_ALIVE' being set to 'No'. > 2013-01-03 20:26:54: Skipping test 470: depends on config line matching: 'connection-sharing\s+1'. > 2013-01-03 20:26:54: Skipping test 471: depends on config line matching: 'connection-sharing\s+1'. > 2013-01-03 20:26:54: Skipping test 475: depends on 'FEATURE_CONNECTION_KEEP_ALIVE' being set to 'No'. > 2013-01-03 20:26:54: Skipping test 484: depends on 'FEATURE_GRACEFUL_TERMINATION' being set to 'Yes'. > 2013-01-03 20:26:54: Skipping test 509: depends on 'FEATURE_TOGGLE' being set to 'No'. > 2013-01-03 20:26:57: Executed 658 regression tests. Skipped 7. 658 successes, 0 failures. Running with the same three actionfiles configured I get: ./privoxy-regression-test.pl --privoxy-address 127.0.0.1:8118 --show-skipped-tests --max-level 109 2013-01-03 15:33:40: Asking Privoxy for the number of action files available ... 2013-01-03 15:33:40: Gathering regression tests from 3 action file(s) delivered by Privoxy. 2013-01-03 15:33:40: Executing regression tests ... 2013-01-03 15:33:46: Skipping test 467: depends on 'FEATURE_CONNECTION_KEEP_ALIVE' being set to 'No'. 2013-01-03 15:33:46: Skipping test 468: depends on 'FEATURE_CONNECTION_KEEP_ALIVE' being set to 'No'. 2013-01-03 15:33:46: Skipping test 469: depends on config line matching: 'connection-sharing\s+0'. 2013-01-03 15:33:46: Skipping test 470: depends on config line matching: 'connection-sharing\s+1'. 2013-01-03 15:33:46: Skipping test 471: depends on config line matching: 'connection-sharing\s+1'. 2013-01-03 15:33:46: Skipping test 475: depends on 'FEATURE_CONNECTION_KEEP_ALIVE' being set to 'No'. 2013-01-03 15:33:46: Skipping test 484: depends on 'FEATURE_GRACEFUL_TERMINATION' being set to 'Yes'. 2013-01-03 15:33:46: Skipping test 509: depends on 'FEATURE_TOGGLE' being set to 'No'. 2013-01-03 15:33:47: Skipping test 563: depends on config line matching: 'user-manual\s+(\.?\.?/|[A-Za-z]:)'. 2013-01-03 15:33:47: Skipping test 564: depends on config line matching: 'user-manual\s+(\.?\.?/|[A-Za-z]:)'. 2013-01-03 15:33:47: Skipping test 565: depends on config line matching: 'user-manual\s+(\.?\.?/|[A-Za-z]:)'. 2013-01-03 15:33:47: Skipping test 566: depends on config line matching: 'user-manual\s+(\.?\.?/|[A-Za-z]:)'. 2013-01-03 15:33:47: Skipping test 567: depends on config line matching: 'user-manual\s+(\.?\.?/|[A-Za-z]:)'. 2013-01-03 15:33:47: Skipping test 568: depends on config line matching: 'user-manual\s+(\.?\.?/|[A-Za-z]:)'. 2013-01-03 15:33:47: Skipping test 569: depends on config line matching: 'user-manual\s+(\.?\.?/|[A-Za-z]:)'. 2013-01-03 15:33:47: Skipping test 570: depends on config line matching: 'user-manual\s+(\.?\.?/|[A-Za-z]:)'. 2013-01-03 15:33:47: Skipping test 571: depends on config line matching: 'user-manual\s+(\.?\.?/|[A-Za-z]:)'. 2013-01-03 15:33:47: Skipping test 572: depends on config line matching: 'user-manual\s+(\.?\.?/|[A-Za-z]:)'. 2013-01-03 15:33:47: Skipping test 573: depends on config line matching: 'user-manual\s+(\.?\.?/|[A-Za-z]:)'. 2013-01-03 15:33:47: Skipping test 574: depends on config line matching: 'user-manual\s+(\.?\.?/|[A-Za-z]:)'. 2013-01-03 15:33:47: Skipping test 575: depends on config line matching: 'user-manual\s+(\.?\.?/|[A-Za-z]:)'. 2013-01-03 15:33:47: Skipping test 576: depends on config line matching: 'user-manual\s+(\.?\.?/|[A-Za-z]:)'. 2013-01-03 15:33:47: Skipping test 577: depends on config line matching: 'user-manual\s+(\.?\.?/|[A-Za-z]:)'. 2013-01-03 15:33:47: Skipping test 578: depends on config line matching: 'user-manual\s+(\.?\.?/|[A-Za-z]:)'. 2013-01-03 15:33:47: Skipping test 579: depends on config line matching: 'user-manual\s+(\.?\.?/|[A-Za-z]:)'. 2013-01-03 15:33:47: Skipping test 580: depends on config line matching: 'user-manual\s+(\.?\.?/|[A-Za-z]:)'. 2013-01-03 15:33:47: Skipping test 581: depends on config line matching: 'user-manual\s+(\.?\.?/|[A-Za-z]:)'. 2013-01-03 15:33:47: Skipping test 582: depends on config line matching: 'user-manual\s+(\.?\.?/|[A-Za-z]:)'. 2013-01-03 15:33:47: Skipping test 583: depends on config line matching: 'user-manual\s+(\.?\.?/|[A-Za-z]:)'. 2013-01-03 15:33:48: Executed 636 regression tests. Skipped 29. 636 successes, 0 failures. The extra 22 skips I get evidently refer to a config line I have not enabled, but for my money this is close enough to state that 3.0.20 passes regression testing on the OS X platform. Huzzah! Ian > > And with my standard configuration: > > fk@r500 ~/git/privoxy $privoxy-regression-test.pl --max-level 200 --show-skipped-tests > 2013-01-03 20:27:56: Asking Privoxy for the number of action files available ... > 2013-01-03 20:27:56: Gathering regression tests from 9 action file(s) delivered by Privoxy. > 2013-01-03 20:27:57: Executing regression tests ... > 2013-01-03 20:27:59: Failure for test 104/104/0/104. Supposedly-blocked URL: 'http://rover.ebay.com/ar/1/2/3?mpt=123&adtype=1&size=728x90' > 2013-01-03 20:28:02: Ooops. '-block' is not among the final results. > 2013-01-03 20:28:02: Failure for test 217/217/0/217. Sticky Actions: '-block' and URL: 'http://apps.facebook.com/onthefarm/track.php?creative=&cat=friendvisit&subcat=weeds&key=a789a971dc687bee4c20c044834fabdd&next=index.php%3Fref%3Dnotif%26visitId%3D898835505' > 2013-01-03 20:28:06: Skipping test 399: Ignore flag is set. > 2013-01-03 20:28:06: Skipping test 409: Ignore flag is set. > 2013-01-03 20:28:06: Skipping test 410: Ignore flag is set. > 2013-01-03 20:28:08: Skipping test 482: depends on 'FEATURE_EXTERNAL_FILTERS' being set to 'Yes'. > 2013-01-03 20:28:08: Skipping test 483: depends on 'FEATURE_EXTERNAL_FILTERS' being set to 'Yes'. > 2013-01-03 20:28:08: Skipping test 484: depends on 'FEATURE_EXTERNAL_FILTERS' being set to 'Yes'. > 2013-01-03 20:28:09: Skipping test 509: Ignore flag is set. > 2013-01-03 20:28:09: Skipping test 510: Ignore flag is set. > 2013-01-03 20:28:14: Skipping test 771: depends on 'FEATURE_CONNECTION_KEEP_ALIVE' being set to 'No'. > 2013-01-03 20:28:14: Skipping test 772: depends on 'FEATURE_CONNECTION_KEEP_ALIVE' being set to 'No'. > 2013-01-03 20:28:14: Skipping test 774: depends on config line matching: 'connection-sharing\s+1'. > 2013-01-03 20:28:14: Skipping test 775: depends on config line matching: 'connection-sharing\s+1'. > 2013-01-03 20:28:14: Skipping test 779: depends on 'FEATURE_CONNECTION_KEEP_ALIVE' being set to 'No'. > 2013-01-03 20:28:14: Skipping test 788: depends on 'FEATURE_GRACEFUL_TERMINATION' being set to 'Yes'. > 2013-01-03 20:28:15: Skipping test 813: depends on 'FEATURE_TOGGLE' being set to 'No'. > 2013-01-03 20:28:19: Skipping test 1022: Ignore flag is set. > 2013-01-03 20:28:19: Skipping test 1024: Ignore flag is set. > 2013-01-03 20:28:19: Executed 1011 regression tests. Skipped 17. 1009 successes, 2 failures. > > The two test failures here are the result of me intentionally overriding > sections in default.action in my own files without being able to explain > this to privoxy-regression-test.pl as described above. > > I usually silence them by setting the ignore flag, but of course > each default.action update removes the flags again and I just > "updated" it before this test ... > > Fabian > ------------------------------------------------------------------------------ > Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, > MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current > with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft > MVPs and experts. ON SALE this month only -- learn more at: > http://p.sf.net/sfu/learnmore_122712_______________________________________________ > Ijbswa-developers mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/ijbswa-developers ------------------------------------------------------------------------------ Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft MVPs and experts. ON SALE this month only -- learn more at: http://p.sf.net/sfu/learnmore_122712