Re: Tests

On 12 Nov 2012, at 21:44, Urs Holzer wrote:
> David recently suggested to run Curl's test suite againts Polipo. I 
> decided to implement a similar approach in Python. It is available on 
> Github:
> 
> https://github.com/uholzer/polipo/tree/tests
> 
> Advantages over Curl's test suite:
> - The data sent by Polipo to the server and to the client are both 
> tracked.
> - The behaviour of the client and the server can be controlled tightly, 
> for example sending a request in steps instead of all at once is 
> possible.
> - Polipo's configuration can be customized for each test case.
> 
> Disadvantage:
> - There are no test cases. Adapting Curl's test cases should be possible 
> but is not trivial.
> 
> Running a testcase is done by invoking
> ./runtests.py N
> where N is the number of the test case. (Look into the directory "data" 
> for the test cases.) For now, one can only run a single test case per 
> invocation. The data Polipo sends is not yet verified and handling of 
> other errors (like timeouts) is still missing. Generally, it needs more 
> work, but its a start.
> 
> What do you think?


"There are no test cases" -- that pretty much sums up what I think. :-)

It looks very nice, simple, and with good documentation. But I'm afraid
it is misdirected effort: None of us is going to write a full HTTP test
suite. Will Manley already tried writing a custom test framework for
polipo[1] and, though it did help us track down & fix a polipo
revalidation bug[2], as a long-term polipo testing solution it never
went anywhere.

I[3] suggested curl's test suite because most of curl's existing HTTP
tests should continue to work just the same when run through polipo,
giving us a broad level of confidence in polipo's general HTTP
compliance. To test specific polipo caching behaviour (and regressions
for specific polipo bugs), then yes we'd need separate tests and
additional functionality from the test framework (be it a modified
version of curl's framework, or an entirely separate one).

I think that "confidence in general HTTP compliance" is more important
than "specific polipo caching behaviour" when it comes to modifying
polipo's source code and making new releases; and thus focusing on
curl's test suite would yield more value for the effort.

[1] https://github.com/wmanley/polipo/commits/automated-testing2

[2] https://github.com/jech/polipo/commit/100ed994

[3] Actually the original idea was from Fabian Keil:
http://www.mail-archive.com/[email protected]/msg00976.html


------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov