Re: Suggestion: generate_random method of validators for fuzz testing

Hi Peter -- sorry I didn't follow up on this before.  I'm just catching 
up on some email I missed, before I got out of town and get more behind 
once again.

Yes, I think fuzz testing on FormEncode validators would be quite 
interesting.  I'm guessing this could look like:

def test_validator(validator, times=1000):
     for i in xrange(times):
         input = generate_random() # maybe a type of input too?
         try:
             result = validator.to_python(input)
         except Invalid:
             pass
         else:
             raise ValueError('This input found valid: %r -> %r' % 
(input, result))


Of course, lots of these will succeed for certain validators, so the 
validators you'd test would have to be specifically chosen.


Peter S Russell wrote:
> Hi,
> 
> I've had an idea for FormEncode, I wonder if there's any interest in
> seeing this implemented.
> 
> The idea is this:  A validator can be seen not just as a function to
> check for valid input, it is also a specification of the data that is
> valid input for a function.  If validators were able to generate data as
> well as check data, then you could easily generate test cases for your
> functions, in a style similar to QuickCheck in Haskell[1].  This is an
> example of Fuzz testing.
> 
> Some example use cases:
> 
> + I have a TurboGears controller method that takes input from a form.
> This method should always return an HTML page regardless of the input,
> The HTML page should either be a response or an error message, but not a
> 500 error (as a result of an unhandled exception).  It is impossible to
> exhaustively test this, so I'm tempted just to write a couple of unit
> tests, and leave it at that.  I almost certainly haven't considered some
> corner cases.  If I could pass it a thousand random data points, I could
> be much more certain that it really worked.
> 
> + I am writing a new validator that subclasses another validator.  A
> sensible test for this validator would be that any valid input to the
> base class, when given to the child should result in an appropriate
> python object, or an appropriate error message.  If I could generate a
> lot of sample data, I could more easily test this.
> 
> + I have a project that requires me to import a huge amount of sensitive
> data to an SQLObject driven database (I really do).  The data is a dump
> from another database in CSV format.  The import function is really a
> complicated validator (it isn't because I was less wise when I wrote the
> thing, but it really should be!).  I want to generate some very large,
> realistic looking sample CSV files, to test uploading data to the DBMS,
> without having to look at real CSV files.  Doing this by hand is quite
> time consuming, and not very effective because I tend to take shortcuts,
> so I'm not likely to exercise all of the code.  My upload function
> provides a specification for the format of the data, so I feel like I'm
> really doing the same thing twice when I write my function to generate
> test data.
> 
> What do people think?  Personally I think this could make formencode
> validators far more useful.
> 
> [1] http://www.cs.chalmers.se/~rjmh/QuickCheck/


-- 
Ian Bicking | [email protected] | http://blog.ianbicking.org

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642