Re: Continuous Integration: Running tests with Travis CI

On Wed, 9 Oct 2013 17:22:02 -0700
Marc Abramowitz <[email protected]> wrote:

> I think the main thing is we have to rig up the FreeTDS unit tests so
> that they can get the server credentials from environment variables
> rather than the PWD file. This is so we can use built-in encryption
> features of Travis CI to keep the credentials secret so they are not
> abused. 

Hi Marc, 

As long as the credentials are passed in the clear, they're not
secure.  Keeping them in the process's environment isn't much better.
It might be better to publish the credentials and use permissioning on
the server to ensure that the user can only run the unit tests.  

One way to do that would be to use stored procedures.  That would be a
tedious, but not *very* difficult, change in db-lib, because the SQL is
external to the test.  Convert the SQL to procedures, and change the
SQL files to execute them.  The other tests would be more involved,
perhaps 20 hours of work.  

The db-lib tests also support command-line options for credentials.
Wrappers could retrieve them from wherever and pass them on the command
line.  

A third possibility ISTM is to have the PWD file owned by the testing
process with 0400 permissions.  

> there are several unittest directories and each one duplicates
> logic for reading PWD (and they are very similar but not always
> identical). 

Yeah.  :-/

> If you guys want to refactor the PWD reading logic, that's cool too -
> I just wasn't going to try that because I figured there was a high
> chance of me screwing something up. 

If you want to refactor some of that, I doubt you can do much harm.  At
worst the tests won't work correctly for a while in some nightly
snapshots.  Price of progress.  Hardly a disaster.  

> If you agree with my approach

Anything that gets you where you want to go and is acceptable to the
server's host is OK by me, provided you leave the tests in working
order when you're done.  

> Then you or I would set up a GitHub mirror of Gitorious. 

That would be you.  ;-)  

I probably should point out there's nothing particularly rigorous about
the tests.  The true rigorous testing is carried out every day by
FreeTDS's thousands of users.  

It's not feasible to test e.g. ODBC one function at a time without
writing a whole faux-server.  The tests were written ad hoc according
to need and convenience to demonstrate that some aspect of previously
broken functionality now works.  In my case at least, I know the author
frequently was learning about the function's specified behavior even as
he was writing the test.  

They help to flush out compiler and library portability issues.  They
are sometimes useful to isolate behavior on a user's system, too: we
can ask on this list if such-and-such a test works correctly, thereby
eliminating the entire application stack.  

By no means do they test every branch.  Many tests even report success
based on whether or not the library function returned successfully
without bothering to verify that the output matches an expected
result. 

I don't want to discourage you.  Quite the opposite: the tests leave
open many paths to improvement.  I'm just saying that "all tests
passed" isn't a very high bar at present.  If in the process of
automating their execution in a new environment you see a way to make
them more useful, you should feel free to change them. It's not as
though you'll be defacing a holy writ.  

Regards, 

--jkl