Correctly testing external binaries

[email protected] ("Cosimo Streppone")
Newsgroups perl.qa
Message-ID <op.wrrmgcxxs5ttvb@c0739>
Hi QA folks,

Net::Statsd::Server is a UDP server daemon packaged as CPAN module,
with a little 'bin/statsd' wrapper script.

In the latest release, 0.09, I tried to add some 'integration tests':

   https://metacpan.org/source/COSIMO/Net-Statsd-Server-0.09/t/integration-tests

implemented through the Test::Statsd module:

   https://metacpan.org/source/COSIMO/Net-Statsd-Server-0.09/lib/Test/Statsd.pm

Now, the idea behind this is that I can perform some server-independent
tests. The start_statsd() method forks off a given binary, with a
config file as only argument, and then tests are performed throwing
UDP packets at some local port.
When tests are done, stop_statsd() kills off the child process.

The goal is to be able to test the multitude of statsd server  
implementations
out there just given a binary and a config file.

Immediate problems:

1) binary can be an executable or a script that needs $^X,
    /nodejs/python/... How do I exec() reliably?

    https://metacpan.org/source/COSIMO/Net-Statsd-Server-0.09/t/integration-tests/counts-are-valid.t

2) Omitting $^X is not desirable, because I got a slew of
    CPAN testers fails. #!/usr/bin/perl is obviously a wrong choice
    in so many situations.

    Sample report:
    http://www.cpantesters.org/cpan/report/02669934-6a35-11e2-b5a4-ab61270fb964

Less immediate problems:

3) Should this generic test suite be an independent CPAN distribution?

I'd appreciate any advice you might have.

-- 
Cosimo
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.