Re: Test for potential programmer

2009/12/2 Piers Rowan <[email protected]>

> Carlo Sogono wrote:
>
>> What kinds of tests should I be giving people to measure their
>> aptitude to become good programmers/problem solvers?
>>
>>
>
> There are a variety of online testing options:
>
> http://www.fadvassessments.com/products.pl/skillcheck/it
>
> But....


But many such tests really seem quite dubious.  Just for fun, in first year
Uni (2001), I had Brainbench "Master" level certifications in a wide range
of IT skills that I admit were not only well past my formal training but
also a MASSIVE hyperbole on my actual knowledge and capability.

The ability to succeed in such online tests often factors highly for:
* critical thinking about how to select likely answers in multi-choice or
short answer tests
* psychological and linguistic heuristics for multiple choice tests
* vocabulary recognition and memory capacity
* general rote learning of the subject

probably in about that order.  While some of these may suggest knowledge of
the subject, and possibly even some of the intellectual competencies that
are useful, they are by no means a guarantee, and occasionally have false
negatives as
well as false positives.

I've observed some organizations with (deservedly) great reputations for
software engineering excellence using these tools, but as a very small part
of the recruitment process, a quite general and not always massively useful
filter.

Pen and paper tests, if not carefully thought out, can be as bad or worse.

For people with some programming skills, a *short* exercise involving
semi-realistic code writing can be highly informative in many ways.
Organizations like Topcoder, Google, Facebook, and a few high quality
Australian software teams have obviously known this for some time.  Having
seen what people come up with and how it's analyzed by line managers /
potential team mates, I see great value in this method.

 More generally than that, and particularly if genuinely no existing
programming knowledge is necessary, all I can suggest is think hard about
how to abstract on the particular competencies that will be needed, e.g.

What sort of issues are common for code in the programming language and
problem domain and level (e.g. system, library, application, data
modelling)?  Are we looking at careful meticulous thinking about interfaces
and data validation, strong linear thinking, apply existing mathematical
algorithms or performance, develop completely new algorithms, reasonably
complex code correctness (e.g. concurrency), ability to churn out an
approximately working solution very quickly?  How can you validate the
relevant thinking traits?

If you will care a lot about code readability or documentation, how does
this person communicate and organize their ideas in other formal contexts?
Are they terse, vague, obfuscated, repetitive or can they very cleanly and
completely explain a concept?

How well does this person experiment to learn?  (in other words, if
completely a non-programmer, will they get "the compiler is your friend",
will they be able to use that and do they have the resourcefulness to
identify what they don't know and find the docs on the missing pieces)... in
fact, if they are a programmer, I think many would say resourcefulness
trumps knowing particular APIs.

What is the development process like and how does the candidate break down
problem solving and tasks within the domain they've operated in previously?
Is there a match?  If not, how big an issue is this and how much do they
show the ability and willingness to adapt?

How monolithic and/or long lived is the existing code base (if applicable)
and what sort of reading skills, critical thinking, and systematic approach
to 'following a trail' are they going to need to understand it?

If collaboration is required or you have a pair-programming approach, how
does this person work with others in problem solving?

Is this person dealing with internal clients or external clients (analyst
programmer), designing a technical solution, or just coding within well
defined parameters?  What are the implications for (a) communication skills
(b) foresight and vision (c) any knowledge of what can go right or wrong
with various approaches (and if they don't have this, when, where and how
are they going to get it)



The point being, how can you give them a sensible and realistic test
scenario to show these attributes, or how would you (or they) identify an
isomorphic requirement in their past experience, or inherent personality
trait alignment to the type of approach needed.  If you can hire someone
with good programming experience in the problem domain, so much the better,
but if not, you can still make an educated decision as to likely uptake of
the concepts and approach needed.


Do detailed *verbal* reference checks. A reference check is more important
> than an interview. Use your interview answers to ask questions of the
> referees.
>

+1

Interviewing the referees can be highly informative.  On the surface, most
people will pick referees that will say positive things about them, and
coach the referees accordingly, however to the extent you can ask how, why
and example based questions of referees, they can add a lot of context and
understanding that should be able to substantiate the case for hiring, and
intentionally or unintentionally give away things you need to be aware of.
And get the referee on side, as an education exercise in how to work with
this person, if possible.  If you learn nothing more useful than a yes/no on
hiring from a referee interview, the reference could be too superficial.  As
a recruiter, I've both discovered interesting things for clients on these
calls (with placements that went ahead), and got out of the way to let
clients make the calls themselves, depending on requirements and their
comfort or interest in doing this.  Either way, it should be done, and not
just as a standard check list of things to ask for risk management.



A few thoughts from a teenage x86 assembly hacker, turned serious CS
student, turned IT sales guy, software engineering recruiter and now largely
back towards the programming side of things.


HTH

Cheers,
Jamie

-- 
SLUG - Sydney Linux User Group Mailing List - http://slug.org.au/
Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html