Quis Custodiet Ipsos Custodies?

I finally gave a presentation at the Nordic Perl Workshop 2010 today. It was a difficult birth, not the presentation, but the conference. It was planned to hold the workshop on May 1 but due to a volcanic eruption in Eyjafjallajökull. The workshop on May 1 was cancelled (instead Salve J. Nilsen got a bunch of Perl hackers to hack on Kaizendo.

When people started asking whether Nordic Perl Workshop 2010 would ever be held in 2010 Jonas Brømsø Nielsen took on the job of organizing it. I instead organized the Reykjavik Digital Freedom Workshop which included a workshop on open genealogy databases,Creative Commons, OpenStreetMap, the Nordic Perl Workshop, and others.

My talk was titled Quis custodiet ipsos custodies?, summarized as follows:

Testing software is a difficult but rewarding task. Software testers have to foresee all that can go wrong and viciously try to make software fail before it hits production in order to prevent failurs in a production environment. Various test procedures and approaches exist to aid the software tester and software developers in general. Unit tests, system integration tests, regression tests, black box testing, box testing, and white box testing.

Software quality matters and because all humans make mistakes we need to test our software. Tests are important but, like the code itself, tests are created by humans so how can we be sure that they can catch all defects in our code? This is an old philosophical question, initially raised by Plato in The Republic, but rephrased by Juvenal in his Satires: “Quis custodiet ipsos custodes?” Who guards the guardians? Who tests the tests?.

One approach to testing the tests is called mutation testing, an approach initially proposed by Richard Lipton. Mutation testing is about making slight modifications to the original software to see whether the tests will catch the modified versions. A popular tool for Java based mutation testing is Jeff Offut’s µJava (muJava). This talk introduces the reimplementation of Offut’s µJava in Perl. The answer to who tests the Java tests, can therefore be: Perl.

A PDF version of the slides I used for my talk.

Creative Commons leyfi Jodimu