Weak tests. Even with 100% coverage that does not mean the code is completely tested. See weak tests.
What is it?
Mutation testing is a structural testing technique, which uses the structure of the code to guide the testing process. Each mutated version is called a mutant. They need to be killed. Test suites are measured by the percentage of mutants that they kill.
Mutation Testing Types:
- Value Mutations: An attempt to change the values to detect errors in the programs. The most common strategy is to change the constants.
- Decision Mutations: The decisions/conditions are changed to check for the design errors. Like arithmetic operators , relational operators and logical operators(AND, OR, NOT)
- Statement Mutations: Changes done to the statements by deleting or duplicating the line which might arise during copy paste.
What is PIT?
“PIT is a state of the art mutation testing system, providing gold standard test coverage for Java and the jvm. It’s fast, scalable and integrates with modern test and build tooling.”
How does it work?
A traditional coverage is being run first in a separate thread. The mutations are being calculated for the class.
Let’s take one example.
public class Human {
private final int numberOfChocolatesCanEatPerDay = 5;
private int numberOfChocolateEatenToday;
public Human() {
}
public int getNumberOfAirplanesHeCanLift() {
return 0;
}
public void eatChocolate(int numberOfChocolatesToEat) {
if (numberOfChocolatesToEat + numberOfChocolateEatenToday > numberOfChocolatesCanEatPerDay) {
throw new IllegalArgumentException("Can't eat that much");
}
numberOfChocolateEatenToday += numberOfChocolatesToEat;
}
public int getNumberOfChocolateEatenToday() {
return numberOfChocolateEatenToday;
}
public int getNumberOfChocolatesCanEatPerDay() {
return numberOfChocolatesCanEatPerDay;
}
}
The class is visited and relevant methods are being extracted. It goes through the bytecode and checks if the mutators contains instruction codes from it. If so it means the class is a candidate for that mutator. At the end of this iteration we will have the list of mutators. Notice that the actual mutations it’s not happening at this step. After the mutations are calculated they are being assigned the actual unit tests that are available from the traditional coverage step. The actual mutation information is being sent to a “minion” where the mutated bytecode is being created by replacing the original instruction codes with the ones specified by the mutator. The next step is to hotswap the original class with the mutated bytecode. This is done by using a java agent called HotSwapAgent.
The process is:
- for each class
- for each mutant
- run mutant against the unit test
- for each mutant
Let’s see what does the mutants look for our code. Bear in mind that each method will have one or more mutants.
ConditionalsBoundaryMutator creates 1 mutant:
public void eatChocolate(int numberOfChocolatesToEat) {
if(numberOfChocolatesToEat + this.numberOfChocolateEatenToday >= this.numberOfChocolatesCanEatPerDay) {
throw new IllegalArgumentException("Can\'t eat that much");
} else {
this.numberOfChocolateEatenToday += numberOfChocolatesToEat;
}
}
ReturnValsMutator creates 3 mutants:
public int getNumberOfAirplanesHeCanLift() {
return false ?0:1;
}
public int getNumberOfChocolateEatenToday() {
return this.numberOfChocolateEatenToday != 0?0:1;
}
public int getNumberOfChocolatesCanEatPerDay() {
return this.numberOfChocolatesCanEatPerDay != 0?0:1
}
MathMutator creates 2 mutants:
public void eatChocolate(int numberOfChocolatesToEat) {
if (numberOfChocolatesToEat - this.numberOfChocolateEatenToday > this.numberOfChocolatesCanEatPerDay) {
throw new IllegalArgumentException("Can\'t eat that much");
} else {
this.numberOfChocolateEatenToday += numberOfChocolatesToEat;
}
}
public void eatChocolate(int numberOfChocolatesToEat) {
if (numberOfChocolatesToEat + this.numberOfChocolateEatenToday > this.numberOfChocolatesCanEatPerDay) {
throw new IllegalArgumentException("Can\'t eat that much");
} else {
this.numberOfChocolateEatenToday -= numberOfChocolatesToEat;
}
}
NegateConditionalMutator creates 1 mutant:
public void eatChocolate(int numberOfChocolatesToEat) {
if (numberOfChocolatesToEat + this.numberOfChocolateEatenToday <= this.numberOfChocolatesCanEatPerDay) {
throw new IllegalArgumentException("Can\'t eat that much");
} else {
this.numberOfChocolateEatenToday += numberOfChocolatesToEat;
}
}
The junit test used:
public class HumanTest {
@Test
public void testHumanCannotLiftPlanes() {
Human human = new Human();
assertTrue("Humans should not be able to lift planes", human.getNumberOfAirplanesHeCanLift() == 0);
}
@Test(expected = IllegalArgumentException.class)
public void testHumanCannotEatMoreThanHisLimit() {
Human human = new Human();
human.eatChocolate(15);
}
@Test
public void testHumanCanEatUnderHisLimit() {
Human human = new Human();
int number = 3;
human.eatChocolate(number);
assertTrue("Human not full", human.getNumberOfChocolateEatenToday() == number);
}
@Test
public void testHumanCanEatHisLimit() {
Human human = new Human();
int number = 5;
human.eatChocolate(number);
assertTrue("Human not full", human.getNumberOfChocolateEatenToday() == human.getNumberOfChocolatesCanEatPerDay());
}
}
Let’s look at the results.
public void eatChocolate(int numberOfChocolatesToEat) {
if(numberOfChocolatesToEat - this.numberOfChocolateEatenToday > this.numberOfChocolatesCanEatPerDay)
{
throw new IllegalArgumentException("Can\'t eat that much");
} else {
this.numberOfChocolateEatenToday += numberOfChocolatesToEat;
}
}
Indeed if we change the +(plus) to a -(minus) in our class the unit tests pass. What did we miss? Well this can be a little tricky. It may not be obvious and could take some time to figure it out what we left out. Fortunately for this example we forgot to test 2 cases:
@Test
public void testHumanCanEatUntilHeIsFull() {
Human human = new Human();
int number = 3;
human.eatChocolate(number);
assertTrue("Human not full", human.getNumberOfChocolateEatenToday() == number);
assertTrue("Human can still eat", human.getNumberOfChocolatesCanEatPerDay() - human.getNumberOfChocolateEatenToday() == 2);
human.eatChocolate(2);
assertTrue("Human full", human.getNumberOfChocolateEatenToday() == human.getNumberOfChocolatesCanEatPerDay());
}
@Test(expected = IllegalArgumentException.class)
public void testHumanCannotEatMoreThanHisLimitCase2() {
Human human = new Human();
human.eatChocolate(5);
assertTrue("Human full", human.getNumberOfChocolateEatenToday() == human.getNumberOfChocolatesCanEatPerDay());
human.eatChocolate(5);
}
Let’s run again pitest.
Check the class as well.
Now that looks good. Should all mutants be killed? Ideally. But hard to achieve practically on real projects. Even 100% coverage is something hard to achieve on those.
All of these come with a performance cost. Like explained above each mutant is being generated on the fly and run against the unit test. And a class usually has many mutants. The performance cost is directly proportional with the codebase size. This is recommended for nightly builds as it supports most of CI environments.
Luckily things can be improved.
- target classes/tests – choose what you want to test; useful before commiting your tests
- incremental analysis – reuse your previous runs
- dependency distance – a test that directly calls a method from a mutant has a distance of 1 ,a test that calls a method on a class that uses the mutee has a distance of 2 and so on so forth.
- number of mutations per class
- mutation filter – filter the mutations based on criterias
Plugins? Sure. IntellijIdea, Eclipse, Maven, Gradle, SonarQube.
Mutation testing does not resolve the everything. It just helps you improve your tests and make your code less buggy. For more information go to www.pitest.org.
Learn. Write. Repeat. 
1 thought on “Mutation testing”