"My code works perfectly because I do unit tests." This is something all developers like to hear when they ask for the quality control of a software development. Although we have unit tests that allow us to prevent errors in the early stages, should we also be worried about the quality of the unit testing code?
To all the developers of the world, I'm going to reveal one of the great mysteries of programming: unit tests
One of the characteristics that defines a good unit test is that it must be treated with the same professionalism as with a source code. So how do I know if I'm defining my unit tests well? Don't worry, that's what techniques like mutation testing are for.
So what is mutation testing? The concept is very simple: it is a type of test that makes small modifications to our source code known as mutants. The ultimate goal is to kill mutants.
Each mutant that survives is equivalent to a unit test that has not contemplated a case that can lead to an error.
Mutants are created using mutation operators, which mimic common programming errors such as sign changes, zero divisions, inverted conditions, etc. The end goal is to help develop effective tests in order to help identify gaps in the set of tests created. Let's look at an example using the following code:
if (a && b) {
c = 1;
} else {
c = 0;
}
The conditional mutation operator will create the following mutant:
if (a || b) {
c = 1;
} else {
c = 0;
}
The code has been altered in such a way that it now does the opposite of what is expected. So, basically, any unit test that runs that code will fail. If it does not fail, that means we've done something wrong and we have a surviving mutant. So, how can you start?
If we're developing unit tests for Java code, we can use the PIT Mutation Testing tool.
This tool offers a large list of mutation operators to employ in our tests. In addition, we can run it using Maven, Ant, or command line. We just need to have a set of unit tests previously defined in our project.
We can select a set of mutation operators, tell them where the sources and test project are, and the tool will generate a report with the results of the execution of the tests on the mutants. It also calculates mutation coverage, which is nothing more than a relationship between the number of knocked down mutants and survivors.
There's also a plugin for SonarQube which we can use to detect the number of surviving mutants in the code.
Some Advantages:
1) It brings a new kind of bug to the attention of developers.
2) It is the most powerful method for detecting hidden defects, which could be impossible to identify using conventional testing techniques.
4) It brings a good level of error detection towards unit tests.
- 5) It discovers ambiguities in the source code.
- 6) It is more reliable and helps obtain stable systems.
Some Disadvantages:
1) It is extremely expensive and time-consuming since there is a lot of mutant code that needs to be generated.- 2) It is very difficult to apply without an automation tool.
- 3) Each mutation will have the same number of test cases as the original program. Therefore, each mutation may need to be tested against the original test set.
- 4) It implies changes in the source code, so it is not at all applicable for black box tests.
Conclusion
In short, we know the importance of performing unit tests, but we have to take care of them just as we do with the source code. In order to analyze possible mistakes made in our own tests, one of the options we have are the mutation tests. There are tools that implement these types of tests that are free and easy to use.
