What are heuristics? How can you combine them with testing techniques? And how can your use them to enrich your API and UI tests in a systematic way when validating your software?
Meet Heuristics, Quality's Best Friend
I recently conducted some in-depth research on how to improve the quality of my software, which led to thinking about how I could improve my tests in the applications I work on using techniques, tools, or anything else that helps me to more accurately design the test scenarios I need.
I found a lot of interesting solutions, and one that caught my attention is the use of heuristics for API or UI testing. A heuristic is a way to solve a problem, making it easy to understand. It's like creating a guide that helps us remember important information when testing so that we don't overlook anything.
According to the Cambridge Dictionary, heuristic is defined as “(of a method of teaching) allowing students to learn by discovering things themselves and learning from their own experiences rather than by telling them things.”
This is exactly the proposal of several heuristics as applied to software.
Well-Known Heuristics
An interesting and well-known heuristic is VADER, which was created by Stuart Ashman. VADER is focused on testing Rest APIs. Ashman named the heuristic after Darth Vader as a mnemonic (a memorization technique) since the objective of VADER is to work with Verbs, Authorization, Data, Errors, and Responsiveness. It can be said that these are the most important elements when using an API and should be taken into account in our tests.
What Ashman envisioned is represented in the figure below. The goal is to describe your test scenarios based on these five pillars:
Verbs: when working with a Rest API and talking about verbs, we need to know the function of each http verb and if each is used correctly.
Authorization: when talking about authorization, we must be able to identify all security flaws.
Data: we need to understand the return structure of this data, as well its parameter types, paginations, expected character size, and all types of data that are received by the endpoints.
Responsiveness: we must ensure that the behavior of the API is as expected; this includes request failures, timeouts, and especially scenarios where it does not respond satisfactorily to a large number of requests.
Errors: we need to be able to easily identify the cause of errors; in this way, all errors need to be properly handled and their status code must be used properly.
Adapted image based on Stuart Ashman diagram
More information on VADER can be found in a detailed article on Medium by Vanessa Redes.
Another very interesting heuristic called PAMMDA was developed by Gabriel Santos to test the integration of channels. It focuses on the following:
System Parameters: work with system parameters.
API: validate the API resources used for the front end application.
Chat Manipulation: work with chats in the application.
Messages: ensure system messages appear correctly.
Performance: test whether messages are sent and received successfully.
Automation: ensure that system automations are working as desired.
You can find other heuristics on Julio de Lima's YouTube channel and on his Revista TSPI website.
It is important to mention that VADER and PAMMDA will not always solve your problems; sometimes you may need to test for more than these heuristics cover, and in this case, you may want to consider developing your own heuristic!
Applying Heuristics
"Heuristics are fascinating in theory, but can I actually apply them in my daily work?" This is the question I asked myself when I started to dive deep into these issues. I thought: "I already use a lot of testing techniques, some of which are extremely complex, so why would I apply other techniques? What if I start using a lot of heuristics and forget about the techniques I already use?"
It was with this in mind that Conor Fitzgerald spoke about the relationship between heuristics and testing techniques and gave some examples of how the two can be complementary. In other words, we should use our current testing techniques allied with heuristics. Some examples can help clarify what this might look like:
- Validate error messages: The error messages are system outputs through which we can understand if the code is treating user inputs correctly. In tests that use equivalence partitioning, inputs and outputs are very important points to note.
- Test limit values: It's possible to identify problems in the system when using values that exceed the limits of the fields.
- Goldilocks: This is another heuristic that consists of inserting very large, very small, and optimally sized values into fields.
- Violate the data format roles: Put incorrect information in fields like telephone, e-mail, and postal code.
- Start, middle, and end: Modify the position and type of some elements. Put a String in place of Int, insert a special character in the middle of a number field, or exclude items in the beginning or end of a list.
These examples show us that there are virtually limitless solutions to validating applications, both simple and complex. The goal of heuristics is to make it easy to remember a testing solution when you come across a new application, when you have to repeat your tests in other environments, or when you have code updates.
Heuristics provide a really cool way to develop a new skill, especially for us QA Engineers, as they change the way we think about software and force us to be more creative. This is particularly important because we always need to think about the best way to validate our applications without our work becoming exhaustive and imprecise.
