On Various Testing Levels

One of my very first few posts (dates end of 2008) dealt with the premise that One test level is not enough to reach a good qulity product. Then I argued on one hand that Integration tests are not enough, but on the other Unit tests are not enough as well. Only the Power of combining both approaches will truly help make your product successful.

Has anything changed since then?

Since then I had the chance to meet  a lot of great people who taught me a lot about Quality and specifically about testing. I had the chance to meet people like Lisa Crispin which exposed me to the Agile Testing Quadrants, Michael Bolton who taught me the difference between tests and checks and Elisabeth Hendrickson who really showed me that being a good tester takes a lot more than what I imagined.These people also helped me in refining and improving my understanding.
However I still think that relying on a single level of testing will not be an effective strategy.

Checked, Accepted and Explored

I think that I first heard this from Elisabeth Hendrickson:

Every feature (story) developed should be at least “Checked” “Accepted” and explored. Otherwise it is not Done

Checked

For most I use the unit test level to do this. Checked for me means that the code is doing what I think it does. Basically that it will exhibit no unexpected logical error and that the thing I intended it to do when I coded it is indeed what it does.

Accepted

For this I usually try to get as close as I can to E2E. Excluding the GUI. Accepted for me means that the system behave in a useful way. that is the user (or other person) that “ordered” the feature is indeed satisfied that the system behave in what he accept as a good way.

Explored

For this I use manual testing. Explored means that there was a person involved that played around with the system and tried to judge the general quality of the solution. usually there are specific question to answer in this phase, but sometimes this takes the form of free style testing. the goal of this stage is to use the brain power of a tester to see what else can be improved and whether or not the solution is indeed satisfactory.

Is that ALL?

Definitely not!
This is the bare minimum. In many contexts this is not enough. sometimes the system is just too big and needs testing at other levels, there are definitely the non functional requirements that need to be addressed and much more.
However this is a very good place to start. In some projects you can definitely achieve good enough results with only this. And if not, starting with here will give you a strong foundation to base the rest of your test efforts on.

The Twist

But what if were not in an agile context? Well the principles stay the same. I believe that using these 3 simple aspects even when you are not developing an end to end feature will do you good. Even when you are working in an team focusing on a horizontal layer of the product its just a matter of aligning your definition of what is the system under test. Each piece of functionality you deliver still needs to be checked accepted and explored. Its just that the accepted part takes a twist.
Since you are now working on part of the system, most likely your “customer” is not an end user but a fellow developer that needs to either integrate with you work or simply use it. However if you treat him like your customer he can still define his acceptance criteria. Most likely the resulting tests will be at the component (or subsystem) level and will not encompass the entire system. but either way the thing that you deliver will be checked, accepted and explored.

Posted in: ATDD,AUT,Development Practices,Engineering Practices,Integration Tests,Unit Tests