The Matrix Reloaded

I wrote this post quite a long time ago – right on the heels of my original test matrix posts. Why I never posted it is beyond me. I’m posting it now to get it out of my “drafts.”


A few posts back, I discussed The Marick Test Matrix and my minor modifications to the matrix. In those posts, I described how to classify different types of testing into the four quadrants of the matrix. It turns out that you can also use the same matrix to classify testing tools, like this:


Let’s look at each quadrant, in more detail, starting on the right hand side:


This quadrant represents those types of tests that identify defects in business (or non-technical) terms. In other words, you don’t need to be a programmer to figure out that there is a defect.

Typically, these tests are not automated. So, there are no automation tools to discuss, here.


This quadrant represents those types of tests that identify defects in technical terms. In other words, you probably need to be a programmer to figure out that there is a defect. Therefore, one would expect the tools in this quadrant to be highly technical and to require specialized skills. In fact, there are people who specialize in this work. They are typically called testers; but, their knowledge of programming is often greater than the average developer.

The dominant tool in this space is Mercury LoadRunner. Microsoft also has tools in this space, including the Visual Studio Team Test Load Agent and the Microsoft Web Application Stress tool (MS WAS).


This quadrant represents those types of tests that define requirements in business terms. As such, you would expect the tools in this category to be (relatively) non-technical. Business Analysts and end users should be able to use these tools to create automated tests without knowledge of computer programming. In fact, these tests should be written by those team members with the most business expertise.

FIT, FitNesse, STiQ, WebTest and Selenium are all examples of tools that allow tests to be expressed in business terms. All of these tools are well suited to use by Business Analysts.


The testing that takes place in this quadrant defines requirement in technical terms. Therefore, you would expect to see lots of low-level, code-based tools, here. These tools are generally used by computer programmers (e.g. developers and testers).

JUnit and NUnit are the big dogs in this space. Other tools include MSTest, WatiN, MBUnit, xUnit, RSpec (for Ruby), and NUnitASP.



1 response
I remembered why I wrote this. At the time I was thinking about this stuff, a team I was coaching was trying to decide what tools to use to do Business/Requirements testing. I wrote this piece in favor of the FIT/FitNesse or STiQ camp, and to rebut the NUnit/WatiN camp. (Because those tests should be authored by users and analysts, not developers!)

Given the poor experience the team had with FIT and STiQ, and their current course (NUnit and WatiN written by developers and testers based on Test Cases created in Rally by analysts), I'm not sure I still agree with myself.

Maybe there's a puzzle piece missing in the quadrant: Individuals with business facing skills may need help with the technical aspects of software development. If I could draw, I might update the quadrant to show this by bulging the two bottom squares into the two top squares, ala puzzle pieces.

Are there other cross quadrant dependencies that I haven't yet articulated? Probably. What are they? I'll tell you when I figure it out.