The Matrix Reloaded

I wrote this post quite a long time ago – right on the heels of my original test matrix posts. Why I never posted it is beyond me. I’m posting it now to get it out of my “drafts.”


A few posts back, I discussed The Marick Test Matrix and my minor modifications to the matrix. In those posts, I described how to classify different types of testing into the four quadrants of the matrix. It turns out that you can also use the same matrix to classify testing tools, like this:


Let’s look at each quadrant, in more detail, starting on the right hand side:


This quadrant represents those types of tests that identify defects in business (or non-technical) terms. In other words, you don’t need to be a programmer to figure out that there is a defect.

Typically, these tests are not automated. So, there are no automation tools to discuss, here.


This quadrant represents those types of tests that identify defects in technical terms. In other words, you probably need to be a programmer to figure out that there is a defect. Therefore, one would expect the tools in this quadrant to be highly technical and to require specialized skills. In fact, there are people who specialize in this work. They are typically called testers; but, their knowledge of programming is often greater than the average developer.

The dominant tool in this space is Mercury LoadRunner. Microsoft also has tools in this space, including the Visual Studio Team Test Load Agent and the Microsoft Web Application Stress tool (MS WAS).


This quadrant represents those types of tests that define requirements in business terms. As such, you would expect the tools in this category to be (relatively) non-technical. Business Analysts and end users should be able to use these tools to create automated tests without knowledge of computer programming. In fact, these tests should be written by those team members with the most business expertise.

FIT, FitNesse, STiQ, WebTest and Selenium are all examples of tools that allow tests to be expressed in business terms. All of these tools are well suited to use by Business Analysts.


The testing that takes place in this quadrant defines requirement in technical terms. Therefore, you would expect to see lots of low-level, code-based tools, here. These tools are generally used by computer programmers (e.g. developers and testers).

JUnit and NUnit are the big dogs in this space. Other tools include MSTest, WatiN, MBUnit, xUnit, RSpec (for Ruby), and NUnitASP.

On Clarity and Abstraction in Functional Tests

Consider the following tests:

public void LoginFailsForUnknownUser1()
    string username = "unknown";
    string password = "password";

    bool loginSucceeded = User.Login(username, password);

    Assert.That(loginSucceeded == false);
public void LoginFailsWithUnknownUser2()
    using (var browser = new IE(url))
        browser.TextField(Find.ById(new Regex("UserName"))).Value = "unknown";
        browser.TextField(Find.ById(new Regex("Password"))).Value = "password";

        browser.Button(Find.ById(new Regex("LoginButton"))).Click();
        bool loginSucceeded = browser.Url.Split('?')[0].EndsWith("index.aspx");

        Assert.That(loginSucceeded == false);

Note the similarities:

  • Both methods test the same underlying functional code; and,
  • Both tests are written in NUnit.
  • Both tests use the Arrange / Act / Assert structure.

Note the differences:

  • The first is a unit test for a method on a class.
  • The second is a functional test that tests an interaction with a web page.
  • The first is clear. The second is, um, not.

Abstracting away the browser interaction

So, what’s the problem? Aren’t all browser tests going to have to use code to automate the browser?

Well, yes. But, why must that code be so in our face? How might we express the true intention of the test without clouding it in all the arcane incantations required to automate the browser?

WatiN Page Classes

The folks behind WatiN answered that question with something called a Page class. Basically, you hide all the browser.TextField(…) goo inside a class that represents a single page on the web site. Rewriting the second test using the Page class concept results in this code:

public void LoginFailsWithUnknownUser3()
    using (var browser = new IE(url))
        browser.Page<LoginPage>().UserName.Value = "unknown";
        browser.Page<LoginPage>().Password.Value = "password";

        bool loginSucceeded = browser.Page<IndexPage>().IsCurrentPage;

        Assert.That(loginSucceeded == false);
public class LoginPage : Page
    public TextField UserName
        get { return Document.TextField(Find.ById(new Regex("UserName"))); }
    public TextField Password
        get { return Document.TextField(Find.ById(new Regex("Password"))); }
    public Button LoginButton
        get { return Document.Button(Find.ById(new Regex("LoginButton"))); }

Better? Yes. Now, most of the WatiN magic is tucked away in the LoginPage class. And, you can begin to make out the intention of the test. It’s there at the right hand side of the statements.

But, to me, the Page Class approach falls short. This test still reads more like its primary goal is to automate the browser, not to automate the underlying system. Plus, the reader of this test needs to understand generics in order to fully grasp what the test is doing.

Static Page Classes

An alternative approach I’ve used in the past is to create my own static classes to represent the pages in my web site. It looks like this:

public void LoginFailsWithUnknownUser4()
    using (var browser = new IE(url))
        LoginPage.UserName(browser).Value = "unknown";
        LoginPage.Password(browser).Value = "password";

        bool loginSucceeded = IndexPage.IsCurrentPage(browser);

        Assert.That(loginSucceeded == false);
public static class LoginPage
    public static TextField UserName(Browser browser)
        return browser.TextField(Find.ById(new Regex("UserName")));
    public static TextField Password(Browser browser)
        return browser.TextField(Find.ById(new Regex("Password")));
    public static Button LoginButton(Browser browser)
        return browser.Button(Find.ById(new Regex("LoginButton")));

This is the closest I have come to revealing the intention behind the functional test without clouding it in all the arcane incantations necessary to animate a web browser. Yes, there are still references to the browser. But, at least now the intention behind the test can be inferred by reading each line from left to right. Furthermore, most of the references to the browser are now parenthetical, which our eyes are accustomed to skipping.

What do you think?

I’d like to know what you think. Are your functional tests as clear as they could be? If so, how’d you do it? If not, do you think this approach might be helpful? Drop me a line!

My Marick Test Matrix

I introduced the Marick Test Matrix in the previous post. As much as I like it and have come to rely on it. One thing has always bugged me about the matrix: The terms Brian used to describe his horizontal axis seem, IMHO, obtuse. Instead, I prefer these simpler terms:

  • Support Programming = Define Requirements
  • Critique Product = Identify Defects

Given those changes, here’s my updated matrix:

I prefer these axes because now I can refer to each quadrant by a simple name:

  • Business/Requirements
  • Business/Defects
  • Technology/Requirements
  • Technology/Defects

The Marick Test Matrix

One of the leading voices in agile testing is a guy named Brian Marick. Brian is an independent consultant and an author. I find his blog to be an invaluable resource.

The Marick Test Matrix

Back in 2003, Brian published an influential series of articles on agile testing. He was attempting to point the way forward for agile testers. But, in the process, he came up with an elegant method of cataloguing testing methods that has become known as the “Marick Test Matrix.”

I’d like to introduce the matrix here in the hopes of fostering a discussion about what we test and how we test it.

Brian’s work categorized tests by asking two questions:

  1. Is the test business facing or technology facing?
  2. Does the test support programming or critique a product?

When you combine the two questions (or axes), you get a grid (or matrix), like this one:

But, what the heck do these things mean? That’s what the remainder of this post is about…

Business Facing Tests

A business facing test is one that is expressed in terms that are well understood by a business expert. For example:

  • If you withdraw more money than you have in your account, the system should automatically extend you a loan for the difference. (Notice that the italicized words are business terms.)

Business facing tests are best authored by people who understand the business (e.g. product owner, business analyst, etc.).

Technology Facing Tests

A technology facing test is one that is expressed in terms that are well understood by a technology expert. For example:

  • Different browsers implement Javascript differently, so we test whether our product works with the most important ones. (Notice that the italicized words are technology terms.)

Technology facing tests are best authored by people who understand the technology (e.g. developer, tester, etc.).

Tests that Support Programming

A test that supports programming is one that defines what the software should do. For example:

  • Clicking on the Account Details link should take the user to the Account Details screen.
  • Calling the Add method with 2 and 2 should return 4.

These tests may be written before the software exists. These tests are often automated and executed after a change is made to the software to ensure that the software still works as it should (i.e. regression). Once one of these tests passes, it should never be allowed to fail again.

Tests that Critique a Product

A test that critiques a product is one that tries to identify problems in completed software. In other words, this is the class of tests where the tester is actively trying to break the software in order to find bugs. For example:

  • When I logged on as Joe, I saw Tom’s data.
  • When I clicked the blue button after clicking the red button 400 times, the system threw an error.
  • When I configured the load testing tool to send 1,000 simultaneous users to the site, average response times increased to over 10 seconds.

In general, these tests are not automated – until a problem is identified, at which point an test that clearly reproduces the problem can be added to the tests that support programming.

So what?

Let’s take a look at where some standard types of testing might go on the matrix:

Unit Tests

Unit tests are used by developers to ensure that the code they are writing does what they expect it to. In essence, these tests form a specification for a single unit of code. By that definition, unit tests are tests that “support programming.” These tests are (or should be) very close to the code under test, which makes them “technology facing.” So, unit tests belong in the lower left quadrant of the matrix. The benefit of automating these tests is very high.

Functional Tests

Functional tests are used by development teams to ensure that the software they are writing does what they expect it to do. In essence, these tests form a specification for an entire system. That means that these are tests that “support programming.” But, functional tests are (or should be) written in a way that business users understand, making them “business facing.” So, functional tests belong in the upper left quadrant of the matrix. The benefit of automating these tests is high.

Exploratory Tests

Exploratory testing is the practice of trying to identify problems in an application. (Microsoft refers to this practice as a “Bug Bash” where many people are invited to use the software and prizes are given out to the person who identifies the most/worst bugs.) By definition, this makes these tests that “critique a product.” Exploratory tests are considered “business facing” due to the fact that the testers are using the software the same way a real end-user might. So, exploratory tests belong in the upper, right quadrant of the matrix. There is no benefit to automating exploratory tests – until a defect is identified. At that point, a new functional test can be added to ensure that the defect is resolved.

Performance Tests

Performance tests are used to determine the speed of an application within a specific set of parameters. Specialized tools are used to perform this testing. As such, these tests require a good deal of technical knowledge and are therefore “technology facing.” Performance tests require working software, and are therefore tests that “critique a product.” So, performance tests belong in the lower, right quadrant of the matrix.

Putting it all together

My blog editor now tells me that I’m approaching 1000 words. So, to prove the axiom, here’s the best summary I can think of:

That’s enough for one day. In a subsequent post, I’ll dive into why it’s important to cover all four quadrants.

For more information, check out Brian Marick’s original Agile Testing posts, or Google Marick Test Matrix.