The Marick Test Matrix

One of the leading voices in agile testing is a guy named Brian Marick. Brian is an independent consultant and an author. I find his blog to be an invaluable resource.

The Marick Test Matrix

Back in 2003, Brian published an influential series of articles on agile testing. He was attempting to point the way forward for agile testers. But, in the process, he came up with an elegant method of cataloguing testing methods that has become known as the “Marick Test Matrix.”

I’d like to introduce the matrix here in the hopes of fostering a discussion about what we test and how we test it.

Brian’s work categorized tests by asking two questions:

  1. Is the test business facing or technology facing?
  2. Does the test support programming or critique a product?

When you combine the two questions (or axes), you get a grid (or matrix), like this one:

But, what the heck do these things mean? That’s what the remainder of this post is about…

Business Facing Tests

A business facing test is one that is expressed in terms that are well understood by a business expert. For example:

  • If you withdraw more money than you have in your account, the system should automatically extend you a loan for the difference. (Notice that the italicized words are business terms.)

Business facing tests are best authored by people who understand the business (e.g. product owner, business analyst, etc.).

Technology Facing Tests

A technology facing test is one that is expressed in terms that are well understood by a technology expert. For example:

  • Different browsers implement Javascript differently, so we test whether our product works with the most important ones. (Notice that the italicized words are technology terms.)

Technology facing tests are best authored by people who understand the technology (e.g. developer, tester, etc.).

Tests that Support Programming

A test that supports programming is one that defines what the software should do. For example:

  • Clicking on the Account Details link should take the user to the Account Details screen.
  • Calling the Add method with 2 and 2 should return 4.

These tests may be written before the software exists. These tests are often automated and executed after a change is made to the software to ensure that the software still works as it should (i.e. regression). Once one of these tests passes, it should never be allowed to fail again.

Tests that Critique a Product

A test that critiques a product is one that tries to identify problems in completed software. In other words, this is the class of tests where the tester is actively trying to break the software in order to find bugs. For example:

  • When I logged on as Joe, I saw Tom’s data.
  • When I clicked the blue button after clicking the red button 400 times, the system threw an error.
  • When I configured the load testing tool to send 1,000 simultaneous users to the site, average response times increased to over 10 seconds.

In general, these tests are not automated – until a problem is identified, at which point an test that clearly reproduces the problem can be added to the tests that support programming.

So what?

Let’s take a look at where some standard types of testing might go on the matrix:

Unit Tests

Unit tests are used by developers to ensure that the code they are writing does what they expect it to. In essence, these tests form a specification for a single unit of code. By that definition, unit tests are tests that “support programming.” These tests are (or should be) very close to the code under test, which makes them “technology facing.” So, unit tests belong in the lower left quadrant of the matrix. The benefit of automating these tests is very high.

Functional Tests

Functional tests are used by development teams to ensure that the software they are writing does what they expect it to do. In essence, these tests form a specification for an entire system. That means that these are tests that “support programming.” But, functional tests are (or should be) written in a way that business users understand, making them “business facing.” So, functional tests belong in the upper left quadrant of the matrix. The benefit of automating these tests is high.

Exploratory Tests

Exploratory testing is the practice of trying to identify problems in an application. (Microsoft refers to this practice as a “Bug Bash” where many people are invited to use the software and prizes are given out to the person who identifies the most/worst bugs.) By definition, this makes these tests that “critique a product.” Exploratory tests are considered “business facing” due to the fact that the testers are using the software the same way a real end-user might. So, exploratory tests belong in the upper, right quadrant of the matrix. There is no benefit to automating exploratory tests – until a defect is identified. At that point, a new functional test can be added to ensure that the defect is resolved.

Performance Tests

Performance tests are used to determine the speed of an application within a specific set of parameters. Specialized tools are used to perform this testing. As such, these tests require a good deal of technical knowledge and are therefore “technology facing.” Performance tests require working software, and are therefore tests that “critique a product.” So, performance tests belong in the lower, right quadrant of the matrix.

Putting it all together

My blog editor now tells me that I’m approaching 1000 words. So, to prove the axiom, here’s the best summary I can think of:

That’s enough for one day. In a subsequent post, I’ll dive into why it’s important to cover all four quadrants.

For more information, check out Brian Marick’s original Agile Testing posts, or Google Marick Test Matrix.

Stand Up!

If you have experience with Scrum or other agile software development methodologies, you are likely familiar with the concept of “stand-up” meetings – the short, daily status meetings where each team member answers three questions:

  • What did I do yesterday?
  • What do I plan to do today?
  • What is the way?

To keep the meetings short, discussion is deferred until everyone has answered the three questions. To encourage short meetings, people often stand – thus, the name. The point of these meetings is to keep the team and any interested outsiders informed of what is happening on the project.

But, you may not be aware of some additional things that help teams keep their stand-up meetings effective and productive:

Only team members speak

To keep the meeting focused, only team members are allowed to speak. A team member is defined as someone who is pulling tasks from the current Sprint backlog. Interested parties, including the Scrum Master, Product Owner, executives and others, are encouraged to attend the meeting to receive status updates. But, they are discouraged from speaking until after the stand-up meeting ends.

Many teams host a short “collaboration” session immediately following the stand-up. This is the appropriate time for interested parties to make comments, ask questions, etc.

Speak to the backlog

When answering the three questions, team members should only refer to work that directly affects the Sprint backlog. Your teammates need to know where the project stands, not how many meetings you attended.

Here’s an example of how NOT to answer the three questions:

  • Yesterday, I went to some meetings and then had lunch with John. I wrote some code in the afternoon, but I had to leave early for a doctor appointment.
  • Today, I have some more meetings.
  • The only thing in my way is that I need to take Fluffy to the vet.

Here’s an example of the correct way to answer the three questions:

  • Yesterday, I worked on Story 412. I completed the database and web services tasks. And, I started on the UI task.
  • Today, I plan to finish the UI task then start working on tasks for the next available story. Right now, it looks like that might be Story 416.
  • There is nothing preventing me from finishing Story 412. But, I have some questions about how to implement Story 416. Can we talk about it after stand-up?

For more on the subject of running effective, productive stand-up meetings, take a look at It’s Not Just Standing Up: Patterns of Daily Stand-up Meetings from Martin Fowler. (For those who don’t know: Martin Fowler is the Chief Architect at ThoughtWorks and the author of several highly respected books, including UML Distilled, Refactoring, and Patterns of Enterprise Application Architecture.)

Using NUnit to run 32-bit tests on a 64-bit platform

Esoteric? Yes. But, these are the problems I run into…

I recently installed the 64-bit version of Windows Vista in order to gain access to all 4GB of memory on my computer. (32-bit versions of Windows can only access about 3.5GB.) But, when I tried to run some NUnit tests, I started running into problems.

First, my tests use the Microsoft Jet driver to access a Microsoft Excel spreadsheet. Unfortunately, there is no 64-bit version of the Jet driver. So, when I tried to execute my tests, they failed to load the driver.

64-bit Vista does come with a 32-bit version of the Jet driver. In order to access the 32-bit driver, your code must be 32-bit. So, I told Visual Studio to use the x86 configuration rather than the “Any CPU” configuration.

But, once I recompiled, NUnit refused to load my 32-bit DLL. It claimed that it could not load the DLL or one of its references:

Turns out that NUnit was also compiled with the “Any CPU” option. So, when it launches on my 64-bit computer, it runs as a 64-bit app. This means that it cannot load 32-bit DLLs. Thus, the error.

The solution was to use a utility called CORFLAGS to add a 32-bit flag to NUnit.exe. Here’s the command I used:

corflags nunit.exe /32BIT+

Here’s a link to the article I found explaining this: http://geekswithblogs.net/Lance/archive/2006/12/28/102191.aspx.

Track Active Item in Solution Explorer

Here's another one of those tips that I'm blogging just so that I won't forget it.

Under Tools/Options, in Projects and Solutions, Visual Studio 2005 and 2008 both provide an option entitled "Track Active Item in Solution Explorer". This option causes Visual Studio to select the active item (current open file) in Solution Explorer.

I find this an extremely useful option. It helps me maintain context when I'm jumping from one open file to another while working on a complex problem. It puzzles me as to why the Visual Studio team decided that this option should be off by default. Oh, well. Now we both know how to use it.

Resetting Keyboard Shortcuts in Visual Studio

Visual Studio just went schizophrenic on me: F5 started running File/Open/File, rather than Debug/Start Debugging; F9 refused to toggle breakpoints; and, none of the other default keyboard shortcuts appeared to be working normally. It took me a while to find the resolution to the problem. So, I'm blogging it for posterity.

In a Visual Studio Command Prompt, enter the following commands:

devenv /resetuserdata

This command clears out your recent projects list, dumps your RSS feed, and more. It will execute for several seconds (perhaps even a minute), then return control to the command prompt. (Interestingly, the /resetuserdata switch does not appear to be documented – either in the command line help, or on MSDN. I found it on a blog.)

devenv /resetsettings

This command starts Visual Studio. Go ahead and close Visual Studio.

devenv

This command starts Visual Studio again, and allows you to choose your development environment defaults.

These commands did the trick for me in Visual Studio 2008. I believe they would also work with Visual Studio 2005, but I have not confirmed this.