Coding Standards

I wish I could remember who told me this…

It does not matter which side of the road we drive on, as long as we all drive on the same side as everyone else.

My Marick Test Matrix

I introduced the Marick Test Matrix in the previous post. As much as I like it and have come to rely on it. One thing has always bugged me about the matrix: The terms Brian used to describe his horizontal axis seem, IMHO, obtuse. Instead, I prefer these simpler terms:

  • Support Programming = Define Requirements
  • Critique Product = Identify Defects

Given those changes, here’s my updated matrix:

I prefer these axes because now I can refer to each quadrant by a simple name:

  • Business/Requirements
  • Business/Defects
  • Technology/Requirements
  • Technology/Defects

The Marick Test Matrix

One of the leading voices in agile testing is a guy named Brian Marick. Brian is an independent consultant and an author. I find his blog to be an invaluable resource.

The Marick Test Matrix

Back in 2003, Brian published an influential series of articles on agile testing. He was attempting to point the way forward for agile testers. But, in the process, he came up with an elegant method of cataloguing testing methods that has become known as the “Marick Test Matrix.”

I’d like to introduce the matrix here in the hopes of fostering a discussion about what we test and how we test it.

Brian’s work categorized tests by asking two questions:

  1. Is the test business facing or technology facing?
  2. Does the test support programming or critique a product?

When you combine the two questions (or axes), you get a grid (or matrix), like this one:

But, what the heck do these things mean? That’s what the remainder of this post is about…

Business Facing Tests

A business facing test is one that is expressed in terms that are well understood by a business expert. For example:

  • If you withdraw more money than you have in your account, the system should automatically extend you a loan for the difference. (Notice that the italicized words are business terms.)

Business facing tests are best authored by people who understand the business (e.g. product owner, business analyst, etc.).

Technology Facing Tests

A technology facing test is one that is expressed in terms that are well understood by a technology expert. For example:

  • Different browsers implement Javascript differently, so we test whether our product works with the most important ones. (Notice that the italicized words are technology terms.)

Technology facing tests are best authored by people who understand the technology (e.g. developer, tester, etc.).

Tests that Support Programming

A test that supports programming is one that defines what the software should do. For example:

  • Clicking on the Account Details link should take the user to the Account Details screen.
  • Calling the Add method with 2 and 2 should return 4.

These tests may be written before the software exists. These tests are often automated and executed after a change is made to the software to ensure that the software still works as it should (i.e. regression). Once one of these tests passes, it should never be allowed to fail again.

Tests that Critique a Product

A test that critiques a product is one that tries to identify problems in completed software. In other words, this is the class of tests where the tester is actively trying to break the software in order to find bugs. For example:

  • When I logged on as Joe, I saw Tom’s data.
  • When I clicked the blue button after clicking the red button 400 times, the system threw an error.
  • When I configured the load testing tool to send 1,000 simultaneous users to the site, average response times increased to over 10 seconds.

In general, these tests are not automated – until a problem is identified, at which point an test that clearly reproduces the problem can be added to the tests that support programming.

So what?

Let’s take a look at where some standard types of testing might go on the matrix:

Unit Tests

Unit tests are used by developers to ensure that the code they are writing does what they expect it to. In essence, these tests form a specification for a single unit of code. By that definition, unit tests are tests that “support programming.” These tests are (or should be) very close to the code under test, which makes them “technology facing.” So, unit tests belong in the lower left quadrant of the matrix. The benefit of automating these tests is very high.

Functional Tests

Functional tests are used by development teams to ensure that the software they are writing does what they expect it to do. In essence, these tests form a specification for an entire system. That means that these are tests that “support programming.” But, functional tests are (or should be) written in a way that business users understand, making them “business facing.” So, functional tests belong in the upper left quadrant of the matrix. The benefit of automating these tests is high.

Exploratory Tests

Exploratory testing is the practice of trying to identify problems in an application. (Microsoft refers to this practice as a “Bug Bash” where many people are invited to use the software and prizes are given out to the person who identifies the most/worst bugs.) By definition, this makes these tests that “critique a product.” Exploratory tests are considered “business facing” due to the fact that the testers are using the software the same way a real end-user might. So, exploratory tests belong in the upper, right quadrant of the matrix. There is no benefit to automating exploratory tests – until a defect is identified. At that point, a new functional test can be added to ensure that the defect is resolved.

Performance Tests

Performance tests are used to determine the speed of an application within a specific set of parameters. Specialized tools are used to perform this testing. As such, these tests require a good deal of technical knowledge and are therefore “technology facing.” Performance tests require working software, and are therefore tests that “critique a product.” So, performance tests belong in the lower, right quadrant of the matrix.

Putting it all together

My blog editor now tells me that I’m approaching 1000 words. So, to prove the axiom, here’s the best summary I can think of:

That’s enough for one day. In a subsequent post, I’ll dive into why it’s important to cover all four quadrants.

For more information, check out Brian Marick’s original Agile Testing posts, or Google Marick Test Matrix.

Stand Up!

If you have experience with Scrum or other agile software development methodologies, you are likely familiar with the concept of “stand-up” meetings – the short, daily status meetings where each team member answers three questions:

  • What did I do yesterday?
  • What do I plan to do today?
  • What is the way?

To keep the meetings short, discussion is deferred until everyone has answered the three questions. To encourage short meetings, people often stand – thus, the name. The point of these meetings is to keep the team and any interested outsiders informed of what is happening on the project.

But, you may not be aware of some additional things that help teams keep their stand-up meetings effective and productive:

Only team members speak

To keep the meeting focused, only team members are allowed to speak. A team member is defined as someone who is pulling tasks from the current Sprint backlog. Interested parties, including the Scrum Master, Product Owner, executives and others, are encouraged to attend the meeting to receive status updates. But, they are discouraged from speaking until after the stand-up meeting ends.

Many teams host a short “collaboration” session immediately following the stand-up. This is the appropriate time for interested parties to make comments, ask questions, etc.

Speak to the backlog

When answering the three questions, team members should only refer to work that directly affects the Sprint backlog. Your teammates need to know where the project stands, not how many meetings you attended.

Here’s an example of how NOT to answer the three questions:

  • Yesterday, I went to some meetings and then had lunch with John. I wrote some code in the afternoon, but I had to leave early for a doctor appointment.
  • Today, I have some more meetings.
  • The only thing in my way is that I need to take Fluffy to the vet.

Here’s an example of the correct way to answer the three questions:

  • Yesterday, I worked on Story 412. I completed the database and web services tasks. And, I started on the UI task.
  • Today, I plan to finish the UI task then start working on tasks for the next available story. Right now, it looks like that might be Story 416.
  • There is nothing preventing me from finishing Story 412. But, I have some questions about how to implement Story 416. Can we talk about it after stand-up?

For more on the subject of running effective, productive stand-up meetings, take a look at It’s Not Just Standing Up: Patterns of Daily Stand-up Meetings from Martin Fowler. (For those who don’t know: Martin Fowler is the Chief Architect at ThoughtWorks and the author of several highly respected books, including UML Distilled, Refactoring, and Patterns of Enterprise Application Architecture.)