Skip to content

Research gone good – Spec Explorer

February 27, 2011

The problem

Whatever testing technique you choose it does not guarantee that you actually cover what your app can do!

Take for example a mobile application – it’s a collection of screens and you can automate navigation using “coded ui” like technologies, but what if you forget the “back” functionality? Or what if requirements change, and you have to redo the navigation? And requirments will change. That’s about as certain as hidrogen and stupid people.

You will eventually need to change test code to cover the new flows. Then you run into 2 problems:

  1. Some flows are not obvious (specs are not maintained, you fell asleep in a scrum meeting and other chaotic pleasentries that happen close to the release date). If you forget to remove an old flow, you will get test failures. If you forget to add a new flow, you will not receive a single warning!
  2. How can you guarantee full coverage of your application ? (not code coverage, but covering all the possible flows )

Introducing Spec Explorer!

For quite some time, Microsoft has been investing in a research project called Spec Explorer. It is available as a free download here and it integrates with your VS.

Spec Explorer can be used to define a model as a finite state machine. Here’s a preview:

spec explorer generated model

Nicely drawn graph from a finite state machine model

The way it works is that you define the model using some special constructs – attributes and a special file. Then the tool generates a nice graph like that, which is the finite state machine of an the ideal project. Yup, you have to define your own model that does the same thing as the original, but this is far easier than it sounds. You don’t need a fully blown implementation, just a very simple one to teach Spec Explorer how to draw that graph. The graph is made from:

  • nodes – the state of the program (ex: how much money the user has in his account and if he is underage)
  • transitions – flow elements, user actions (ex: user buys a video, so the next state has updated money info, or the user cannot buy the video because it is marked adult and he is underage)

While you need to teach Spec Explorer all the business rules, it will generate all possible flows of you app!

Not counting the learning curve, it takes about 5-7 man days to program a model of an application that is reasonably well speced and that would require 8 man months of actual work. Plan for an extra 5 man days of learning, Spec Explorer is pretty abstract despite the nice drawing it does. But the latter is a fixed cost, make sure  you suffer it only once. And it is agile-friendly, you can develop parts of your model as features roll out, you don’t need the whole thing at once.

Model and Implementation

So, you have an ideal model, you can build that graph, what next? Here comes the beauty – Spec Explorer can extract all the flows as unit tests. In other words, it will can generate a number tests together cover ALL states and ALL transitions.

Hold on now, these are all based on the ideal model you build, so what’s the purpose? Well, the tests will not call into the ideal model but into the real app. Let’s take an example of a generated test, of an underage teen trying to buy an adult movie:

User teenUser = new User(UserType.Underage);

You are free to add anything you want in those methods. Of course, you should add navigation and asserts. Add them once, Spec Explorer will use them in hundreds of tests! And the beauty of it is that you mathematically cover all flows!

In my next blog post I’ll show how to create a model for a client application.

No comments yet

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: