imedo Development Blog

One of the biggest problems in software development has nothing to do with coding. It is about communication, the “code monkey versus software artist problem”.

Business people tend to specify as much as possible, but often lack the necessary knowledge of the technical details. From their point of view software developers are code monkeys who are supposed to implement specifications. On the other side programmers often see themselves as software artists who need total freedom since they are the ones who know the code.

The truth lies – like in most cases – somewhere in between. Feature requests are a result of business decisions which are made in the e.g. marketing or sales departments and usually need to be realized as soon as possible. But only software developers can estimate the needed resources to implement these features. They know which particular tasks have to be done, since they know the code.

One process which currently seems to be the solution for bringing business and software development together is agile software development with Scrum. It uses the rules of extreme programming (XP) and is therefore lightweight with low specification overhead and short release cycles. But still some specification needs to be done. Although the user stories of XP are really short they can contain quite a bit on information if they follow certain rules. Here are six rules which can help to get more information from one story. These rules do not conflict with the advices for writing stories in XP, but can be seen as a more practical approach which is based on them.

1 One use case per story

The first rule is to limit one story to one use case, i.e. only one use case for one feature is added or altered. For example if a new feature called “simple web forum” with the following use cases needs to be implemented.

• Guest views forum entry
• User writes forum entry
• User edits forum entry
• Moderator deletes forum entry

Each of these use case should be developed as one single story.

2 Consistent story format

Rule number two is to use a consistent format for the stories. The feature description part of the language Gherkin used by Cucumber is a good way of keeping stories short, but readable. In the following example is shown how it can be used to describe a use case.

Use Case: User writes forum entry
  In order to produce user-generated content for one topic
  As a logged in user
  I want to write a new forum entry

One use case description consists of four lines. The first line is a short summary of the use case. Line number two, the “in order to” line, is reserved for a brief business value explanation. And the last two lines are supposed to describe the uses case itself: Who needs to do what.

3 Cover all wanted scenarios with a story

Rule number three is about the used semantic. A story should always be formulated as generic as possible so that it covers all wanted scenarios. This is because giving only a specific scenario like in the following bad example means that only this scenario should be implemented.

Bad example:

Use Case: Little Red Riding Hood searches for "Wolf"
  In order to see how the Big Bad Wolf looks like
  As Little Red Riding Hood
  I want to search for "Wolf"

Good example:

Use Case: Guest searches for keyword
  In order to get information about a keyword
  As a guest
  I want to search for a keyword

4 Use of scenarios to describe use case

Sometimes the expected behavior for a scenario of a use case is different than for others, e.g. if access control is an issue. In this case the possible scenarios should be described in the story as well. Here, the scenario syntax of Gherkin can be used.

Use Case: User searches for content
  In order to get information about a keyword
  As a user
  I want to search for a keyword

  Scenario: User searches for private content
    Given a user Big Bad Wolf
    And a user named Little Red Riding Hood
    And Big Bad Wolf's private document with information about the Big Bad Wolf
    When Little Red Riding Hood searches for "Wolf"
    Then Little Red Riding Hood should not see the document about the Big Bad Wolf

5 Roles need to be known to the system

The fifth rule is that a role in a use case description of a story should always be known to the system, otherwise it is supposed to be added to the system. Like in the examples of rule 3 and 4 the Little Red Riding Hood might be a user name, but is probably no defined role. But e.g. the role “guest” stands for an unregistered visitor.

6 Use acceptance criteria for additional specification

In order to define additional things for one story it should be possible to give acceptance criteria for the QA team. This is rule number six. Acceptance criteria should be short and in list form. For the example with the keyword search one could be the following.

- A search form should be on every page

Popularity: 1% [?]

In the 0.3.9x releases of Cucumber the output generation has been changed in a way which prevents our output formatter from marking scenarios with undefined steps as pending.

Here’s the updated code: imedo_ci_formatter.rb

Popularity: 1% [?]

Not long ago we presented our output formatter for Cucumber (see Cucumber output formatter for continuous integration ). Meanwhile Cucumber 0.2 and 0.3 have been released and the API for output formatters has been changed, so that it was necessary to upgrade our formatter for feature tests on continuous-integration systems.

Here is the code. For information on using it see Cucumber output formatter for continuous integration or the Cucumber wiki.

The code:imedo_ci_formatter.rb

Popularity: 1% [?]

After presenting a way of analyzing the features of a web application and determining the to-be value for feature tests (see model-driven feature testing) here is a little piece of code that may help you with the controlling of your feature tests.

We are using Cucumber to run our feature tests and in order to have up-to-date test results for our latest code we created a task in our continuous integration system which runs all feature tests once a day.

The only problem was that the output formatters included in Cucumber do not give much information about the results of the scenarios. But these are the numbers we are interested in since scenarios are the actual feature tests. Hence we wrote our own output formatter. It evaluates the executed scenarios and not the steps and it dismisses the fancy colors since it is meant to be used with a continuous integration sytem which only knows black and white.

To use our output formatter save it in features/support/ and run it by adding the format option to the Cucumber command line, e.g.:

CUCUMBER_OPTS="--format ImedoCiFormatter" cucumber features/

Below is the source code for our Cucumber output formatter. It first initializes arrays for error messages, passed scenarios, failed scenarios, and pending scenarios as well as flags to mark scenarios as failed or pending. When executing a feature the name of this feature is printed out. Afterwards all of its scenarios are executed. Therefore each scenario is checked whether it contains steps when it is started. If not, the scenario is marked as pending. If it does contain steps, they are executed.

In case one step failed the current scenario is marked as failed and the error message ist stored. If one step is pending the whole scenario is marked as pending, too.
At the end of each scenario the result is evaluated and printed out.

After all feature tests have been executed the pending scenarios, error messages, and stats of feature tests are dumped in a summary.

The code: imedo_ci_formatter.rb

Popularity: 1% [?]

Last year Bryan Helmkamp gave an interesting presentation on story-driven development at the Gotham Ruby Conference. In this presentation he showed a graph called the “web app testing pyramid” which displays the optimal proportion of different test types for web applications. The base of the pyramid contains fast low level test which can test small pieces of code like single algorithms while the pyramid’s peak represents the slowest tests which not only need a server with the running application but also a client with at least one started web browser.

The following graph shows the version of the pyramid we are using. We are not speaking of scenario tests like Bryan Helmkamp but of feature tests which are technically the same, though they are not written by developers for verifying code but by software testers to verify the functionality of features (see model-driven feature testing)

The fastest test type are unit tests. They are best suited for test-driven development (TDD) since they can easily be written by developers for verifying their code and for regression tests. Unit test should be used to test all parts of the code which don’t need additional resources like a database or plugins. Therefore unit tests should outnumber every other kind of tests.

All parts of the code that cannot be covered by unit tests for different reasons need to be tested by functional tests. These kind of tests allows to load extra resources like a database. They also can and should be used for TDD to verify that code is actually working.

Story-based tests – like Cucumber stories combined with Webrat – are really slow, hence they are not well suited to replace unit tests or functional tests but are a good supplement to your test portfolio since they can test on a higher level. They are furthermore the proper choice for feature tests which are supposed to verify all use cases of features by testing selected scenarios (see model-driven feature testing). The number of these scenarios should be lower than the number of functional tests since a lot more usage scenarios are possible than tested by feature tests.

Nowadays web applications without Javascript (JS) or Cascading Style Sheets (CSS) are almost died out. But the overhead of testing JS and CSS is huge and it’s highly browser-dependent. Selenium grid is probably the most widely used solution for running automated browser tests. It needs a complex infrastructure with different operating systems and browsers. All tests are run through the browsers which makes them by far the slowest type of tests – only manual tests are slower. Therefore browser tests should be used carefully, only scenarios in which JS or CSS functionality needs to be verified should be tested this way. But don’t use these kind of tests to write full coverage tests for your JS, there are unit test frameworks for JS, too.

Keeping the pyramid of web application testing in your mind should help you to optimize the run time of your tests by not loosing test coverage.

Popularity: 1% [?]

While unit tests, functional tests, and integration tests are supposed to test source code, feature tests are supposed to test functionalities. Every test case simulates an interaction of a user with the application. But nobody can predict interactions of a user with a system. So feature tests are usually written to the best of the author’s knowledge. At least that was the way we wrote feature tests in the beginning. The result was a really heterogeneous test coverage of our features. Some features are over tested and some are not tested at all. One may say that there is no such thing like “over tested”, but compared to unit tests it takes forever to run feature tests. Therefore its number should be limited to a minimum—-the source code needs to be tested by unit, functional, and integration tests anyway. So unlike source-code testing where the goal is to test as many lines of code as possible, feature testing is about testing as few cases as necessary. Our solution for determining the minimum number of needed test cases uses model-driven architecture. So we call it model-driven feature testing. It uses an UML model of our application to define the necessary feature tests.

In order to be able to define a feature test we need four essential elements: the stakeholder (the user), the feature itself, a use case, and a scenario. We are using Cucumber stories to define our feature tests, so the end result looks like the following example:

Feature: Search
  As a guest
  I want to search for a keyword
  So that I can find documents containing this keyword

  Scenario: Search for keyword

In the following section we describe how to build up an UML model which can be used to define feature tests. Therefore we are using a simple web forum application as example.

Stakeholder analysis

The first part of creating a model of our application was to identify the stakeholders that need to interact with the application. For a web application like our simple web forum these are usually roles like guest, logged in user, and administrator. The next step is to add them into the UML model. This looks somewhat like in the following figure.

Feature analysis

Now that we have identified the actors we need to examine the functionalities or features of the application which means to categorize and list them hierarchically.

The best suited UML element to represent a feature is the component. An UML2 component diagram for a typical simple web forum with the features forum, forum topic, forum post, and a website search could look like in the figure below.

Use case analysis

After identifying the stakeholders and features, we need to survey the existing use case between the stakeholders and each feature. Using the features of the example web forum with the stakeholders from the stakeholder analysis, the use cases are the following.

Guest reads a forum post
Logged in user creates a forum topic
Logged in user writes a forum post
Administrator creates a forum
Administrator deletes a forum topic
Administrator edits a forum topic
Administrator deletes a forum post
Administrator edits a forum post

These use cases can be displayed in an UML2 use case diagram which would look like in the following figure.

Scenario identification

Having found out the stakeholders, features, and use cases we now know what users can do with our software, but in order to define test cases we need actual usage scenarios. The identification of these scenarios is the last step and will finish our UML model. Basically an unknown number of scenarios exists for each use case and at least one. But since feature tests are not supposed to test for edge cases like integer overflows and should just verify functionalities, one to three scenarios are usually sufficient for one use case.

For visualizing feature test scenarios we use UML2 sequence diagrams like the one below for a logged in user writing a forum post.

This diagram can be easily translated into a Cucumber story scenario. But we simplified a little bit, so that not every message in the sequence diagram is represented in the story.

Scenario: Write forum post
  Given a logged in user named "Alice"
  And a forum "Nice forum"
  And a forum topic "Testing" in the forum "Nice forum" 

  When Alice visits the forum topic "Testing" in forum "Nice forum"
  And Alice writes the forum post "I test model-driven." 

  Then Alice should see Alice's forum post "I test model-driven."

Some UML tools offer the possibility to generate code fragments, so we are thinking about generating Cucumber stories automatically from the UML model in the furture. But by now we are still writing the stories by hand.

After writing down all defined scenarios with Cucumber stories we are now able to see what percentage of our functionalities is tested, since we not only have the number of successful scenarios from the test results, but a should-be number from the UML model.

Popularity: 1% [?]