imedo Development Blog

In the 0.3.9x releases of Cucumber the output generation has been changed in a way which prevents our output formatter from marking scenarios with undefined steps as pending.

Here’s the updated code: imedo_ci_formatter.rb

Popularity: 1% [?]

Not long ago we presented our output formatter for Cucumber (see Cucumber output formatter for continuous integration ). Meanwhile Cucumber 0.2 and 0.3 have been released and the API for output formatters has been changed, so that it was necessary to upgrade our formatter for feature tests on continuous-integration systems.

Here is the code. For information on using it see Cucumber output formatter for continuous integration or the Cucumber wiki.

The code:imedo_ci_formatter.rb

Popularity: 1% [?]

After presenting a way of analyzing the features of a web application and determining the to-be value for feature tests (see model-driven feature testing) here is a little piece of code that may help you with the controlling of your feature tests.

We are using Cucumber to run our feature tests and in order to have up-to-date test results for our latest code we created a task in our continuous integration system which runs all feature tests once a day.

The only problem was that the output formatters included in Cucumber do not give much information about the results of the scenarios. But these are the numbers we are interested in since scenarios are the actual feature tests. Hence we wrote our own output formatter. It evaluates the executed scenarios and not the steps and it dismisses the fancy colors since it is meant to be used with a continuous integration sytem which only knows black and white.

To use our output formatter save it in features/support/ and run it by adding the format option to the Cucumber command line, e.g.:

CUCUMBER_OPTS="--format ImedoCiFormatter" cucumber features/

Below is the source code for our Cucumber output formatter. It first initializes arrays for error messages, passed scenarios, failed scenarios, and pending scenarios as well as flags to mark scenarios as failed or pending. When executing a feature the name of this feature is printed out. Afterwards all of its scenarios are executed. Therefore each scenario is checked whether it contains steps when it is started. If not, the scenario is marked as pending. If it does contain steps, they are executed.

In case one step failed the current scenario is marked as failed and the error message ist stored. If one step is pending the whole scenario is marked as pending, too.
At the end of each scenario the result is evaluated and printed out.

After all feature tests have been executed the pending scenarios, error messages, and stats of feature tests are dumped in a summary.

The code: imedo_ci_formatter.rb

Popularity: 1% [?]

Last year Bryan Helmkamp gave an interesting presentation on story-driven development at the Gotham Ruby Conference. In this presentation he showed a graph called the “web app testing pyramid” which displays the optimal proportion of different test types for web applications. The base of the pyramid contains fast low level test which can test small pieces of code like single algorithms while the pyramid’s peak represents the slowest tests which not only need a server with the running application but also a client with at least one started web browser.

The following graph shows the version of the pyramid we are using. We are not speaking of scenario tests like Bryan Helmkamp but of feature tests which are technically the same, though they are not written by developers for verifying code but by software testers to verify the functionality of features (see model-driven feature testing)

The fastest test type are unit tests. They are best suited for test-driven development (TDD) since they can easily be written by developers for verifying their code and for regression tests. Unit test should be used to test all parts of the code which don’t need additional resources like a database or plugins. Therefore unit tests should outnumber every other kind of tests.

All parts of the code that cannot be covered by unit tests for different reasons need to be tested by functional tests. These kind of tests allows to load extra resources like a database. They also can and should be used for TDD to verify that code is actually working.

Story-based tests – like Cucumber stories combined with Webrat – are really slow, hence they are not well suited to replace unit tests or functional tests but are a good supplement to your test portfolio since they can test on a higher level. They are furthermore the proper choice for feature tests which are supposed to verify all use cases of features by testing selected scenarios (see model-driven feature testing). The number of these scenarios should be lower than the number of functional tests since a lot more usage scenarios are possible than tested by feature tests.

Nowadays web applications without Javascript (JS) or Cascading Style Sheets (CSS) are almost died out. But the overhead of testing JS and CSS is huge and it’s highly browser-dependent. Selenium grid is probably the most widely used solution for running automated browser tests. It needs a complex infrastructure with different operating systems and browsers. All tests are run through the browsers which makes them by far the slowest type of tests – only manual tests are slower. Therefore browser tests should be used carefully, only scenarios in which JS or CSS functionality needs to be verified should be tested this way. But don’t use these kind of tests to write full coverage tests for your JS, there are unit test frameworks for JS, too.

Keeping the pyramid of web application testing in your mind should help you to optimize the run time of your tests by not loosing test coverage.

Popularity: 1% [?]

While unit tests, functional tests, and integration tests are supposed to test source code, feature tests are supposed to test functionalities. Every test case simulates an interaction of a user with the application. But nobody can predict interactions of a user with a system. So feature tests are usually written to the best of the author’s knowledge. At least that was the way we wrote feature tests in the beginning. The result was a really heterogeneous test coverage of our features. Some features are over tested and some are not tested at all. One may say that there is no such thing like “over tested”, but compared to unit tests it takes forever to run feature tests. Therefore its number should be limited to a minimum—-the source code needs to be tested by unit, functional, and integration tests anyway. So unlike source-code testing where the goal is to test as many lines of code as possible, feature testing is about testing as few cases as necessary. Our solution for determining the minimum number of needed test cases uses model-driven architecture. So we call it model-driven feature testing. It uses an UML model of our application to define the necessary feature tests.

In order to be able to define a feature test we need four essential elements: the stakeholder (the user), the feature itself, a use case, and a scenario. We are using Cucumber stories to define our feature tests, so the end result looks like the following example:

Feature: Search
  As a guest
  I want to search for a keyword
  So that I can find documents containing this keyword

  Scenario: Search for keyword

In the following section we describe how to build up an UML model which can be used to define feature tests. Therefore we are using a simple web forum application as example.

Stakeholder analysis

The first part of creating a model of our application was to identify the stakeholders that need to interact with the application. For a web application like our simple web forum these are usually roles like guest, logged in user, and administrator. The next step is to add them into the UML model. This looks somewhat like in the following figure.

Feature analysis

Now that we have identified the actors we need to examine the functionalities or features of the application which means to categorize and list them hierarchically.

The best suited UML element to represent a feature is the component. An UML2 component diagram for a typical simple web forum with the features forum, forum topic, forum post, and a website search could look like in the figure below.

Use case analysis

After identifying the stakeholders and features, we need to survey the existing use case between the stakeholders and each feature. Using the features of the example web forum with the stakeholders from the stakeholder analysis, the use cases are the following.

Guest reads a forum post
Logged in user creates a forum topic
Logged in user writes a forum post
Administrator creates a forum
Administrator deletes a forum topic
Administrator edits a forum topic
Administrator deletes a forum post
Administrator edits a forum post

These use cases can be displayed in an UML2 use case diagram which would look like in the following figure.

Scenario identification

Having found out the stakeholders, features, and use cases we now know what users can do with our software, but in order to define test cases we need actual usage scenarios. The identification of these scenarios is the last step and will finish our UML model. Basically an unknown number of scenarios exists for each use case and at least one. But since feature tests are not supposed to test for edge cases like integer overflows and should just verify functionalities, one to three scenarios are usually sufficient for one use case.

For visualizing feature test scenarios we use UML2 sequence diagrams like the one below for a logged in user writing a forum post.

This diagram can be easily translated into a Cucumber story scenario. But we simplified a little bit, so that not every message in the sequence diagram is represented in the story.

Scenario: Write forum post
  Given a logged in user named "Alice"
  And a forum "Nice forum"
  And a forum topic "Testing" in the forum "Nice forum" 

  When Alice visits the forum topic "Testing" in forum "Nice forum"
  And Alice writes the forum post "I test model-driven." 

  Then Alice should see Alice's forum post "I test model-driven."

Some UML tools offer the possibility to generate code fragments, so we are thinking about generating Cucumber stories automatically from the UML model in the furture. But by now we are still writing the stories by hand.

After writing down all defined scenarios with Cucumber stories we are now able to see what percentage of our functionalities is tested, since we not only have the number of successful scenarios from the test results, but a should-be number from the UML model.

Popularity: 1% [?]