Archive for the ‘Testing’ Category
We started using git in late 2008 mainly not because it was cool but because it solved one particiular problem: Slow deployment.
To understand this you have to know that our codebase was rather big and monolithic back then. One big rails project with a lot of small files and lots of plugins. This meant we had about 3 times as much files to deploy (3 files per file in every directory in the .svn-directory) using capistrano which bascially does a checkout for a new deploy.
This process took about 45 minutes in the end (yes, 45 minutes!) which was not acceptable. We could have thought about other solutions but as there were other advantages to git we switched to git and the deployment process went back to about 2 minutes.
Over the last two years we tried different models and workflows and ended up with what many people consider the standard workflow for git:
- master is the master branch and main development branch, CI runs off master
- We create feature branches and local “try and experiment” branches but they normally live no longer than a few days and are constantly rebased to master.
- If all tests in our continuous integration environment succeed, the code will automatically be merged into a branch called “live“. We deploy only from “live” and can therefore deploy at any time knowing that all tests have been run on the code we are deploying
A few things we learned:
- never rebase remote branches – I think you can do it right somehow but if you don’t it will blow up your repository
- Changing history is ok – locally – never amend or rebase – a commit that has been published – or: don’t use git push -f
- git bisect is fantastic to find code breaking commits (running tests for each bisected commit) or in specific code changes (using grep <change>) e.g. “Which commit did remove of some word in copy on a website”
- git is fast – most of the time
- Developing a certain discipline regarding branching is very important. Branches are useful for certain things, but a lot of long lived, diverging branches is certainly a smell
- The ability to split and merge repositories without losing history is very useful (some information will get lost, when splitting and then remerging the same repo, but we never did that anyways)
- The git index/additional step required when commiting is rather annoying at first but no one wants to live without it after a while
- Local branches, commit –amend and rebase -i allow us to re-fit the commits into logical chunks which aren’t always apparent at first when developing a new feature. This is really useful if a commit is reviewed in the future
- Having all commits available on every developer machine is very useful for speed, offline-development and backup reasons
What could be better? What’s missing? We thought about that last week and didn’t come up with anything. Git seems to be a rather perfect SCM – at least for us at the moment.
Popularity: 2% [?]
Here is something for people concerned of their websites health. A new tool called TinyMon enables you to test your website from an users perspective unattended and continuously.
You can simply check if that website of yours is still online, or test more complex stuff like online shop transactions or sign up-processes that require clicking an activation link in an e-mail send to an applicants mailbox. If you have used Selenium, Webrat or Waitir before, then usage of TinyMon will be a breeze. If you didn’t, it is very easy and does not need any programming skills to use. A wee bit of HTML/CSS knowledge might be required for sites with duplicate content or very specific Health Checks.
The great thing about TinyMon is, compared to Selenium, Webrat&Co: You can get all of the functionality and continious checking power without setting up a Framework. Oh, did I mention that it is free…? Just head over to the TinyMon website, register and start making Health Checks.
Within the registered TinyMon user area, an account represents a cluster of websites belonging to the same client. It holds the access rights to sites and Health Checks within sites and can be made accessible in varying degree to other users. A user that is given access to an account can always see all sites and checks within this account.
It provides a base URL, so a sub domain is would a new Site, unless you provide a full Path in visit Steps for Health Checks for a Sites sub domain. The clean way however, is to treat sub domains like new a Site.
You can choose from a few predefined templates, or create you first step from Scratch. Picking a “Up-Check” Template will provide you with the two most important Step types of TinyMon. That is “Visit” and “Check content”. In the Template you have to provide a path for the visit step and a string for the check-content step. Path will be happy enough with a “/”, then it simply visits your Site’s base URL, while Check Content hopefully resembles some unique text from the body of your page, that is not used in http error pages.
Click “Information” and then “Run”. After a while refresh the page or click Check runs to see how your test worked out. Up-Checks are usually done within 10 Seconds, depending on TinyMon’s workload. If it passed, you can go back to “Information” and “Enable” your check, otherwise go back to “Edit Steps” and work out the Problem according to the error message listed in your Check Run.
Continuously test your websites sign up or log in mechanism, or test other complex scenarios unattended. If a condition is not met, TinyMon will try again two more times. If the error persists, it will grab the mark-up of the failing page (so you can inspect it with firebug from within TinyMon) and send out a notification email or prowl-notify you, editors and observers of the Sites’s account.
Popularity: 6% [?]
If you need a quick overview whether the internal links of your website are working and the markup is valid, checkout the Multipage Validator tool.
Popularity: 1% [?]
The the updated code can be found here: imedo_ci_formatter.rb
Popularity: 1% [?]
In the first part of this article it was shown how to use CSS selectors for matching elements on complex web pages. But selectors are not the only way of matching HTML elements, Webrat also supports matching via XPath.
XPath matchers can be combined with CSS-selector matchers. This is really useful if not, for example, the content of an element should be matched but the element itself like in the following example. Here a form is used to display data as default value in its input elements. This can be the case in web applications in which data should be edited easily without additional clicks.
<div id="content"> <form> Label 1: <input id="entry_1" name="entry" value="Entry 1"/> Label 2: <input id="entry_2" name="entry" value="Entry 2"/> <input type="submit"> </form> </div>
Matching the default values of input elements with Webrat can look like the following lines of code. It is a nested combination of CSS-selector and XPath matching in order to have a readable error message if the actual output differs from the expected.
response.body.should have_selector("#content") do |content| content.should have_tag("form") do |form| form.should have_selector("#entry_1") do |input| input.should have_xpath("@value") do |value| value.should contain("Entry 1") end end end end
XPath matchers are currently not support by Webrat’s within() method and therefore cannot be used to limit the area on a web page where an action should take place, e.g. clicking of a link. They can only be used for evaluating the response of an action.
Popularity: 2% [?]
Writing feature, integration, or acceptance tests with Webrat is a lot easier with simple web pages than it is with huge pages which provide much content. It can get really annoying to find the correct links or buttons to click. The same applies for matching content elements on complex pages in order to evaluate whether a test case was successful or not, since the same content can appear on different areas of one web page. Both issues can escalate if the position of layout elements is variable like in A/B usability tests or if a front-end layout manager is used. CSS selectors can help to find a layout element on a web page independent of its position. In this article is demonstrated how CSS selectors can be used with Webrat.
In order to present the usage of CSS selectors with Webrat the following example is used. It shows a layout element called “content” which contains a table with two rows and each row provides the same options for its entry: Edit and delete.
<div id="content"> <table> <thead> <tr><th>Name</th><th>Options</th></tr> </thead> <tbody> <tr id="entry_1" class="odd"> <td>Entry 1</td> <td><a href="">Edit</a> <a href="">Delete</a></td> </tr> <tr id="entry_2" class="even"> <td>Entry 2</td> <td><a href="">Edit</a> <a href="">Delete</a></td> </tr> </tbody> </table> </div>
In order to find the corresponding delete link for one particular entry in the example, it is necessary to define a scope in which the link can be found. With Webrat this can be achieved with the within() method.
within "#entry_2" do |scope| scope.click_link "Delete" end
Within() does only support the following selectors which have been defined in W3C CSS level 1.
- Type selectors, e.g. “table”
- Class selectors, e.g. “tr.odd”
- ID selectors, e.g. “#content”
The nesting of within() calls is currently not supported which means that well chosen IDs in the HTML code are necessary for good tests.
In order to evaluate the response of the example after clicking the delete link of the second entry, the following statement would be sufficient, but in case the test did not work as expected it would return an error message which includes the content of the whole web page.
response.body.should_not contain("Entry 2")
Narrowing the part of the HTML code to the CSS selector which should be evaluated helps to get a more readable error message, since it only includes the selected part. In opposite to the the within() method have_selector() and have_tag() can be used nested, too. If a selector, e.g. in case of an error, is not available on a web page, an error message for the current scope is generated. This could be the whole page if the selector was the first one. Therefore it is recommended to start with a selector which is available even if an error occurs.
response.body.should have_selector("#content") do |content| content.should have_tag("tbody") do |table| table.should_not contain("Entry 2") end end
Using selectors in order to match elements on a web page has the disadvantage of a bigger dependency to the HTML code. Changing the name of a CSS selector can break test case. But this disadvantage should be insignificant compared to the achievement of error messages in which the actual problem is easily identifiable.
Popularity: 8% [?]
Business people tend to specify as much as possible, but often lack the necessary knowledge of the technical details. From their point of view software developers are code monkeys who are supposed to implement specifications. On the other side programmers often see themselves as software artists who need total freedom since they are the ones who know the code.
The truth lies – like in most cases – somewhere in between. Feature requests are a result of business decisions which are made in the e.g. marketing or sales departments and usually need to be realized as soon as possible. But only software developers can estimate the needed resources to implement these features. They know which particular tasks have to be done, since they know the code.
One process which currently seems to be the solution for bringing business and software development together is agile software development with Scrum. It uses the rules of extreme programming (XP) and is therefore lightweight with low specification overhead and short release cycles. But still some specification needs to be done. Although the user stories of XP are really short they can contain quite a bit on information if they follow certain rules. Here are six rules which can help to get more information from one story. These rules do not conflict with the advices for writing stories in XP, but can be seen as a more practical approach which is based on them.
1 One use case per story
The first rule is to limit one story to one use case, i.e. only one use case for one feature is added or altered. For example if a new feature called “simple web forum” with the following use cases needs to be implemented.
- Guest views forum entry
- User writes forum entry
- User edits forum entry
- Moderator deletes forum entry
Each of these use case should be developed as one single story.
2 Consistent story format
Rule number two is to use a consistent format for the stories. The feature description part of the language Gherkin used by Cucumber is a good way of keeping stories short, but readable. In the following example is shown how it can be used to describe a use case.
Use Case: User writes forum entry In order to produce user-generated content for one topic As a logged in user I want to write a new forum entry
One use case description consists of four lines. The first line is a short summary of the use case. Line number two, the “in order to” line, is reserved for a brief business value explanation. And the last two lines are supposed to describe the uses case itself: Who needs to do what.
3 Cover all wanted scenarios with a story
Rule number three is about the used semantic. A story should always be formulated as generic as possible so that it covers all wanted scenarios. This is because giving only a specific scenario like in the following bad example means that only this scenario should be implemented.
Use Case: Little Red Riding Hood searches for "Wolf" In order to see how the Big Bad Wolf looks like As Little Red Riding Hood I want to search for "Wolf"
Use Case: Guest searches for keyword In order to get information about a keyword As a guest I want to search for a keyword
4 Use of scenarios to describe use case
Sometimes the expected behavior for a scenario of a use case is different than for others, e.g. if access control is an issue. In this case the possible scenarios should be described in the story as well. Here, the scenario syntax of Gherkin can be used.
Use Case: User searches for content In order to get information about a keyword As a user I want to search for a keyword Scenario: User searches for private content Given a user Big Bad Wolf And a user named Little Red Riding Hood And Big Bad Wolf's private document with information about the Big Bad Wolf When Little Red Riding Hood searches for "Wolf" Then Little Red Riding Hood should not see the document about the Big Bad Wolf
5 Roles need to be known to the system
The fifth rule is that a role in a use case description of a story should always be known to the system, otherwise it is supposed to be added to the system. Like in the examples of rule 3 and 4 the Little Red Riding Hood might be a user name, but is probably no defined role. But e.g. the role “guest” stands for an unregistered visitor.
6 Use acceptance criteria for additional specification
In order to define additional things for one story it should be possible to give acceptance criteria for the QA team. This is rule number six. Acceptance criteria should be short and in list form. For the example with the keyword search one could be the following.
- A search form should be on every page
Popularity: 4% [?]
Webrat supports file upload in web forms with the method attach_file. Due to a lack of multipart form support for integration tests in Action Pack 2.0.2, this does not work with Rails 2.0.2.
It has been added in version 2.1., but for those who are still using 2.0.2, with the following code you can upgrade the particular part of Action Pack to 2.1. Just add it to your tests so that it overwrites the loaded Action Pack code.
Here’s the updated code: imedo_ci_formatter.rb
Popularity: 1% [?]
Generating PDFs in a Rails application is a fairly common task. Maybe you want to create a letter, report, document or maybe an invoice. Either way the stuff that normally ends up in an PDF is important and you want to make sure the right stuff ends up there.
This pretty much sounds like a case for automated testing. But how do you test PDF content? One option would be to generate the PDF and then create a HTML out of the PDF using pdftohtml, parse the HTML and make some assertions. As you can guess, this approach isn’t very feasable, because the generated HTML isn’t very easy to parse.
Most of the time PDF generation in Rails applications is done using the RTex Plugin – the PDF is generated via LaTeX. This makes testing a lot easier because you can just parse and check the generated LaTeX-Source.
Everyone how has seen a LaTeX source file may ask: How the hell do I parse that?
In our case we added some “helper” comments like ”% SUM BEGIN” and ”% SUM END” before and after the part we were interested in and then used basic RegEx to parse out the interesting part. You have to manually check that the markup still looks as expected due to the newline handling of LaTeX (one is ok, two = new paragraph). Most of the time it is sufficient to look for ERB-Tags and use < %- instead of < %.
This approach works pretty well for us. One question which you should always keep in mind when you write tests is: What do I test on this level of testing and what do I leave out.
For the PDF/LaTeX-Testcase we choose to test the basic interaction between the objects that provide values for the PDF generation and the Template. We don’t test all combinations, just a few basic cases. Testing all or at least a lot of combinations, edge cases etc. is clearly a concern of unit tests.
Popularity: 1% [?]