Maintainable UI Tests with Selenium WebDriver and Ext JS: Test Wrappers

My current task is to support our teams in automatically testing a web application based on Ext JS. There is one team developing the main application while others develop plugins to this web application – with just the same requirement: Have a well tested deliverable after each two-weeks-sprint.

The problem: Have a maintainable and reusable test base for running the tests.

The solution: Provide Test-Wrappers for Ext JS components such as Buttons or Panels.

Ext JS is perfect for building Rich Internet Applications (short: RIA) such as an Editor for CMS contents. Based on JavaScript it provides a component structure with an object hierarchy. So a button and a panel are both childs of a base component which can be visible, hidden, enabled or disabled. But a panel is a container which can contain buttons and other components where a button again can be clicked.

What is more obvious than to provide a test base which relies on this object hierarchy? So have a TestButton class as wrapper to access the Ext JS button, click it, verify its label and so on.

Our test base, we call it test harness as you put it over the web application, is based on Java using Selenium WebDriver as direct access to the UI layer. If you have a well designed test harness, testers will not even need to know about WebDriver.

It is only the test harness, which knows how to access for example components through the Ext JS Component-Manager. It is only the test harness which has to deal with asynchronous checks to the UI. The complexity is well hidden from the testers which makes writing tests much easier – and they are easier to read.

Another advantage of such a test harness despite clean test code is the maintainability of the tests. In 2008 I described this in Testing Experience in the article Boon and Bane of GUI Test Automation (see 04/2008 or my article at QFS):

“Test execution script and GUI Maps can be maintained separately. As a consequence: If only the layout changes, you only have to adapt the GUI Map at one specific location.”

In contrast to the test harness the GUI Maps where just a database to look up how to access UI elements. In web context it could be a set of XPaths which can be access with a key from the tests.

The test harness allows much more, which I described as framework:

“To get around duplicated code, I decided to build up a framework. The goal was to create a framework to get as fast as possible to the test case you want to execute. If you need some kind of document to write to, no matter what it is: the framework will offer a function for this.”

More concrete: If your RIA has something like a file chooser, the file chooser test wrapper can provide a method like:

createDocument(folder, documentName)

which will do all required actions for you as for example:

1. Open and select the specified folder in the folder tree.
2. Wait until the folder is selected.
3. Click Create Document Button.
4. Wait for the tree editor to receive the focus.
5. Enter the document name right into the tree.

And the GUI Map I described before evolved into Ext JS expressions mainly based on the Ext JS component manager. So in complete Ext JS notion accessing a button might look like this:

Ext.getCmp("fileDialog")
   .find("itemId", "toolbar")
   .find("itemId", "createDocument")

In the harness you will have a FileDialog class which you can query for its Toolbar which again you can query for the CreateDocumentButton. And all of them will only know their direct containers and their very own expression. So the button will know that it can be found if the container is asked for an element with itemId = "createDocument".

Advantage: If the access to a parent element ever changes (and it will!) you just have to update the parent element. All child elements will automatically adopt the change.

So, this is, as always, the most basic rule to keep your UI tests maintainable:

Separate access to the UI from your test code.

Choice between Given, When, Then

Writing your first Stories for Behavior Driven Development (BDD) you will soon come to the point when it is about to consider if a step is Given, if it is When or if it is Then. So what’s better:

Given Application A is running
When I log in

or is it better to write:

Given Application A is running
Given I logged in

Mind that the second Given could be replaced by And. I just wanted to emphasize that it is now a Given statement.

My answer: It depends. Derived from one of my previous posts Given, When, Then – Which Tense? my conclusion is:

• Given: Should summarize which is not actually in focus of the story but required to understand the context of the test or state of the system under test.
• When: Describes the steps required to reach the state you want to test. If steps cannot be taken your applications behavior might have changed but your story is not necessarily broken. Thus if When statements are written with help of JUnit you should consider to user org.junit.Assume to validate the state.
• Then: This is the state you want to test. If this fails your story is for sure broken. Thus again if you are using JUnit you would use org.junit.Assert in here.

Considering this here are the three stories which deal with log in:

Given Application A is running
When I log in as user with username N
Then I will see the UI of Application A
And I will see username N somewhere on the screen

So here the log in process is part of the steps to reach the state to test, which is to have access to the application’s UI.

Given Application A is running
And I am logged in as user N
When I change my password to password P
Then I will receive a notification email
And my credentials change to username N with password P

Thus log is only required to understand the context/state of the system. And finally:

Given Application A is running
And there is a registered user N
When the login screen appears
Then I am able to log in with valid credentials for user N

This time the log in process is in focus of the scenario we want to test.

Is this guidance of any help for you? Or do you have another pattern when to choose When, when to choose Given and when to choose Then?

BDD Steps: Free your mind!

I just started to implement the steps for something like the following story:

It started to get complicate as I was in need of a parameter converter (JBehave) which converts “Article” into an object representing the document type. After some time I realized that the story was again written by someone who knows the product in too much detail.

I stopped work on the converter and rewrote the story – and now the story tells what the customer really wants:

where “rendered property” (we are editing web-contents here) is another learning to specify cleanly what you want – here: If you want to show the text in the preview later on you first should ensure to choose a property which will become a visible part of the web page.

And suddenly there was no need for a parameter converter anylonger.

I don’t vote against parameter converters – I just have to repeat to myself again and again: Write what you (as customer) really want! – and don’t try to incorporate the knowledge about the test environment (here: document type of type Article exists) into the story.

Doing so will keep your tests more robust over time (perhaps one day there is no “Article” anylonger) and sometimes (as it was here) it will even ease your test code.

For how to handle the references in the stories above you might want to read my blog post “BDD Design Patterns: Reference Tracker”.

BDD Design Patterns: Reference Tracker

Writing BDD stories and implementing the steps you will soon come to the point to deal with references between steps, like:

Given I have a document A on disk
When I open document A
And I change the content of document A
And I save document A
Then document A on disk will be updated
And document A will contain the new content

Reading this you will find two references to be tracked:

• the document itself
• the content which got changed

I found many implemented solutions for this pattern, each of them a little different. Some using global variables and a central class to hold all steps, others using singletons.

Summarizing the solutions I found I introduced a new pattern which got quite positive feedback: the Reference Tracker.

The basic idea is that you will have three items to track:

1. the type of the reference (here: document or content)
2. the reference name (here: A once, unnamed the other)
3. the actual reference value

Having this you just need two methods to feed the tracker and to retrieve values from it:

The cleanup() method should ensure that you don’t collide with new scenarios. In JBehave context you should call it in @AfterScenario.

For the example above you would implement the interface with:

• Key-Type: String (“document”, “content”)
• Value-Type: Object (once file, once String, XML or whatever)

The complete Gist contains another implementation called TypedReferenceTracker which assumes that the value-type is the same as the key, so that there is a class for documents and one for contents. It makes the getter a little easier to use.

Find the complete Gist at GitHub.

Code Kata with DDT in JUnit

Quote from Wikipedia

Code Kata is a term coined by Dave Thomas, co-author of the book The Pragmatic Programmer, in a bow to the Japanese concept of kata in the martial arts. A code kata is an exercise in programming which helps a programmer hone their skills through practice and repetition. As of October 2011, Dave Thomas has published 21 different katas.

Today I participated in my first Code Kata initiated by one of our Scrum Masters. My intention was to get to know this concept in more detail and to see if it is a forum to train BDD skills.

Looking back I am sure that a BDD Code Kata needs some preparation but I also got more practice on Data Driven Testing (DDT). We decided to code the kata Roman Numerals.

We started very simple minded with a test, testing that 1 is converted to I, then 2 to II and of course, simple minded as we are we implemented it by concatenating as many I’s as the number has. And of course this implementation first failed for 4.

First we started writing one test method for each new number we wanted to test. The first discussion which came up:

Shouldn’t we just write one test method testing all numbers?

I answer: No – because if you have 10 assertions and the second one fails you will have no hint about the state of the subsequent 8 assertions as JUnit will just stop this very test. Having multiple short test methods has the advantages:

• Methods are much easier to read.
• The different failed tests might help to isolate the source of the problem. Let’s say a test for all even number fails than you might have a first hint what to look for.

DDT – what?

Data Driven Testing is to have a set of data to feed a test with. They might come from a file or any other data generator. The allurement having data driven tests is that the code to test might be very short but you can do many boundary tests with nearly no additional costs.

The following is an example how to do DDT with JUnit. It’s the result of the short work we did today and of course far from being perfect or complete:

Conclusion

I will take part in the next Code Kata next week. It was very refreshing to be able to train techniques I use rarely (like DDT) and to discuss several approaches to a problem in a group.

Links

• Image from morguefile.com
• Image edited with Aviary Advanced Image Editor
• Software Testing Glossary
• Coding Dojo Wiki
• it-agile about Code-Katas and Coding-Dojos (German)

BDD Steps: Be specifically unspecific

I just participated in an interesting sprint retrospective meeting where the problem came up that step definitions are sometimes very ambiguous as well as for the human reader as for the JBehave parser.
Here is one typical though virtual example:

Problems of the steps above:

• The Given statement contains information with no value for the customer. The customer is not interested about the username and the password. Thus even with this start it will be hard to understand the complete scenario.
• The first When statement misses important information. Is this a valid or invalid login? When implementing this step developers might as side effect create a new user with these credentials. The next one re-using this step in a story might not know about this side effect.
• The second When statement collides with the first one if you parse parameters. If you are unlucky you even might have two step definitions in your code:
  
  with the effect that it is up to JBehave if the user logs in as "named qwert" or "qwert" depending on which step it has chosen to take.

What we can learn from it is to be specifically unspecific... or to be more precise, apply these rules to your stories:

1. Give every step a value (if not even every single phrase).
   In the first statement it might be better to state either that I logged in as normal user or that I logged in as a specific user I want to reference later in the test.
2. Never specify field values explicitly.
   Actually this refers to the first rule as most of the time the real content of a field is not important. It might be important of what type the input is (alphabetical, numeric, valid login) but most of the time the real input is not important.
3. Be as specific as needed and as unspecific as possible.
   Perhaps the rule with the most problems to decide what is the best approach. It means that you should not be too specific for example by stating "When I fill the username field with username qwert" as perhaps some day your username field disappears and is replaced by a fingerprint scanner. But you should state if the login is valid or invalid and if it is invalid: if it is because the user doesn't exist or because the password is wrong.
   

So the following steps will be easier to understand... even without the context of prior or subsequent steps. And they never state any field value:

Then vs. Period–Writing JBehave Stories

Writing stories/scenarios which get automatically executed as in JBehave is sometimes a special art – at least if you want to re-use existing step definitions. Sticking to some rules will help to make your steps re-usable, one is for example to agree on the tense to use in your stories as I mentioned earlier.

I stumbled across another rule when I was writing steps and then implementing them. Here an example:

Implementing the @Then step might look like this:

Referring to the title of this post you might guess the problem: For some yet unknown reason the first scenario marked the @Then step as pending. Solution after hours of debugging (just kidding): The period at the end of the sentence is missing in the step definition!

So you could add an @Alias for this (no, really, you don't want that). Next time you will stumble across parameterized steps where the parameter is at the end of the sentence. Suddenly the parameter will get a period at the end. And suddenly usernames for example don’t match.

The solution is quite simple. I found it in some existing stories of a team:

Don’t use periods!

Really, I hate this approach as from my point of view every new rule will prevent stories to be written. But I think this rule helps so much, that you should stick to it.