The Automated Tester

Automating the Capture of Web Timings with Selenium 2

Measuring the speed of how long a page takes to load is becoming one of the most important metrics that we as software developers or testers can measure. This is because milliseconds can mean the difference between a sale being paid for or abandoned. A couple of years ago Amazon did research that showed for every 100ms slower the page was in loading they lost 1% of sales.

The downside to all of this is that we, as developers, testers and even ops, can only control the speed of things within our code and within our datacenters. With this in mind we should be trying to record all of the information that we can from within the browser. Last year at GTAC 2009 ( presentation video) David Henderson and I discussed how we used YSlow and Selenium to record page metrics and load times of the web application. We made the total primed page size decrease by 85% with the work we did.

The unfortunate thing about the way we did this was that we had to hack YSlow because it didn't give us all the information that we wanted. We also needed to hack Selenium because it blocked a few of the headers to do with element expiry. This is not really ideal but with the new Web Timings API we are starting to be able to do all of this with just a bit of JavaScript.

To access this we need to get hold of the performance object and then get it to return a dictionary of all the timing data. The values that are returned are milliseconds from 1/1/1970. They give you all cool bits of information like how long it takes to unload a page you're on before navigating to the new page and how long that takes to load. It also tells you how long domain lookups take and how long redirects take. These are all neat tidbits of information when performance tuning your web application.

So below is a C# extension method that I have created that extends the WebDriver object to collect the information for you. Not all browsers support this at the moment but as web timings becomes the standard the code will start returning data. Current browsers that support this are Chrome 6 and IE9 pre3. A patch has been submitted to Mozilla for Firefox 4 so hopefully this will be applied soon.

 
using System;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using OpenQA.Selenium;

namespace AutomatedTester.PagePerf
{
    public static class Extensions
    {
        public static Dictionary<string,object> WebTimings(this IWebDriver driver)
        {
            var webTiming = (Dictionary<string, object>)((IJavaScriptExecutor)driver)
                .ExecuteScript(@"var performance = window.performance || window.webkitPerformance || window.mozPerformance || window.msPerformance || {};
                                 var timings = performance.timing || {};
                                 return timings;");
			/* The dictionary returned will contain something like the following.
             * The values are in milliseconds since 1/1/1970
             * 
             * connectEnd: 1280867925716
             * connectStart: 1280867925687
             * domainLookupEnd: 1280867925687
             * domainLookupStart: 1280867925687
             * fetchStart: 1280867925685
             * legacyNavigationStart: 1280867926028
             * loadEventEnd: 1280867926262
             * loadEventStart: 1280867926155
             * navigationStart: 1280867925685
             * redirectEnd: 0
             * redirectStart: 0
             * requestEnd: 1280867925716
             * requestStart: 1280867925716
             * responseEnd: 1280867925940
             * responseStart: 1280867925919
             * unloadEventEnd: 1280867925940
             */ 
            return webTiming;
        }
    }
}

Selenium StackExchange Commitment

First let me start by thanking those who followed the Selenium StackExchange Proposal. We managed to meet all the criteria that was set by the StackExchange Team and have moved onto the second stage.

The second stage is the commitment stage. This is where we get people to commit that they are going to use it to ask questions or answer questions when they are put on the site. This is a very important stage because it shows how strong the Selenium Community is and in my opinion we do have a strong community. Unfortunately, my opinion doesn't matter that much in the StackExchange process so if you haven't committed to the Selenium StackExchange Site please click on the link and commit.

You never know when you may need an answer to a question that may already have been asked and it's easy available to you with the new site

Writing maintainable tests

A question I get asked regularly, and is regularly asked on forums I visit, is how do "I make my tests maintainable?". This is normally from a tester upset that the developers are commenting out tests because they feel its too much work to update the tests. The next popular question is "how can I get my non-technical colleagues creating automated tests?". For me the answer to both is very similar. Make the tests self-documenting and easy to read (maybe using a DSL). You, as a developer, would do this with your production code so why not do it with your tests.

Now, what do I mean by self-documenting and easy to read?

Methods should explain what they are for. This goes for test and production.
I like to format my test methods like ShouldDoX. This is similar to what BDD practioners do for their tests. If another prerson comes along to look at some code that you have written, and your tests, they will be able to see what you meant it to be doing.
```
	 public void ShouldLoginAndChangeStatus() 
	 
```
for a social networking app explains exactly what they are expecting.
Methods and tests should be succinct
Methods that are huge, in test and production, are likely to have bugs in them because people have to scroll to get to the bottom. They are likely to have business logic bugs when that happens. I also like abstract as much as I can so that it makes the code readable.

Which is better for readability?
```
	selenium.click("//a[contains(@blah,'someIdButNotAllOfIt)]")
	
```
or
```
	// linkToCoolStuff is a constant holding the XPath above
	selenium.click(linkToCoolStuff)
	
```

Maybe using a DSL?
We could take this one step further and create a DSL so that we do not have to care how it does it, we just know it will happen. For UI testing I like the idea of the Page Object design pattern for creating tests. You split the UI and the tests apart so that you can have a test looks like the following.

    [Test]
    public void ShouldLoadHomeThenGoToXpathTutorial()
    {
        Home home = new Home(selenium);
        SeleniumTutorials seleniumTutorials = home.ClickSelenium();
        SeleniumXPathTutorial seleniumXPathTutorial = seleniumTutorials.ClickXpathTutorial();
        Assert.True(seleniumXPathTutorial.
					IsInputOnScreen(SeleniumXPathTutorial.FirstInput));
        Assert.True(seleniumXPathTutorial
					.IsInputOnScreen(SeleniumXPathTutorial.SecondInput));
        Assert.True(seleniumXPathTutorial
					.IsInputOnScreen(SeleniumXPathTutorial.Total));
    }

I have created a tutorial on how to use the Page Object Pattern with Selenium to make tests that are extremely maintainable and also allows non-technical people to write tests without having to understand how Selenium works.

Selenium Page Object Pattern

In this tutorial we will have a look at how we can apply the Page Object design pattern. This is an extremely useful design pattern for UI tests as it abstracts away how we get to different pages or screens. Page Object design helps us create a DSL for our tests that if something were to change on the page we don't need to change the test, we just need to update the object that represents the page.

A large number of people prefer to write tests in this manner than having to use Selenium IDE. I have to admit that I am one these people. It does require a bit of up front work but once done non-technical can start writing tests in a easy to use manner and never have to understand how everything holds together.

The following example is done in C# and requires that you have NUnit and at least Visual Studio Express installed.

In the example below I have created my Page Objects to work against my site.

Create a class file for a page. In this case let's start on the Home Page. When this object is instantiated we are going to navigate to that page. From there we

public class Home
{
    private readonly ISelenium _selenium;
    
    /// 
    /// Instantiates a new Home Page object. Pass in the Selenium object created in the test SetUp(). 
    /// When the object in instantiated it will navigate to the root
    /// 
    /// Selenium Object created in the tests
    public Home(ISelenium selenium)
    {
        this._selenium = selenium;
        if (!selenium.GetTitle().Contains("home"))
        {
            selenium.Open("/");
        }
    }

    /// 
    /// Navigates to Selenium Tutorials Page. Selenium object wll be passed through
    /// 
    /// SeleniumTutorials representing the selenium_training.htm
    public SeleniumTutorials ClickSelenium()
    {
        _selenium.Click("link=selenium");
        _selenium.WaitForPageToLoad("30000");
        return new SeleniumTutorials(_selenium);
    }

    /// 
    /// Click on the blog or blog year and then wait for the page to load
    /// 
    /// blog or blog year
    /// Object representing /blog.* pages
    public Blog ClickBlogYear(string year)
    {
        _selenium.Click("link=" + year);
        _selenium.WaitForPageToLoad("30000");
        return new Blog(_selenium);
    }
    // Add more methods as you need them
}

Let's create another object that will represent one of the tutorials on the site

public class SeleniumXPathTutorial
{
    private readonly ISelenium _selenium;

    public const string FirstInput = "number1";
    public const string SecondInput = "number2";
    public const string Total = "total";

    public SeleniumXPathTutorial(ISelenium selenium)
    {
        this._selenium = selenium;
    }

    public bool IsInputOnScreen(string locator)
    {
        return _selenium.IsElementPresent(locator);
    }
}

Let us now create a test class that will hold all of the tests. We will need to create an object for each of the pages and then use the methods in them our tests. As we can see from the code below, the Selenium parts have been kept to a minimum. In actual fact the only bit that shows Selenium is in the Setup() and Teardown() methods.

[TestFixture]
public class SiteTests
{
    private ISelenium selenium;
    [SetUp]
    public void Setup()
    {
        selenium = new DefaultSelenium("localhost", 4444, "*chrome", "http://www.theautomatedtester.co.uk");
        selenium.Start();
    }

    [TearDown]
    public void Teardown()
    {
        selenium.Stop();
    }

    [Test]
    public void ShouldLoadHomeThenGoToXpathTutorial()
    {
        Home home = new Home(selenium);
        SeleniumTutorials seleniumTutorials = home.ClickSelenium();
        SeleniumXPathTutorial seleniumXPathTutorial = seleniumTutorials.ClickXpathTutorial();
        Assert.True(seleniumXPathTutorial.
					IsInputOnScreen(SeleniumXPathTutorial.FirstInput));
        Assert.True(seleniumXPathTutorial
					.IsInputOnScreen(SeleniumXPathTutorial.SecondInput));
        Assert.True(seleniumXPathTutorial
					.IsInputOnScreen(SeleniumXPathTutorial.Total));
    }
}

Now try make a few more tests to try work against the other pages in the site. If you find that you are repeating yourself in the classes split that out again. If you follow DRY (Don't Repeat Yourself) then you will have a set of tests that are extremely maintainable.

By using the Page Object you can make your tests really easy to write and when things change, as they invariably do, you can go from a number of tests failing to suddenly passing with making one change. We have followed good practise in keeping the test code (the asserts) inside the test and not in the page objects. You can also swap out Selenium 1 and put Selenium 2 in its place and not touch the tests.

Selenium StackExchange Proposal

A while ago Dave Hunt, a Selenium Contributor and London Selenium User Group organiser, suggested that perhaps we should have a look at creating a StackExchange site for Selenium. He suggested it after the StackOverflow team announced changes to StackExchange

On Wednesday Dave created our proposal so that we can get the ball rolling. I have signed up and want to help this proposal as much as I can. There are a couple reasons why I am advocating this so much.

Its a lot friendlier to use than Google groups
It will show up possible duplicates in questions. Chances are that if you are having an issue, someone else has had that issue before
It's like a game! It's points based so people will try get the most points that they can.

So what can you as a reader of this blog do? Well, if you are a Selenium User I suggest following to the proposal by logging into Area51 and clicking the follow button. The next thing to do is add some example questions if they aren't there. The more example questions the better as it will give people a better idea of what the StackExchange will be used for. Finally, start voting on the example questions. You do this by clicking on one three links on the left of the question as shown in the image below.

I have benefitted from using the StackOverflow type sites(StackOverflow, ServerFault, etc) since their launch and I think that as a community we can all benefit from this site so please help the cause

What is your Label? Male, Female or IT Professional?

This morning I work up this morning thinking that today was going to be a really good day. The sun was shining and it was really warm. The thoughts going through my head was "I need coffee then check my emails then phone my friend Chris and see if he wants to cycle over to the Isle of Purbeck today"

I did all of those things quite successfully and then thought, lets have a look at what is happening on twitter. This was both a good idea and a bad idea. It was a good idea as I got to see what was happening in the world. The bad idea part was I saw battle of the sexs happening between Lanette Cream and James Bach about diversity in the industry.

This appears to have upset a number of people as their were some negative thoughts from both parties. The main crux of the argument was one was saying we need to encourage diversity and the other saying thats not true.

For me I think we need to encourage individuality. I grew up in South Africa and have seen the aftermath of Apartheid and have been victim to affirmative action. Affirmative Action was the main reason why I decided to move to the UK after university. Filling positions to meet quotas is always going to do more harm than good. There were stories, don't know how true, of secretaries becoming directors of companies in South Africa so that the "balance" was corrected.

This idea makes me mad! Everyone should get to somewhere on their own merits and driving forward this idea that we need to get more woman into IT is going to do more harm. We need to get more people in general into the industry. This shouldn't matter if they are male, female, pink, green or indifferent. Over the last decade there has been a drop in graduates for maths and science degrees. There are fewer Computer Scientists, engineers and mathematicians. We should be driving that forward and hopefully that will organically get people from diverse backgrounds into our industry.

I had a brief discussion over twitter with Michael Bolton where we seemed to agree that we need to have different perspectives on applications when we test. This does comes from hiring different sexs and races but that doesn't guarantee anything. People who go to similar schools will have similar perspectives on things. Being from South Africa I had people of different races and ethnicities in my classes but we all have similar perspectives. This alone makes me hate the diversity argument on sex/race.

I am an IT professional so treat me like one!

The Value of a good bug report

I have recently come back from a well deserved holiday in South Africa. Now that I am back I thought that I would carry on working on the .NET Selenium 2 code. I am just about to port the Support project over but thought that I would check that ther weren't any blocking issues that people were experiencing with the .NET Code.

So over the last couple days I have been going though the issues that are on Google Code and have been suprised by the quality of the bug reports that people have been submitting. This is from people who spend most of their day writing bug reports or spend their days reading bug reports to fix issues.

The question that I kept asking myself is why do they think this is a good enough bug report? Maybe they have to pay their company per character they type when not on intranet sites? They think that Selenium Developers are psychic? The make certain assumptions like everyone in the world writes tests like them against applications that are exact clones to their application.

I am both a developer and a tester so have the unique perspective when looking at bug cases. When I raise a case I make sure that I think about the what is needed for someone, who may not know the system, to fix the issue. This, for me, is an extremely powerful way to raise cases. I always raise my cases with every exact step needed to solve the issues. And if the bug is UI related I will try add a screenshot just in case I may have forgotten something.

Marlena Compton says that Being a writer made her a better tester. I have to agree with that but I think being a developer has made me a better tester too. Being a writer has made me articulate the issue clearly. Being a developer has made me write down what is needed to fix this issue without causing the other person to go grey.

Next time you raise a case, think about how you have phrased it!

Lights! Camera! Action! Video recording of Selenium

Last night I was trying to show a client of mine how a Selenium script looks when running on my linux box. My client has an issue that when they run the test the browser just dies. Screen sharing was an option but when we have done this before the screen sharing doesn't show the screen that well. The other issue was trying to get time when we were both free.

I wanted just a video of what was happening and then I can watch it to see what the issue may be. This way my client could do it in their own time when they have a moment. There are tools out there to record a person's screen. There is camtasia which is really good. Unfortunately I did not want my client to worry about when to start recording and when to stop recording. I wanted a no frills no fuss solution for them. They are not technical in the slightest which is why they hired me.

My next thought came back to something that Jason Huggins said to me last year at GTAC 2009. Jason had put a small API onto pyvnc2swf and had created Castro. Castro is a python library that allows users to programmatically record what is happening on the screen and then create a SWF file so you can view it within a web page for everyone to see. This is exactly what I was after since I could write a script for my client to run and that would record the test. I would hopefully be able to then see where the issue may lie.

It did solve my problem brilliantly and this has become the standard way for me to help my clients. If something isn't working perfectly for them I get them to work through the steps below and then record what the issue could be.

I have only done this on Ubuntu 9.10, hence the use of apt-get, but it should work on all machines except x64 Windows. This is because Python SetupTools doesn't really work on x64 Windows that well because of the way Windows controls its registry.

Setup VNCServer :
System > Preferences > Remote Desktop Prefences

Check "Allow others to view your desktop

In a terminal window run the follow:

sudo apt-get install python-dev python-pygame python-tk python-setuptools ffmpeg flvtool2
sudo easy_install castro

Once all that has been installed you are now ready to create your video of your test.

 
from selenium import selenium
import unittest, time, re, castro

class mypms(unittest.TestCase):
    def setUp(self):
        self.verificationErrors = []
        self.video = castro.Castro()
        self.video.start()
        self.selenium = selenium("localhost", 4444, "*firefox", "http://www.theautomatedtester.co.uk/")
        self.selenium.start()
        self.selenium.window_maximize()
		
    def test_my(self):
        sel = self.selenium
		sel.open("/")
		sel.click("link=blog")
		sel.wait_for_page_to_load("30000")
		
    def tearDown(self):
        self.selenium.stop()
        self.video.stop()
        self.video.process()
        self.assertEqual([], self.verificationErrors)

if __name__ == "__main__":
    unittest.main()

If you were to run the script above it would dump the file into /tmp and you can view the video over there. This is a very handy tool and a quick way to see where scripts are going wrong and hopefully be happy like me.

woohooo got #Castro working to record video for my #Selenium tests on my linux boxless than a minute ago via TweetDeckDavid Burns
AutomatedTester

London Selenium User Group

Last night was the 2nd ever Selenium User Group in London and what a night it was! Arriving at their office we were ushered up to 4th floor where in true Google style we were given food, lots of food and beer so that we could all hang out and have a chat.

The event was mainly organised by Dave Hunt and he did a brilliant job organising the speakers and with the help of Simon Stewart organising a venue with food and beer.

Dave Hunt did a talk on how to move from Selenium 1 in the IDE and move to Selenium 2. He showed how you can use the new Selenium IDE Plugin Architecture that Adam Goucher got going. Dave created the WebDriverBacked Formatter plugin with Adam and he showed it off to good effect.

It was then my turn to speak. I was speaking about Selenium 2 and what it means for the Average QA. In the talk I described the major differences like the API is a lot more concise in Selenium 2 and also describes some of the things I described in my "Why Should I move to Selenium" blog post I did earlier this year. I have put my presentation below for all to see.

What does Selenium 2 mean to the Average QA on Prezi

And I have put my demos on GitHub for all to see and play with.

Unfortunately I had to leave at this point so I missed Eran's talk and the pub session afterwards but from what I hear it was a really good way to finish the night.

Thanks to all that turned up for the User Group meeting and making a great event!

Contracts have been signed

Today I officially received my contract to write a book on Selenium. I am going to be writing a beginners guide showing people how they are start test automation working using Selenium IDE then moving on to Selenium RC and Selenium Grid. There will also be chapters on Selenium 2.

The outline I went for was to help people who want to learn Selenium so they can maintain existing tests but also start writing tests using Selenium 2 and migrating their Selenium 1 tests over to Selenium 2.

So look out for it later this year!