Skip to main content

Using data to drive and enhance our Test Approach

In light of the recent (well I say recent, I mean just over a month) news about Edward Snowden whistleblowing on the NSA and the Prism programme which detailed how the NSA were afforded direct access to data held by companies such as Facebook, Google and Apple, I thought it would be interesting to discuss how data in general can be used (legally) in helping define a test approach.

Obviously the Snowden whistleblowing is about a lot more than just data like I'm going to be discussing here, it's about being private online, in my eyes Snowden has done a very brave and very good thing for his country and for the internet community. What happens next with regards to the American public and their willingness to challenge the government will determine if it was worth it or not. (You can read a timeline of events on Snowden here).

However, in relation to testing, a big pool of data with regards to your application/website will come from analytics, be it WebTrends or Google Analytics. I am sure that many companies are already taking advantage of analytics data to determine what browsers are most popular and what devices are used to access their companies website or application, in having this data we are able to determine what browsers and devices we support and thus test on. This list will grow larger and larger as more and more companies release devices or as different versions of browsers are released, we already for instance have Internet Explorer 7,8,9 and now 10, I'm just thankful that we stopped supporting IE6 as that was a monstrosity. So this is just one way that we are using data to support our testing.

Also in our analytics data are things like user journeys, and common things that users do, we can use this to drive our testing, if there's a common journey that a number of users do, then we can ensure in our testing that there's nothing that will disrupt that journey, often the journey is common sense, but it's a useful piece of data non the less.

We can also use other forms of data to determine where bugs are likely to appear in the system, if you have a system that has evolved quite a bit and you have a lot of bugs that have been raised and subsequently fixed, if you know that a certain area of the system is prone to having regression type bugs, then you can be sure that you have tests that cover this area in your test pack, or even perform exploratory testing around it as necessary. This is where CI builds can come in handy, by having tests run every night against a build, you can determine fairly easily if certain areas of the system are prone to bugs, and as such protect against this going forward.

Finally, analytics can be used to determine how often certain scenarios happen, for instance, we have a timeout page that has certain analytics on it. When this page is hit, we know what the user was doing when they hit the page, and it will help in researching what the problem was and how it arose.

Data doesn't have to be limited to analytics, we can increase this "data" to using tools such as SQL server profiler, if we have this running during an a test run then this data in future can be used to drive any improvements that we want to make to the databases to improve efficiency or even to find bugs that might otherwise had been missed.

Similarly, we can run tests with console windows open to see any errors, we can use HttpWatch to see requests that the browser is making, we can view logs on servers. This is all data that is available to us, and as testers we should be using it regularly as part of our test runs.

Obviously, this data might not be available to everyone, so is there a risk that smaller companies get left behind? In my eyes, no, smaller companies will eventually have this data available to them, be it in having multiple releases that have had tests run against them, they too will have this data, if they decide to use it or not is up to them.

The point I'm getting at, is that there is an abundance of data out there, some readily at our fingertips, and some we might have to tweak/work for a bit, but the more data as testers we have then the more pinpointed our testing can be, and not one piece of this data alone would be any use to us, but combining it all together helps our test approach become more direct and focused.

These are just some ways that we use data from real life customers to help us in our day to day jobs as testers, are there any other ways that you can think of that we can use any data from the past, to help drive our testing approach?


Popular posts from this blog

Advantages of using Test Management tools

Before I start talking about test management tools, let me clarify what I mean by the term test Management tools...  I am not taking about your office excel program where you store your test cases in. I'm talking about bespoke test Management tools, your quality centers or Microsoft test manager...
In the strict case of the term test Management tool, Microsoft Excel can be used as such, but heck, so could a notepad if used in the right way... For the sake of this blog post I am talking about bespoke test Management tools.
Firstly, what test tools are out there? There are many more out there today than when I first started in QA over 5 years ago. When I started the market was primarily dominated by a tool called Quality Center, this would run in a browser (only Ie unfortunately) and was hosted on a server.. Nowadays it's market share has somewhat dwindled, and there are some new kids on the block. 
One of the more popular tools is that of Microsoft Test Manager, it's big…

What is a PBI?

After my last post, I had the question of what is a PBI... so I thought i'd write a short blog post about what they are and why they are used.

A PBI is an acronym for Product Backlog Item. It is a description of a piece of work that your SCRUM team will develop and deliver. When you have a list of Product Backlog Items, you then refer to that collective list as a Product Backlog.

The product backlog is often prioritised and yourteam will work through each PBI, and release on a regular schedule... I am however going deep into the world of Agile development, which isn't entirely what this post is about, so I will stop myself now.

A Product Backlog Item is made up of the following:

Title - This is often a one liner that gives the team an idea of what the PBI is about, although it can just be an ID for the item and the team work off of that.

Description - Breaks down the PBI in a bit more detail, and can be written in any style, however I prefer it to be written as follows: 

By writin…

Dealing with Selenium WebDriver Driver.Quit crashes (Where chromedriver.exe is left open)

We recently came across a problem with Selenium not quitting the webdriver and this would then lock a file that was needed on the build server to run the builds.

We were using Driver.Quit() but this sometimes failed and would leave chromedriver.exe running. I looked around and found this was a common issue that many people were having. We (I say we, as we came to the solution through paired programming), came up with the following, that would encapsulate the driver.quit inside a task and if this task takes longer than 10 seconds, then it will clean up any processes started by the current process, in the case of the issue on the build server, it would kill any process started by Nunit.

        public static void AfterTestRun()
            var nativeDriverQuit = Task.Factory.StartNew(() => Driver.Quit());
            if (!nativeDriverQuit.Wait(TimeSpan.FromSeconds(10)))

        private s…