Skip to main content

Testing of Automated Tests?

I've often mentioned how important automated tests are throughout my blog. However, automated tests can also give false positives... This is where paired programming and testing of the automated tests come in to play. The example I am going to use employs the Page Object model with BDD, more of which can be read about here and here, until I get round to creating a new blog post on it...


Scenario Outline: Search for generic term
Given I am on the homepage
And I have entered <searchterm> into the search box
When I click search
Then search results for <searchterm> are displayed

Let me explain anyway, if we have an assertion in the step definition for the Then step (written in C# using NUnit) that does the following:






[Then(@"search results for (.*) are displayed")]
        public void SearchResultsForAreDisplayed(string searchTerm)
        {
            Assert.IsTrue(_searchPage.SearchTitleTextDisplayed(searchTerm));
        }

(_searchPage.SearchTitleTextDisplayed(searchTerm) is a method that lives in the page object for Search Page, that returns true if the SearchTitleText is displayed, the same can be assumed for the methods mentioned below)

It's asserting that the search results header is displayed on the page, signifying one would suspect that the search was successful.

However, upon closer inspection that search results header is also displayed when there are no search results:



So now, we have a test that looking at the Step Definition is asserting that the search results header is displayed, which it is, and passing the test as a success for having search results displayed. The problem, which I'm sure you've already guessed, is that the search results header is also displayed when there are no search results, so this means the test will pass regardless if search results are displayed or not.

A much better approach, would be to assert that products are displayed and that there are refinements as well as the search results header. So we now have:

        [Then(@"search results for (.*) are displayed")]
        public void SearchResultsForAreDisplayed(string searchTerm)
        {
            Assert.IsTrue(_searchPage.SearchTitleTextDisplayed(searchTerm)), "Search Title is not displayed";
            Assert.IsTrue(_searchPage.SearchResultsDisplayed()), "No Search Results are displayed";
            Assert.IsTrue(_searchPage.RefinementsDisplayed()), "There are no refinements displayed";
        }

These new assertions, mean that if one of them fail, the test fails, and you get information on the assertion that failed.

 You could argue the case for having one assertion, that correlated all of the above into one simple assertion at the step level:

        [Then(@"search results for (.*) are displayed")]
        public void SearchResultsForAreDisplayed(string searchTerm)
        {
            Assert.IsTrue(_searchPage.SearchPageResultsDisplayed(searchTerm)), "Search Results Page is not displayed correctly";
        }


However, out of the 2, I prefer the original method, with 3 different assertions, purely for 2 reasons:

  • The step is easy to read and understand what it is doing.
  • If one of the assertions fail, then the message displayed is more informative for debugging purposes than the second method (you could return the message that fails to the assertion, but it's still not as clear in my opinion)
There is the positive of the step definition getting larger as you want to assert more and more things on the step (with the earlier method), I would however say that ideally there shouldn't be more than 1 assertions per a step (although i am sure there are exceptions to the rule)....

So, based on the above (1 assertion per step and to keep sizes of steps to a minimum), we can improve this even further, you could have multiple steps, with one assertion in each, this is much cleaner, and probably my favoured method. 

        [Then(@"search results for (.*) are displayed")]
        public void SearchResultsForAreDisplayed(string searchTerm)
        {
            Assert.IsTrue(_searchPage.SearchResultsDisplayed()), "No Search Results are displayed";

        }
        [Then(@"search title text of (.*) are displayed")]
        public void SearchTitleTextOfDisplayed(string searchTerm)
        {
            Assert.IsTrue(_searchPage.SearchTitleTextDisplayed(searchTerm)), "Search Title is not displayed";

        }      
        [Then(@"search refinements are displayed")]
        public void SearchRefinementsAreDisplayed()
        {
            Assert.IsTrue(_searchPage.RefinementsDisplayed()), "There are no refinements displayed";

        }


This gives you the benefit of individual error messages should one fail, and at the same time they will fail the test if one of them fails and they keep the step definition down to a minimal size. This results in a new Scenario Outline of the following:

Scenario Outline: Search for generic term
Given I am on the homepage
And I have entered <searchterm> into the search box
When I click search
Then search results for <searchterm> are displayed
And search title text of <searchterm> is displayed
And search refinements are displayed

It's a lot more informative for the tester who is running the test to see exactly what it is doing, even without looking at the code, they can see instantly what checks are being performed.

We could add more checks, but for the sake of this exercise, it's important to grasp what the automated test is checking for, before it is being written. It's also important to structure the test with what you want to test in mind

This example highlights the importance of the tester either writing the steps themselves or pairing with a developer as they create the tests, as a developer may feel that the first example is enough, but obviously, as QA we know we have to test the automated tests themselves, to ensure that the tests don't give false positives. 

If I'm honest, i prefer the pairing with the developer approach, as then knowledge is shared, and the QA and Dev learn a little bit more about the others skill set and it helps build team relationships.


Comments

  1. can you share the code snippet of this example? I am new to Gherkin and specflow very keen to learn this, but having difficulties with the actual implementation using specflow, I have tried to search on the internet but none of them are not complete.

    Thanks,
    Amar

    ReplyDelete
    Replies
    1. Sure thing. It's here: https://github.com/gwaterhouse85/AsosSpecifications

      Delete
  2. I am also strongly motivated to learn the language but getting some difficulties in coding. I'm searching for those problems over Internet but coulds't find the solution. Maybe you can help me out.

    ReplyDelete
  3. REALLY VERY EXCELLENT INFORMATION. I AM VERY GLAD TO SEE YOUR BLOG FOR THIS INFORMATION. THANKS FOR SHARING. KEEP UPDATING.

    NO.1 AQM Services | Application Quality Managment Services | Austere Technologies

    ReplyDelete
  4. wow...nice blog, very helpful information. Thanks for sharing.

    NO.1 API DEVELOPMENT SERVICES | MASSIL TECHNOLOGIES

    ReplyDelete

Post a Comment

Popular posts from this blog

What is a PBI?

After my last post, I had the question of what is a PBI... so I thought i'd write a short blog post about what they are and why they are used.

A PBI is an acronym for Product Backlog Item. It is a description of a piece of work that your SCRUM team will develop and deliver. When you have a list of Product Backlog Items, you then refer to that collective list as a Product Backlog.

The product backlog is often prioritised and yourteam will work through each PBI, and release on a regular schedule... I am however going deep into the world of Agile development, which isn't entirely what this post is about, so I will stop myself now.

A Product Backlog Item is made up of the following:

Title - This is often a one liner that gives the team an idea of what the PBI is about, although it can just be an ID for the item and the team work off of that.

Description - Breaks down the PBI in a bit more detail, and can be written in any style, however I prefer it to be written as follows: 



By writin…

Dealing with Selenium WebDriver Driver.Quit crashes (Where chromedriver.exe is left open)

We recently came across a problem with Selenium not quitting the webdriver and this would then lock a file that was needed on the build server to run the builds.

We were using Driver.Quit() but this sometimes failed and would leave chromedriver.exe running. I looked around and found this was a common issue that many people were having. We (I say we, as we came to the solution through paired programming), came up with the following, that would encapsulate the driver.quit inside a task and if this task takes longer than 10 seconds, then it will clean up any processes started by the current process, in the case of the issue on the build server, it would kill any process started by Nunit.

[AfterTestRun]
        public static void AfterTestRun()
        {
            var nativeDriverQuit = Task.Factory.StartNew(() => Driver.Quit());
            if (!nativeDriverQuit.Wait(TimeSpan.FromSeconds(10)))
            {
                CleanUpProcessByInheritance();
            }
       }

        private s…

Advantages of using Test Management tools

Before I start talking about test management tools, let me clarify what I mean by the term test Management tools...  I am not taking about your office excel program where you store your test cases in. I'm talking about bespoke test Management tools, your quality centers or Microsoft test manager...
In the strict case of the term test Management tool, Microsoft Excel can be used as such, but heck, so could a notepad if used in the right way... For the sake of this blog post I am talking about bespoke test Management tools.
Firstly, what test tools are out there? There are many more out there today than when I first started in QA over 5 years ago. When I started the market was primarily dominated by a tool called Quality Center, this would run in a browser (only Ie unfortunately) and was hosted on a server.. Nowadays it's market share has somewhat dwindled, and there are some new kids on the block. 
One of the more popular tools is that of Microsoft Test Manager, it's big…