Skip to main content

Should a failed test on CI automatically raise a bug?

I got into work this morning, and was looking at some CI Builds, and had a thought that it might be useful to have a bug raised automatically (through an API) when a test that is running on CI failed.... I tweeted about it to get peoples opinion(s)...



I got a few responses on twitter, and one which made me rethink why this might be a good idea and that was:


It could easily lead to and incredible amount of noise if we are raising bugs against every failed test, and also 1 failed test case != 1 bug, in that if 3 automated tests on CI fail then we don't necessarily need 3 bugs. It could also potentially (as my reply states) devalue what a bug actually is, meaning that if a tester raises a bug then it might get ignored/lost in all the noise. 

I also asked our internal QA Slack channel, responses were informative, and again helped lead me away from this potentially noisy and crazy idea.



Both of these points are extremely valid, and if you make it the number 1 priority to fix a broken test then why bother to create a bug? Surely just the visibility of a red cross next to a test run is enough to get people to work on it? Which is very true. 

Another factor to consider is the type of test, if a Unit Test fails, do we need a bug? Most definitely not.... There is definitely more of an argument if an Acceptance Test fails though (though after this discussion today I don't think there is!).

So, what started out as a blog post about CI and bugs, has provided me with 3 insightful thoughts:

1 - The focus of this blog post being that a failed test on CI most definitely does not mean a new bug (whether the process is manual or automated in creating said bug) needs to be raised. It ultimately would devalue a bug, it would mean that people wouldn't necessarily talk about said issues/bug, and 1 failed test != 1 new bug.

2 - Asking people for feedback is invaluable, it can help shape your opinion and give you quick feedback before you go off down a rabbit hole!

3 - Discussions like this are the exact reason why I wanted to have an internal QA community, so we can discuss things and get feedback.


Comments

  1. Quite an insightful post. This has cleared so many of my doubts in this subject & has thrown light on many aspects that I didn’t know before.

    ReplyDelete
  2. there is always first time for everythign in life and it only takes one small step to conquer the greatest fears although this small step could take alot of courage but at the end its always worth it.

    ReplyDelete

Post a Comment

Popular posts from this blog

What is a PBI?

After my last post, I had the question of what is a PBI... so I thought i'd write a short blog post about what they are and why they are used.

A PBI is an acronym for Product Backlog Item. It is a description of a piece of work that your SCRUM team will develop and deliver. When you have a list of Product Backlog Items, you then refer to that collective list as a Product Backlog.

The product backlog is often prioritised and yourteam will work through each PBI, and release on a regular schedule... I am however going deep into the world of Agile development, which isn't entirely what this post is about, so I will stop myself now.

A Product Backlog Item is made up of the following:

Title - This is often a one liner that gives the team an idea of what the PBI is about, although it can just be an ID for the item and the team work off of that.

Description - Breaks down the PBI in a bit more detail, and can be written in any style, however I prefer it to be written as follows: 



By writin…

Dealing with Selenium WebDriver Driver.Quit crashes (Where chromedriver.exe is left open)

We recently came across a problem with Selenium not quitting the webdriver and this would then lock a file that was needed on the build server to run the builds.

We were using Driver.Quit() but this sometimes failed and would leave chromedriver.exe running. I looked around and found this was a common issue that many people were having. We (I say we, as we came to the solution through paired programming), came up with the following, that would encapsulate the driver.quit inside a task and if this task takes longer than 10 seconds, then it will clean up any processes started by the current process, in the case of the issue on the build server, it would kill any process started by Nunit.

[AfterTestRun]
        public static void AfterTestRun()
        {
            var nativeDriverQuit = Task.Factory.StartNew(() => Driver.Quit());
            if (!nativeDriverQuit.Wait(TimeSpan.FromSeconds(10)))
            {
                CleanUpProcessByInheritance();
            }
       }

        private s…

Famous Movie Quotes applied to Software Engineering - Jaws

You're gonna need a bigger boat? How can that relate to Engineering?

Firstly, let me ashamedly admit, that I've never seen the whole of Jaws all the way through. It's on my list of films to watch, but whether I get round to it, is another matter!



Anyway, to apply this to engineering, it's almost like "you're gonna need more testers/developers"...

We hear this all too often when trying to push releases out the door, let's throw men at it... However, as we all know, a bigger boat/more men... isn't always the answer, it's not a guarantee of quality, or even a guarantee of getting things done quicker.

If you have a task that will take 2 hours, simply having 2 people work on it doesn't mean that it is halved, in fact often, the time taken to do the task remains at 2 hours, but the maintainability and the knowledge around that area is increased, so it's a price, in my opinion that is often worth paying.