Skip to main content

Crowdsourcing as a viable alternative?

I was recently at a conference that discussed the use of Crowdsourcing as a form of effective testing for your company.

Firstly let me go over what Crowdsourcing is. Wikipedia describes Crowdsourcing as:

"the practice of obtaining needed services, ideas, or content by soliciting contributions from a large group of people, and especially from an online community, rather than from traditional employees or suppliers"
So in testing terms, it's getting an online community of "testers" to test your software.

So why would you use crowdsourcing to test your software? What are the positives?

The big positive for me is that you're letting the people who would potentially use your application the chance to use it and report any bugs. These testers might use the application differently than you intended though? This to me however is a good thing, if they are using it in a certain way, you can bet that the general public or whoever might use your app would use it the same way, so this is definitely a positive. (That is to say that it shouldn't replace true beta testing).

It also allows you to have access to a large number of devices, for instance you can specify what devices you want tests to run on, or want the website to be tested on. Chances are unless you have an enormous budget for buying devices there are going to be plenty of devices that you don't have and can't afford to have, that could possibly do with testing on.

Finally, you can have a big turn around on the testing. I've heard stories of insane numbers of tests being executed in an even insane amount of time! Something that I dont think a QA team can achieve, unless they have massive numbers of people, but then you have the issue of managing this. If you crowdsource it, then they will manage all of that.

What are the disadvantages? Why wouldn't you use crowdsourcing?

There are in my eyes many disadvantages to crowdsourcing, which I will go over below.

The biggest one for me, is that the testers have no business knowledge of the application. They will raise bugs that aren't bugs, they will not fully understand what the application should be doing, and with this in mind bugs may even be missed because of this.

Secondly, you have no idea who is running the test and if they are any good, there is of course an argument that the people running the tests could ultimately be the people using the software, so they don't have to be good testers, but they do need to have a good eye.

Related to the fact that you have no idea who is testing the software, in having no idea who is testing the software, you are blindly trusting someone to execute a test, and what is to say they have executed the test? You could argue you have a similar problem in that you have to trust your team to run the tests, but your team is something you have put together, and people that you know and hopefully trust.

Then there is the issue that the app/system must be publically available, which might not be a problem to some, but it is if you have a large number of people hitting your application on a small test server somewhere.

It also in my eyes, plays down the importance of testing, testing to me, isn't just about executing test cases, it's about identifying areas of risk, identifying what testing needs to be performed, understanding the application. I fail to see how crowdsourcing can offer you this.

Finally, a lot of crowdsourcing is paid per bug, this is extremely dirty to me. I don't agree that people should be paid per bug, as I don't see that motivating people to actually run the tests, as they will blindly be looking to find bugs and not execute tests. Also, the money paid for a bug would go further to motivate someone where the exchange rate is advantageous, to someone in the UK. Finally, someone could spend all day testing, and doing a good job, but not find any bugs, what is the motivation for that person to come back and perform more testing?

Some might argue that it's a form of beta testing, but again I disagree, you can't guarantee that the people testing would be people that would regularly use the application or system, beta testing should be performed by people who will use your application, not just anyone who wants to earn some money finding bugs in your software.

When would you use crowdsourcing? 

So with all those you might think that I am totally against crowdsourcing, which I am... to a degree. However, I can see when it might be useful, and that is as a complimentary effort to the testing that is already performed by a company. A company shouldn't solely rely on crowdsourcing as in my eyes there are too many unknowns over who is executing the tests.

Also, it might be useful if someone is developing a website/simple app and they need it testing for a small company, and they don't have a QA function, then they might use Crowdsourcing as a quick and dirty way of ensuring their software is ready to be released.

So, in conclusion, whilst it does have it's advantages, and can be useful for small scale applications, I don't think it's suitable for a large application or an application that is evolving that doesn't have many test cases around it. If you need to get through a ton of regression test cases on a multitude of devices, then again it might be useful, but again I would say only as a supplement to any testing that is being performed by a QA team dedicated to the application under test. 


  1. I am very new to this concept, do you recommended any websites where we can get this service from?


Post a Comment

Popular posts from this blog

Advantages of using Test Management tools

Before I start talking about test management tools, let me clarify what I mean by the term test Management tools...  I am not taking about your office excel program where you store your test cases in. I'm talking about bespoke test Management tools, your quality centers or Microsoft test manager...
In the strict case of the term test Management tool, Microsoft Excel can be used as such, but heck, so could a notepad if used in the right way... For the sake of this blog post I am talking about bespoke test Management tools.
Firstly, what test tools are out there? There are many more out there today than when I first started in QA over 5 years ago. When I started the market was primarily dominated by a tool called Quality Center, this would run in a browser (only Ie unfortunately) and was hosted on a server.. Nowadays it's market share has somewhat dwindled, and there are some new kids on the block. 
One of the more popular tools is that of Microsoft Test Manager, it's big…

What is a PBI?

After my last post, I had the question of what is a PBI... so I thought i'd write a short blog post about what they are and why they are used.

A PBI is an acronym for Product Backlog Item. It is a description of a piece of work that your SCRUM team will develop and deliver. When you have a list of Product Backlog Items, you then refer to that collective list as a Product Backlog.

The product backlog is often prioritised and yourteam will work through each PBI, and release on a regular schedule... I am however going deep into the world of Agile development, which isn't entirely what this post is about, so I will stop myself now.

A Product Backlog Item is made up of the following:

Title - This is often a one liner that gives the team an idea of what the PBI is about, although it can just be an ID for the item and the team work off of that.

Description - Breaks down the PBI in a bit more detail, and can be written in any style, however I prefer it to be written as follows: 

By writin…

Dealing with Selenium WebDriver Driver.Quit crashes (Where chromedriver.exe is left open)

We recently came across a problem with Selenium not quitting the webdriver and this would then lock a file that was needed on the build server to run the builds.

We were using Driver.Quit() but this sometimes failed and would leave chromedriver.exe running. I looked around and found this was a common issue that many people were having. We (I say we, as we came to the solution through paired programming), came up with the following, that would encapsulate the driver.quit inside a task and if this task takes longer than 10 seconds, then it will clean up any processes started by the current process, in the case of the issue on the build server, it would kill any process started by Nunit.

        public static void AfterTestRun()
            var nativeDriverQuit = Task.Factory.StartNew(() => Driver.Quit());
            if (!nativeDriverQuit.Wait(TimeSpan.FromSeconds(10)))

        private s…