Sunday, February 10, 2013

Which Way Do We Go George … to select a test automation tool

I have seen this question raised several times during online group discussions.  It is a very good question to ask prior to actually choosing an automated test tool.  Once people have an idea of tools to review, perhaps the popular ones mentioned during group discussions, and then people would like to know what are the tools pros and cons.  This is also a valid question to be asked.  Before one begins choosing a tool to begin a new test automation endeavor, they should do due diligence to select the best tool(s) for the job and the purpose.

Due diligence can take one quite a bit of time to properly assess the need and then research the best tool for the job.  I mean there are so many out there that some may become a bit befuddled and disoriented with the mass selection of both commercial and open source products available on the market today.  So George, which way do we go?

The First Question

Are we choosing an automated test tool for professional or personal reasons?  The job market is screaming for test automation currently.  This trend is encouraging many testers of all types to include it as part of their skill set.  Hey, as long as we need employment, we need to find ways to make ourselves hirable which means we need to sell the right skills companies are interested in.  Therefore, if you are choosing a tool in order to begin self-training, may I recommend you review the article I Want To Use an Open Source Test Automation Tool....But where do I begin?  This article can help you begin with choosing one of the hottest open source tools on the market today, Selenium, and puts you on the path to what you need, why you need it and what you should gain from it. 

Regardless if you are choosing for professional or self-training reasons, feel free to review this remaining article to hopefully guide you to selecting a tool that will fit your needs.


The Checklist

First George, you might feel the need to create a checklist to help you in your search endeavor.  Google is a wonderful tool to locate a good checklist.  Once you find one that looks reasonable, modify it to address your unique needs.  Make sure to prioritize must have requirements and view all other features as an added bonus. Here is a nice whitepaper called Test Automation Tool Evaluation by Clara Abraham that offers some guidelines to help you begin your research for the best tool(s) that will meet your needs. 

Don’t Believe the Hype

Every time I see this question posed in group or forum discussions, five million, five hundred and ninety thousand people respond cheering for their tool of selection.  If that isn’t enough, test tool vendors are bullhorning how their tool solves all of your problems!!  Don’t you believe it!!?  Do your own homework.  Do a proof of concept.  As you narrow down your choice, yeah start a discussion that will help you learn the pros and cons of the tools you are considering.  There are so many people who love to share information, take advantage of it!

From My Own Medicine Cabinet

In addition to help you through your discovery period, here are some things I can add as food for thought during your test tool selection process that I have learned from my own school of hard knocks:

  • First things first.  Can you or can you not obtain a budget to acquire your test automation tools?  In this “I don’t want to pay for a !&?# thing era” you yourself may not have the budget to buy commercial tools or your company may not allocate the funds.  If there is money, hooray!  Either way, this will help you define the scope of tools you will begin reviewing.  No money, no need to review commercial tools.  If you have money, by all means take them into consideration.

  • Second, evaluate the type of testing you would like to perform such as, performance, mobile, SOAP/REST, back end, GUI based, regression, functional, etc.  This will help you identify tool(s) that will support this type of effort.  Evaluate what type of testing is most important to you!  Remember, no tool will give you 100% of what you will require.  For example, one tool I know will effectively perform all of the above-mentioned test types but not performance or mobile.  There you may have to consider a different tool.

  • Third, evaluate if the tool allows you to innovate and invent your own solutions.  Often, we may have requirements that the tool will not natively support.

  • Fouth, evaluate the type of technologies behind the applications/systems you wish to test.  You want to make sure the test automation tool will support the version of technology you are using.  For example, a commercial based test tool I worked with could effectively test most of the popular technologies but not Java FX2.  If this was a high requirement, this awesome tool would basically be worthless to me.

  • Fifth, what type of skills does your current test team have?  Do you have access to a developer or does someone have development skills?  This plays a huge role in your decision.  From experience, commercial tools for newbies are far easier to learn and get some type of return on investment sooner than open source tools when the test team does not have access to a developer who can at least mentor them to get started.

  • Sixth, how much bang do you want to get out of a tool.  Choosing a commercial tool like QTP will not only give you test automation but also provides simpler ways to create user-defined keywords, provides great reports, can integrate with other 3rd party tools like Jira or test management systems, usually offers nice wizards or tools for handling test data or retrieving data from databases.  They are often quite robust with features that support a variety of technologies, platforms and test types.  With open source tools, they often are single-focused.  They are limited in the technologies, platforms and test types they support.  One must be prepared to supplement things like reporting and test environment management with other tools or create them yourself. 

  • Seventh, how much synergy does the tool have with other 3rd party tools like bug tracking or test management.

  • Eighth, testers and/or test automation developers should choose the solution.   The choice should not be made by managers who only pay for the tool.  By the way, be prepared to help them understand test automation is not a magic bullet.  Any test automation tool you choose to acquire will be an investment of time to learn and begin producing something of value.

  • Ninth, evaluate also how the tool can help with the overall testing process, not just executing regression tests.

  • Last, never ever fall for a sales pitch.  If you decide to go with a commercial tool, try it before you buy it.  See if they provide support to get you started so you won’t waste time during your POC phase.  The two main things I look for in a tool are how intuitive it is to learn and what challenges do I face testing MY application.  Not there demo application, MINE!

Other Good References


It may take you some time to do the homework but in the end it is worth it.  I can share one final note.  Test automation can be quite challenging but also a lot of fun learning and figuring out how to utilize it to help with the testing endeavor.  EVERY tool has advantages and disadvantages.  Don’t allow someone who is passionate about their tool of choice influence you.  I have learned depending on my testing challenge, one tool of choice can be more than sufficient and other times I was Dr. Techenstein finding all sorts of spare parts to develop an automated technique for a particular challenge that can easily plug-in to my TestFramework. 

Happy Automation!

Margaret Thomas has worked in the testing profession since 1996 and worked within a number of industries including mobile, finance, medical and insurance.  She developed a strong desire for automated testing when she began working within finance and has utilized a number of testing tools both commercial and open source.  She is also the author of the technical blogs found at

No comments:

Post a Comment