Monday, December 24, 2012

I Want To Use An Open Source Test Automation Tool....But where do I begin?

Revised:  03-29-2013

Revision History
12-16-2012CreatedHelp testers find tools they can use to learn advanced test automation.
03-29-2013Nice to Haves Shopping ListAdded information regarding ThoughtWorks Go, a tool to support continuous delivery of code.
03-30-2013Nice to Haves Shopping ListAdded information about Xampp a combination of tools to support TestLink

I have been following various discussions on various sites regarding test automation.  Some questions that seem to keep reappearing by various users are:

“Where can I find training for Selenium?”
"I want to begin learning test automation, what do you suggest?”
“Where can I find documents for Selenium?”
“How do I begin a test automation project?”

These are good questions usually asked by veteran manual testers, those new to testing or even developers seeking opportunities in the testing field.   I have had the pleasure of working with quite a few people both new to testing as well as new test automation.  So these questions do make sense.  For us seasoned folks who live on Google searching for the answers out there, some of us (not all) might think “why can’t these guys do the same?  I taught myself.  I found the answers why can’t they also use Google?” Those questions are valid but the truth is we all learn differently.  Some can be self-taught while others need a bit of hand holding.  I had the advantage of sort of growing up with these tools and used tools from QA Partner (pre SilkTest) to Selenium WebDriver.  Believe me, it took years before I could get grasp on how to properly leverage these tools.  So the suggestions presented here are from my own trial and error efforts teaching myself as well as teaching others.

This article is for you my new little test automation newbies.  It will address, first and foremost what tools you can use, what they are generally used for and where you can find them.  We will keep it limited here for now.  It will probably take you some time to download, acquaint yourself with these tools before we move onto another article of actually using them.

Open Source vs Commercial

For this article, we will focus on using open source tools.  Why?  Because we can download and legally use them free of charge.  This basically means no matter where you work, what your job is or where your career moves you, thus far open source tools will provide you a cost efficient way to learn the craft and transfer the knowledge and skills to other tools.  If you have a desire to learn a commercial testing tool like QTP or TestComplete which are hot on the market right now, you will have to purchase a license (I don’t want to know if you have other methods of acquiring the tools).

Understanding the Matrix

I have divided the tools into three categories and provided information to help you get started.

The Matrix of Learning

First, do not be intimidated by the girth of information.  I tried to organize it to help you pick a starting point and as you build your competency continue further down the lists to add to your skill set.
Each table provides you the following:
  • A tool you can use as part of your learning process
  • Where you can find the tool (I tried to find only community or open source versions)
  • What the tool is primarily used for and how it supports test automation.
  • A summary of the benefits it provides you in your automation endeavors.
  • A summary of the key things you can target to begin learning the tool.
I provided three categories of tools which are:
  • Must Haves:  To begin your learning endeavor, these are the tools you must have in order to effectively begin learning and developing solutions for test automation.
Note:  Before you dive off into the Additional Support and Nice to Have lists, many of the tools recommended can actually be downloaded and immediately plugged into your IDE (Eclipse or IntelliJ) for example.  So although I provide you with the links where you can find the tools and learn more information, you can alternatively use your IDE to plug the tools in.
  • Additional Support:  These tools can help you with building, managing and distributing the automated test solutions you have built.
  • Nice to Have:  Once you have gained a level of proficiency and confidence, you can begin experimenting and innovating your own ideas using the suggested tools.
By all means, this is not an end all list!  There are other tools out there that you can use for both your learning and test automation development.  Check out this list of test tools the Ministry of Testing has compiled.


Must Haves Shopping List

Tool Location Purpose Benefits What to Learn
Sample web application Sample Insurance App A simple web application you can use for your test automation self-training
  • Well you need something to work with
  • How to test application using automated techniques
Selenium IDE Selenium Downloads A test record and playback tool that plugs into your Firefox browser.
  • Can help you begin understanding the flow of the application under test
  • Can help you with object recognition
  • Can convert the recording into your desired programming language which helps you begin learning how to code
  • Basic object recognition
  • Basic programming
  • Basic testing
  • Basic test script management
Selenium WebDriver Selenium Downloads Drives your test against a web application under test.
  • Use your choice of language, Java, C#, C/C++, Python, Ruby, etc
  • Simply an API that can be incorporated into a development project and used with other APIs or libraries to extend automated testing capabilities
  • Using the Selenium API methods
  • Write code
  • Follow a disciplined coding practice
An IDE like, Eclipse
Visual Studio
Eclipse Downloads
IntelliJ Downloads
Visual Studios

Provides a development environment for code development and management.
  • The editors provide auto-correction and suggestion tools
  • Can link easily to other tools like code management, build, task management, issue tracking, etc.
  • Make is easy to add libraries that provides additional functionality to support TestFramework features
  • Provides tools for running and debugging your code.
  • Installing the IDE
  • How to include new libraries
  • How to create a project
  • How to create a package
  • How to create a class
  • How to create methods
  • How to develop class constructors
  • How to use debugging
  • How to use the console
  • How to make connections to other applications like Bugzilla, Subversion, Maven, etc.
Java JRE Java SDK Java Downloads JRE is required to run java applications and applets while the sdk is required for development.
  • Provides libraries for java development
  • Provides tools for compiling and running your tests
  • Install java
  • Edit environment variables
  • Edit ClassPath
  • Run your java projects
  • Run your java –jar projects
  • Compile your projects
  • Write batch files to automate the java calls
TestNG TestNG Download An API to help handle your tests and provide tools for reporting
  • Flag functions as test cases
  • Develop test result reports
  • How to target and drive functions as tests
  • How to develop and test reports


Additional Support Shopping List

Tool Location Purpose Benefits What to Learn
Ant Ant Download To build and run your tests
  • Provides tools for copying configurations, libraries from your source to your target/release folder
  • Can compile and build your project
  • Can target results to a common folder
  • Can run your tests
  • Install/Unzip Ant
  • Edit Environment variables
  • Write build.xml scripts
  • Run test projects using ant
Maven Maven Download Getting Started with Maven

  • Can be integrated and managed by your IDE
  • Can manage code dependencies for your project
  • Can automatically download and install updated libraries to your project
  • Can use to manage the building and compiling of your project
  • Plug into IDE
  • Use IDE to build, compile, package your TestFramework
  • Create/Copy and edit a pom.xml file
  • Add executions and dependencies to your pom.xml
  • Edit to retrieve updated libraries


Nice to Haves Shopping List

Tool Location Purpose Benefits What to Learn
Spreadsheet: OpenOffice
MS Excel
MS Office Package
Can use it to create and manage test cases and/or testcase data
  • Easy to create and edit test data and/or test case data
  • Basic data entry into the spreadsheet
  • Save data as CSV
  • Access data via ODBC
JUnit Junit Downloads An API to help develop and handle your tests and provide tools for reporting.
  • Develop and organize test scenarios
  • Produce test reports
  • How to include into your development project
  • Write and run unit tests (it is useful to also unit test your TestFramework, see the article Unit Testing the TestFramework
Subversion SubVersion Client and Server Downloads A code and version management system.
  • Allows for the storage and management of your code
  • Collaborate with other peers also learning test automation
  • Install/configure server
  • Install and attach client to server
  • Building a local code repository
  • Synching local code repository with subversion
  • Committing changes
  • Rolling back changes.
Bugzilla Bugzilla Download An issue/bug tracking system.
  • Manage issues found in the sample product
  • Manage issues found by automation
  • HAVE FUN!  Think of ideas on how it would fit into your test process (see Automated Bug Reporting for more ideas.
  • How to capture bugs
  • How to report them automatically using your IDE or TestFramework
  • Think of ideas for issue tracking and linking to other test artifacts.
Google Charts Google Charts documentation, samples and api Provides HTML code and libraries for developing glam reports.
  • Creating glam charts for your test reporting
  • Creating glam reports for test reporting
  • Anything else that proves useful with your test automation efforts.
GraphWalker GraphWalker Downloads A tool for Test Case Modeling.
  • Creative way for developing test cases
  • Reducing test case development time.
  • How to create test case models
  • How to link them to your TestManagement process.
  • How to link them to your TestFramework.
  • How to leverage your keyword methods with test case models.
Firebug Firebug downloads Can be used to inspect and manage the XML documents of your web pages.
  • Use to help discover object identifiers
  • Use to interrogate the web page.
  • Plug into Firefox
  • Utilize the features
  • Automatically capture identifiers and paste into your code.  See XPath to the Limit for more information.
Java Robot Java Robot Download Can be used to support test automation features not supported by Selenium.
  • Support automation of java technologies not supported by Selenium
  • Take advantage of the screen capture features.
  • Include library into your development project.
  • Be innovative in what and how you use it.
Apache Commons Apache Common Download Use it to send email notifications from your TestFramework
  • Send test results automatically via email
  • Include library into your project
  • Get sample starter code for the type of emails you wish to send
  • Coding a method that will gather test results and format to an email
Xmapp Xampp Windows Download A product that provides all of the supporting tools for running the TestLink Test Management tool.
  • Provides Tomcat, Apache, Mercury SMTP mailserver, A MySQL database which supports the running and ussing TestLink
  • Install XAMPP
  • Server management and configuration.
TestLink TestLink Download A test and requirements management tool.
  • Manage your requirements/use cases
  • Manage your releases/sprints
  • Manage test cases
  • Manage test results
  • How to use test management
  • How to use the web service api for the test management system
  • How to incorporate test management as part of your overall test effort.
AutoIT AutoIT Downloads A windows test tool.
  • Perform windows based testing/actions not supported by Selenium
  • Create test scripts using AutoIT IDE
  • Compile scripts
  • Include calls to scripts within your TestFramework
  • Pass parameters from your TestFrame to the AutoIT script
soapUI soapUI Download A testing tool for web services
  • Perform more comprehensive web service testing
Log4j log4j Download A library useful for the development of logging programming events.
  • Structured way to log test information and results
  • Structured way to log TestFramework activity.
  • How to create log templates
  • How to incorporate (if not already) into your project.  Should be included with java
  • How to code logging routines
ThoughtWorks Go Go Download A product that supports continuous delivery of code to be released.
  • Supports agile projects to continuously develop test and release to any desired environment.
  • Install and setup Go and Go Agents
  • Connect Go to a code management system, i.e. SubVersion.
  • Understand the concept of a pipeline
  • Deploy changes to an environment.
  • Develop and link automated tests to pipeline.


I hope this article will help you to get started on your test automation endeavors.  One thing I have often told colleagues and students of mine who were beginning to learn test automation is to “free your mind” in the infamous words of Morpheus, “FREE YOUR MIND!”  When you work for a company, yeah some things we wish to do are restricted by others and politics.  On your own you can take advantage and design and develop whatever you wish to develop and automate as much as of the testing process as you like!  FREE YOUR MIND and you will be able to discover innovations that may one day lead to greater things than simply running tests.


A short story


Please allow me a little indulgence here.  I worked for a financial company that had approximately forty applications within their portfolio of products and they had five environments, one production and four test environments.  My challenge was to develop a test automation strategy which should in time, reduce the need for consultants, proactively test on demand, automatically notify stakeholders of potential problems , run in all environments, and if that wasn’t enough save the company money.  Of course they want to do that!
It was a heterogeneous environment of web applications, DB2 databases, information buses, mainframe applications, window applications and some cloud based applications.  I chose Selenium as my tool of choice because approximately eighty percent of their applications had a web client or a web service api that could be leveraged.  I augmented Selenium using AutoIT, soapUI, Apache, SpiraTeam for TestManagement and other library of tools (some not mentioned above)  to support what we desired to do.  I was successful in my endeavors despite the politics and negatives that screamed things like “it is impossible to do what you are suggesting.”  If they didn't think it was impossible then it was insecurities that provided the obstacles.
I built the following support into the TestFramework:
  • On notification of an updated environment, evaluate the sprint in SpiraTeam.  If the sprint had not ended, create the release in the sprint, if it had ended create a new sprint then create the release.
  • Automatically create the test runs for each environment in SpiraTeam.
  • Download the keyword–driven based tests from SpiraTeam by testsets.
  • For each environment, run the tests for smoke, system, regression, and integration testing.
  • Automatically email the testers of the results for each environment.
  • Automatically email system owners with problems related to their systems or the test environment manager if there was a problem with one of the machines in the lab.
  • Display test results via graph on a large display monitor
  • Display a summary error log on the large display monitor alerting to problems in environment or problems with the test run.
  • Send test results with supporting information and screen shots to SpiraTeam
  • Automatically create defect for failed tests.
All of this and more can be accomplished.  It depends on the unique challenges you are faced and how test automation can help you meet those challenges.

My final words are to have fun with this!  Remember to FREE YOUR MIND!  No one solution is the end all answer.  No one methodology is the end all answer.  There will be elements of your innovations that you can carry with you for a long time (One TestFramework I built using SilkTest lasted about 12 years) and others well it is time for them to be put out to pasture get a face lift (I upgraded the SilkTest TestFramework to Selenium WebDriver)!  Never feel shy about embracing something new and challenging!

Happy Automation!


  1. Hi Margaret,

    Its really a Good article.
    Can you tell me is it possible to report bugs automatically using Selenium IDE or Selenium TestFramework

    1. Yes Anand. You can perhaps review the article Automatic Bug Reporting to gain some ideas on how. Selenium does not have a facility natively to do this but you can use a combination of Selenium and Java or whatever programming language to accomplish through a web service call to your bug tracking system.

  2. I actually enjoyed reading through this posting.Many thanks.

    1. You are most welcomed. I am glad you found it useful!