Tuesday, January 8, 2013

The Battling Test Automation Camps


There are definitely some very passionate souls out there debating the issues of
* what to automate
□ how much to automate
◊ who should automate
○ who should own the automation  

People such as James and Jonathan Bach and others, who are awesome lecturers and educators in defining and terming test techniques, can add to the battling camps of automation versus manual or artificial intelligence versus sapient.  If that wasn’t enough fuel for the fire, you have edges versus test step descriptions or test case scripting versus exploratory checklists or modeling.  It’s like a techy version of the Hatfields and McCoys!!!!  Everyone is entitled to their own view but this overwhelming battle has made its way onto the working environments and interviewing arenas placing some people in an almost no win situation.  I will touch on this topic later.

Boxing Test Automation

One thing that tugs my frustration strings are those testing advocates who want to put test automation in a little box and dictate its limitations by making statements like tools are only script checkers, they only do what you tell them to do or you can only use them for regression testing.  I am going to step out of the norm of refraining from a personal opinion and say that is because they lack the insight and imagination to truly maximize the potential that test automation and other tools can offer.  I agree with leading test advocates that testing is a sapient activity one that is in my opinion, philosophical, scientific, intellectual and creative in its nature.  Testing should and must go beyond the ‘if then else’ flow of the code and this demand should influence test automation.  When you combine the intellectual and analytical input of a tester with the agility of test automation, endless possibilities occur.

The ToolBox

In my career, I have worked for a variety of industries, working with different kinds of products using different development methodologies both traditional and agile.  Each opportunity introduced so many new challenges, techniques and terms and methods they became cyclical in my head…techniques, terms and methods oh my!  Many of us are sitting in the eye of the storm struggling to do our jobs day to day can become overwhelmed with ideas and opinions especially regarding test automation, test management and other related topics.  Check out the LinkedIn discussions.  One simple question can result in a firestorm of responses that can often overwhelm the original question asked! 

I say damn it all to hell!!!

Stimulate your own brain juices to decide what it is you truly want to accomplish and then leverage all of the great information and misinformation to decide what tools you require for your testing toolbox that will help you accomplish our desired goals.  If you are part of a team that will make this decision together, then you will need to provide supporting information to what you will add to the toolbox.

Test automation is not just about picking a tool that will let you either record some or program actions and play it back.  It is about leveraging all the tools you have available to you to help you manage the development process, manage the testing process, keep people informed, manage current and historical data, eliminating the waste of performing routine mundane tasks, helping to mitigate problems and yes of course test execution.  Think about the simple activities surrounding setting up for a test run.  You have to check if a build is ready, download if it is, perhaps clean the machine, install or update it, license it, start it and then perform a smoke test to see if it is ready for testing.  Do you know how much time these activities take if they are not a part of the requirement or user story you are targeting for testing?  Why not have an automated method perform these tasks for you and alert you that a new build is ready and verified for you for testing!  Come on even die hard manual and/or exploratory testers can appreciate that!


So Who Owns Test Automation?

That is an easy answer.  The team!  Regardless if it is an agile or traditional team with one or many people, the team owns it.  The team’s goal should be to mature and maintain their toolbox and the solutions they have developed using their tools.  There should be collaboration on how test automation will support the overall development and testing process.  The team should always proactively seek updates or new tools to support future needs.  Last, the team should ensure they code and version manage their efforts to ensure they have stable tests to always support their test effort.

Who Should Develop Test Automation?

Oh my this is another interesting debate!  Some argue it requires developers while others say testers can do automation.  Although I am a tester who can develop advanced test automation frameworks, I have been on the side of the interview table desperately seeking testers with some development skills.  
People have solved the dilemma by hiring testing specialist with a background in test automation or programming, developers with a background in testing, or create a collaborative union between developers and testers.  Regardless of how a solution is discovered, the individual or union of individuals should have the following skills to contribute to the development group:

  • Experienced with testing fundamentals such as planning, test environment planning and maintenance, test development, test execution, bug and test reporting within the preferable development methodology such as waterfall (yes there are still people using waterfall) or agile.
  • Experienced with quantifying and reporting test results.
  • Experienced using either commercial or open source tools.
  • Experienced programming preferably using the desired language, i.e. Java, Python, C#, C/C++, etc.
  • Experienced managing the test process preferably within the desired methodology such as agile or traditional.
  • Knowledgeable using management systems such as code, build, test, application life cycle, project, etc.
  • Experienced with identifying and selecting the right tools and processes that will contribute to doing the job and leveraging the skills available to utilize the tools.
  • The willingness to learn and mature their own knowledge to support growing and maturing development endeavors.  
  • Possess imagination to innovate ways to improve and support the overall development process.
As long as the individual(s) or team possesses these skills then you have the right team or group to support not just the automation effort but the overall development and testing effort within a given project.

A Tale of Interviewing Dilemma

In the past few months of seeking new employment after my last role ended, I have had the craziest interviewing experiences of my life!  I would invest time, energy and money going to 2 or 3 interviews for several companies and the reasons why I didn’t get the job were just astounding!  The reasons fell into the following categories:

  • They did not like ONE technique used for managing the testing process.
  • I wasn’t enough of a developer although I was applying for a manual testing role.
  • I had TOO MUCH test automation experience.
  • I didn’t have enough programming experience for test automation (although I have single handedly innovated and developed test automation frameworks since 1997 using both commercial and open source tools).
  • I was too agile and they required someone to be both traditional and agile
  • I was not agile enough.

In seemed the interviewers were forgetting the basics.  They should seek someone who has the fundamental skills and knowledge for the job and of course if they seem to be a good fit for the team.  It is awesome when someone can also bring with them the experience and skills that made them successful in prior roles but they most certainly will have to learn to adopt new techniques to fulfill their new role.  One example would be, to adopt new methods and techniques to support agile test efforts.  No one comes pre-packaged with 100% of what the interviewer would like to have for the role.  However, it seems that is what interviewers would like.  I laugh because I remember for one particular interviewer who required someone who had Windows 8 experience.  That was the most important requirement that needed fulfilled.  Windows 8 was just released.  I thought to myself so it is more important that a person has this one particular knowledge of an OS and not a person who has demonstrated the ability to acquire the knowledge and skills needed to do the job…what happens when Windows 9 comes out….that person is no longer qualified because we have a new OS?  As you can see, I was frustrated meeting people who didn’t know what they wanted, who truly didn’t examine a person’s background before inviting them to an interview or simply seeking compartmentalized people components they could plug in to do single tasked software factory work.

At the end of the day we need a job and these people have the keys to giving us a job so please use this information to help you better prepare for the challenges of today’s interviewing requirements.

How Much Should be Automated?

Honestly, if I could automate a rabbit jumping over a leprechaun’s rainbow I would :).  Today, we can leverage many tools to assist our automation endeavor.  We are not limited by our choice of one test automation tool.  In the world of APIs, web services and open source solutions we can leverage many tools to help us automate even the most challenging of things to automate such as real time data, video streams, voice streams, complex calculations, object metamorphosis and more.  If it can be programmed, it can be automated.

NOW WAIT before you guys start the screaming and crucifying me!  

This does not mean automation should replace the intellectual abilities of a tester, NO NO NO NO!  The tester is an important part of building good and robust test automation.  The tester is key to managing the changes our automation must take on build after build and release after release.  The tester is imperative to ensuring test automation is routinely testing high usage features and ensuring if it is not meant to be broken, it shouldn’t be.  

Test Automation is a tester’s extension of his knowledge of what to test, when to test and how much to test.  Test automation should allow the tester to regularly and continuously identify more clever ways to test instead of over-investing his time to performing repeated manual tests of an application.  Last test automation helps the tester verify processes and functions not reachable by a client interface.


I am a true advocate of different things work for different people, different companies and different situations.  Although I am a true test automation advocate, it is not my role to force my ideas on someone else.  I love the online discussions when people ask advice on what tool they should use to support an automation endeavor.  Tons of people and vendors respond giving their biased opinion of what the person should use because they use it or trying to sell it.  It is even funnier to see those people who will respond disagreeing and proudly saying no don’t use that, use this I prefer this tool it’s better.  I love this article by the Software Testing Club and The Ministry of Testing list of test tools.  Input is always great!  However in the end, regardless of which camp you prefer to join, it is up to you to choose the right tools you require to support your job because even in the manual camp, you are using some type of tools to help you with your job. 

Happy Automation!


  1. Hello, I have been a fan of your blog from few months now and everything you write is so very interesting, whether you comment on LinkedIn or you log it in such impressive blog. Just wanted to let you know, I share the same experience you do and am from Automation world as well, implementing something for a new company I have joined and inclining towards Ranorex Tool. Thanks for all the wonderful analysis, comments and sharing your rich experience.

    1. Srilatha thank you so very much for sharing your experience and thank you for taking the time to let me know the blog has provided some insight to you. Your words have given me the biggest smile today :). I have not had an opportunity to use Ranorex. Can you please share a little more about it from your experience with it? What type of automation do you do with it, what technologies does it support and do you have to augment it with other tools?

  2. Hi
    This is more interesting to read the file with various environments. Automation testing is favourable for any time but we have to choose the apt time to continue with. Even if the traditional setup is just behind the tools we can assure the end user that every tjing happened in front of my eyes.

    1. Thank you for your feedback Kaushik. I must admit I am not sure I follow you but I support different groups of people in different organizations support their test process in a way that meets their requirements. It seems organizations that have a responsibility to external partners or governing bodies, they have a stricter Quality Assurance process that require their test efforts to result in tangible evidence of quality presented in a strict format. Niche software companies often do not have such a strict Quality Assurance process. When I supported the software testing effort in these type of strict testing environments, test automation was embodied into the overall test effort which combined ad hoc/exploratory methods and formal user acceptance testing with test automation. This synergy allowed us to both test as wide as required yet still generate the required output our partners/customers/governing entities required. Test automation, as we implemented it, allowed testers to continuously add test cases to support uncovered permutations (without always requiring an update to the framework) as well as serve to verify existing/unchanged functionality still operated as expected. We also leveraged test automation to perform test setup and tear down as well as support updating the test management artifacts.

      I don't know if that is a correct response to your comment but I do appreciate your feedback and your time to read this article and as I said, different processes work for different people. In a way I have been lucky that I always worked for organizations who trusted the test automation strategy I presented and it always supported the testing strategy.

  3. hi Mag, You are most welcome, everything you have written so far is very well thought of and put in words, that I can tell by reading :-).

    Now, just to answer the Test Automation comment from Kaushik, if any company that has at least 30% or more stabilized code (meaning, not many major UI changes done to the production code anymore), it is very important to implement automation that covers the base regression to start with. Later on, we can also embed automation wherever there is high precision, repeated mundane tasks or complex time-consuming scenarios that will save time and energy/effort for manual QA testers who can cleverly/intelligently think and test the system in different ways possible that machine may not and find more critical defects in the system. A company will be successful with time to market and the agile challenges only if its IT implements a healthy blend of Manual and Automation Testing.

    Mag (hopefully, you are ok with this alias), I have evaluated plenty of tools so far for my company and currently, evaluating Ranorex but it stands out in every aspect of Test Automation. We are all microsoft shop, have the SOA implemented ESB service bus services, Desktop apps, Web apps, Windows services, WCF services, data input from sqlserver, C# and .net platforms, soap ui testing, lot of non-web testing so it rules out plenty of tools in the market for me to select from. Ranorex so far has proved unimaginably on the top of the list, still under test though, we have until this end of the month to decide and buy the tool and start our Automation efforts. It handles with ease whether it is simple record/play and then building the framework around it to customize the repeatable, robust code (or) using the code-driven testing without using record/play option. Easy data driven options, Easy keyword driven,usage of .net libraries directly interfacing with Visual studio (like you said in your blog - if it is programmable, it is automatable :-), so true) so, I can be closer to developers and understand the easiest way to automate any interface. Imp to mention, many tools have failed to recognize the objects and Ranorex, with its x-path and true object identification, can really identify just about anything in simple spy tool or recording, I really like it so far. It does within recording, not only image/video checkpoints but also identify the files in href links and open them and I can give a baseline copy to compare the files to ensure the links, files in the links and content in the links are all good. Just mentioned few things I could remember, it can do much more am sure with webinars I have attended. Please try it when you get a chance, you have pretty rich experience and can easily comment on it once you use it. Thanks.

  4. Hello, I have written a big feedback on both comments above and looks like somehow lost the whole thing, the comment never got posted here :-).

    Well, in short, Test Automation is mandatory for any company that has a stable code and mundane repeatable tasks, high precision and complex/time-consuming tasks.

    Mag, Ranorex is one of the several tools I was evaluating and found it very very interesting and useful, simply, it works for both web and non-web. Its features are very rich and unparalleled, like it so far and I would be really amazed if it fails me during evaluation or if I don't end up selecting this tool :-).

    - Sri.

    1. Hi Sri,

      I read your feedback. For some reason it didn't post to the blog but I got the email.

      I definitely agree with your observations. I have used automation even when the product wasn't stable. The model I used for the framework allowed me to make changes to accommodate the changes rapidly and often the test cases remained the same because they were developed to ensure the product met the proposed requirement. If the requirement changed then the test case was modified to accommodate it. In other words, I leveraged TDD for test case and test framework development. However, I most certainly agree with your "healthy blend of manual and automation" concept. I found we performed a lot of the required testing while developing the automation.

      My hypothesis is many people who are against automation both fear change AND have not had an opportunity to work with someone who can develop robust and resilient automation. Today's complex environments working with agile development demand automation. As you said, we have to help the testers have more time to do their jobs and help them test parts of the system not reachable by manual testing techniques.

      I will be honest Sri, I have always worked as both a tester and a test automation developer. I have never had the luxury of just doing one or the other. For me, test automation was about survival in the beginning. When I began test automation, I was 1 QA person to 20 developers and 8 financial products. I developed Rapid Application Test System which combined the manual test efforts with automated. This system ran automatically for every new build allowing me to have free time to evaluate and explore new features and then incorporate the new tests and feature support into the test automated framework. I realized there are many testers who have not had that experience and perhaps this is why they cannot bridge the idea of the two concepts working harmoniously together.

      Ranorex seems interesting. I took a few moments to review their website. Thanks Sri for the feedback.

  5. hi Mag, I have read this comment, just waiting for sometime in my schedule to reply on. Currently, kind of busy with selecting the right tool and presenting it to my leadership with the demo. Have a good day.

    1. Hej Sri!

      No worries at all. I appreciated that you took the time to comment and it made my day to learn someone found the articles useful....Please feel free to write when you can and if you desire to :). Good luck with your tool evaluation.

  6. Hello Mag,

    Your experience in the beginning of automation is very impressive 1:20 :-) such odd ratio. Good to know someone with so broad and rich experience in all areas of Automation, and I can imagine people who have had so much trust and confidence in you to build the nightly runs without human intervention all by yourself as a resource.

    Yes I had a chance to evaluate plenty of tools for the best fit and recently, model based approaches like worksoft certify and TOSCA tools have grabbed my attention. Looking into them as well before I make a wise decision for what is good for the company. If you have an opinion on these two tools, please do let me know.

    These two tools have a very different paradigm to them being object based programming instead of coding. "Do not write a program to test a program" seems like their motto :-)

    I also had a chance today to read your other comments on LinkedIn as well.

    1. Hej Srilatha...good to hear from you! Yes I follow your comments on LinkedIn too :).

      TOSCA presented their tool at the StarEast Conference last year and I was not impressed with it mainly because the presentation was all smoke and mirrors. They never showed the actual tool and nor was it available for preview. That type of marketing I do not like. The model you describing however seems very similar to what I believe IBM MessageBroker does where it is object based programming.

      My experience was birthed in a time where people really didn't know how to maximize the use of a test automation tools. Often the tools became shelfware because of the learning curve involved. Working as a lone tester or with a very small team with odds such as multiple financial type systems and out numbered by developers, we had to develop a lean testing strategy that maximized the use of tools and automation to support not only test execution but the overall testing process.

      The lean test approach used (prior to agile) leveraged anything I could use to support reducing waste and went beyond the automation tools. The automation methodology was strategically designed to minimize the need for maintenance and redevelopment of the framework hence allowing us to invest the little time we had in refactoring and updating. I have used this methodology, since 1999, with 6 different automation tools refactoring to maximize a given tool's feature and of course maturing to support new innovative ideas.

      This is how I/we survived with the odds against us.

      During my last round of interviews, I learned that I was an oddity to many as people have teams of testers with specialized testing skills. However when I grew up doing testing, companies were not making that kind of investment so my testing for me was from requirements management to test environment management to release management as well as customer support and user acceptance management. Looking back on those days, I realize now perhaps why I am a bit crazy and stressed :). I finally accepted a role as Test Manager where I will not be hands on but guiding people who are hands on so this will be a new venture for me to not have to be both the manager and the hands on tester.

      So that is a little about me. It seems you too have a wealth of automation experience and very experienced with multiple tools. I would love to hear more about you...how did you get to automation? what do you primarily work with? etc.