FIRST, LET ME SET THE SCENE
In this corner of the ring we have the testers and in the other corner, we have the developers. Both groups are from various agile development teams. Sitting in the center ring is continuous delivery! Ladies and Gentlemen it is beginning to turn into an orgy of heterogeneous agile and automation practices all trying to ride the homogenous pipeline process of continuous delivery. LET’S GET READY TO RUUUMMMMBBBBBLEEEE!!
LET’S GET A LITTLE SERIOUS NOW
Ok I know I am just being silly however it is the view I have of our company’s test and automation practices. The most intriguing discovery being a part of this fascinating team of brilliant people are the various practices towards test automation and not just by different teams but also how developers and testers approach test automation. It dawned on me that there is such a thing we can call “developer-centric automation” and “tester-centric automation.” This concept goes far beyond the idea of test automation. It really describes the desired approach of an individual or team to solve their test automation requirements.
Developer-centric automation borrows a lot of the same concepts from unit testing. The test automation suite is often approached with the idea that the test suite is part of the development project and actual tests are part of the java\main and java\test structure. Test cases are created like functions/methods (depending on the programming language one is using) and sometimes are supplemented by a developer type of testing tool for example like TestNG. Test data is often married to the test cases or passed in via xml or csv files. The developer-centric method definitely borrows the concept of using an IDE to develop the code, a code management tool like subversion to manage and share the code, a tool like Jenkins for running the project and capture results by leveraging the reporting facilities of tools like TestNG and/or log4j. Developer’s love stack trace type reports and can use these reports to not only verify the test results but also analyze the code’s behavior upon failure. Some developers are beginning to supplement their framework by using BDD tools like Cucumber which allows them to either collaborate with testers or provide a normal written view of the automated test cases.
There are pros and cons to everything and on the pros side, this method of test automation can easily be versioned and managed with the code that supports the application. If it is a java project then often it leverages Maven which helps to build and run the project as well as integrate into tools like ThoughtWorks Go, Jenkins or Hudson. It will often execute quite rapidly. It is less prone to errors and does a better job staying synced with the application under test. It is more adaptable to supporting both front-end and back-end testing. Finally, because often a developer at heart is creating these types of automated solutions they encounter less problems related to object recognition primarily because they can improve their code to make the recognition easier.
The downside of frameworks developed using this mindset is they lack true visibility and traceability to other test artifacts. They are often inflexible by nature which only offers very static regression test type of test automation. They can be guilty of non-reusability for other applications under test simply in the sheer manner that they are developed.
The tester-centric automation will take into consideration aspects of test management, traceability, cross-functional and platform support, and test function reusability. This method demands very detailed reports that outlines the test steps used for every executed test case and visibility of expected and actual results. It also places a huge value on screen shots during execution as well as the propagation of test status to other test artifacts such as user stories, requirements, releases, test sets etc. The sheer desire of how the tester prefers to approach automation is clearly visible in the number of test automation tool suites designed for testers which some developers would dismiss as a non-essential tool. Test automation tool suites provide functionality to allow the tester to provide visibility and insight into the automated test cases and their results. It allows them the flexibility of reviewing and analyzing results by test run, releases as well as baselines. It also has the desire to be very configurable allowing test execution on multiple platforms, version, devices, jurisdictions, OS, browsers and the list can go on. Finally, this method may prefer to maintain a library of mapped object names the test uses to interact with the front end of an application.
The upside is test automation is a natural part of the testing process. It can provide a wealth of information to the tester allowing him not only to analyze results from a given test run but perform comparative analysis across multiple test runs, configurations, test data, times and even light performance testing which can highlight problems in latency, responsiveness, and user experience. Everyone can have insight and visibility into the automated test cases and often understand the results of the test run.
The downside is often automated tests are not managed with the product under tests codebase which can lead to false findings due to the automated test suite being out of sync with the version of the system. Object recognition can prove to be quite tedious and taxing especially when attempting to do so from a blackbox perspective. Most tester-centric automation does not leverage tools to help manage nor share the test code. There is less of a chance to bring together front-end and back-end test automation support depending on which test tool you are using.
All of us desire to have our cake and eat too. We in the development organizations are no different. I found the best test automation frameworks taps into elements from both develop-centric and tester-centric methods. This hybrid approach combines all of the positive qualities while mitigating the negatives. I would argue the only con is finding individuals who can program like a developer yet think like a tester. My belief is the hybrids are coming via the roles of Agile Testers who work very closely with developers and they either possess the skills required to develop frameworks or they are positioned to collaborate with those who possess the skills as part of an agile development team to deliver the best of both world solutions.
AND THE WINNER IS…..EVERYONE!!!! Every approach has its pros and cons. Each approach is designed to fit the team’s test automation requirements. The approach is based on the individual’s or team’s idea of how test automation should support their effort. So this is my humble hypothesis. What is your opinion? What type of innovations and approaches have you had the benefit of participating in or witnessing?