Monday, December 24, 2012

TestCase Data Management



As both a software testing and test automation specialist, I learned data-driven techniques were the best method for automated testing.  Why?  The method contributes to code re-usability.  Well at least until we can process brain waves.  I don’t know that might be messy, I day dream :).  Anyhoo in the beginning, we learned we could use the data from the test cases we developed to do data-driven tests.   Over time, this proved to be a redundant effort because first the test cases were developed by a tester in Word or Excel and then an automated test case was re-created from the word or Excel form.  We eventually streamlined this effort by proactively creating test cases to be supported by either manual or automated test efforts.  Hence it killed two birds with one stone.  Now this is not a new idea.  Several commercial test tool providers provided templates with macros that would allow testers to develop test cases that could be converted to automated test scripts or test data records.   However, we still sought ways to eliminate the waste from the process.

The result of the invested effort supported both traditional and agile methods and supported any test type we elected to do either manually or automated.  We could even leverage this method to write a test plan intended for external company purposes.

If you permit me, I like to share the several techniques I have used for capturing test cases and the pros and cons of each method and then finally sharing the best practice method we elected to use.  First, let’s look at the testing requirements that drove the idea.

Test User Stories

I like to try and be open-minded so chime in if you have a better term than test user stories.  Coming from the traditional testing world, testers would outline test requirements from project requirements.  Hence I do the same with the agile user stories.  I derive test user stories from the project’s user stories.  Why? I couldn’t think of a better name but at times, one user story could spawn multiple test user stories with multiple tasks.  So please, if there is a better term, please share it.

Test User Stories
  1. As a tester I would like the ability to create new test cases that can be used as either manual tests or with the automated TestFramework.
  2. As a tester I would like to reuse existing tests cases as part of new test sets or new test scenarios.
  3. As a tester I would like to easily modify or retire existing test cases while maintaining their test-id assignments.
  4. As a tester, I would like to reuse test case information with the other artifacts for tracibility purposes that may exist in other 3rd party tools, e.g. Jira, TestManagement or ALM tools.
Now that we understand the tester’s requirements or as agile would mandate user stories, let’s explore the different techniques and see how well it can meet the requirements.

Service Layer

Before we begin exploring the test case data methods, the TestFramework will require a method of transporting the data to and from the TestFramework that we can refer to as either a middleware or service layer.   The service layer can be for example, a java class with a variety of methods to manage the data transport and organizing the collection of information the TestFramework will require in order to process.  The basic methods you might want to create are:
  • importTestCaseData
  • returnTestCaseResult
Basically, you are creating methods to support the information you are importing to process and information you desire to append to the test case or upload to your TestManagement system, for example.

Method 1 – Excel

Oh hell the mighty Excel!!!  Excel is an awesome tool for capturing and organizing data.  I found that it minimally met two of my requirements.  The method used was to:
  1. Create test scenario workbooks
  2. Structure the workbook to allow the tester to easily create test cases.  An example of how we captured tests are (of course Excel allows for more advanced formatting using forms and VB):
#DescriptionFunctionParmsVerificationParmsDependencyBug ID
0TestCase Description00
1TestStep DescriptionE_Method1,2,3….V_Method1,2,3,n….
2TestStep DescriptionE_Method1,2,3….V_Method1,2,3,n….

By the time our test case numbers reached the 1000 mark, this proved to be a clunky solution.  The test cases became harder to manage and making global modifications was beyond tedious.  When we cross-referenced this method to verify if it met our requirements, we found we were two test user stories shy.

Test User StorySupportedOverview
1XExcel allowed us to easily capture real test cases that could be used for both manual and automated testing.
2It proved cumbersome to reuse already developed test cases with a different test scenario often causing duplication of effort.
3XWe could modify and retire tests but it proved to be quite cumbersome when we began processing 1000s of tests.
4We could use a TestManagement system with Excel but then we had to build additional support for it.
The last downside we encountered was managing the test case ids in Excel.  For us it was a manual effort as we never had the bandwidth to learn how to make it automatic.  Last, we could export the data to CSV of course but it was easier to leverage ODBC for retrieving data.

Method 2 – XML

For this level of test case data capture, this was my least favorite method.  As a tester, it was very cumbersome to develop and maintain even if we used XMLEditor tools and for some testers it proved to be an intimidating format leaving the work on the automation specialist or developers.   We tried it but quickly abandoned the idea.  Our test cases looked like this:

<description>Test Case Description</description>
<description>TestStep Description</description>

Test User StorySupportedOverview
1XML was a bit cumbersome to capture dozens of test cases at a time using this format.
2It proved cumbersome to reuse already developed test cases with different test scenarios often causing duplication of effort.
3It was not easy to perform global modifications or retire tests.
4We could use a TestManagement system with XML but then we had to build additional support for it.

Method 3 – CSV

CSV files presented the same challenges as XML and why go through the trouble of manually creating one when you can use Excel to export to CSV.  Although one could export the data to CSV, we often encountered test run failures due to forgetting to re-export changes to CSV or run the macro that performed the operation for us.

Test User StorySupportedOverview
1CSV was a bit cumbersome to capture dozens of test cases at a time using this format.   If Excel was used to create the CSV file, we had to remember to export the worksheets to the CSV format.
2It proved cumbersome to reuse already developed test cases with a different test scenario often causing duplication of effort.
3We could modify and retire tests but it proved to be quite cumbersome when we began processing 1000s of tests.
4We could use a TestManagement system with CSV but then we had to build additional support for it.

Method 4 – Database

The database can be a nice option and proved to be a better alternative than Excel especially if you have someone who can design effective database tables along with good pre-defined sql filtering methods.
Test User Story
1XThe database option can allow for the easy capture of dozens of test cases.
2XOne can build clever SQL statements that will allow the reuse of test cases for several different types of test scenarios.
3XA database would allow for the easy modification and retirements of tests.
4XWith a little ingenuity, if a plug in doesn’t already exist, one could perhaps link the data to a test management system.

Method 5 – Other 3rd Party Tools

Many development organizations, especially those using agile processes, use tools like Jira and Greenhopper to manage user stories, tasks, bugs, product versions, etc.  It is only natural they would want to incorporate tests.  Like the test management system, one requires the administrator and perhaps a developer who can:
  • Customize a test case entry form
  • Build workflows to support the test cases
  • Build the necessary links to other artifacts.
  • Or scratch the above and buy an agile test case or test management plug-in to Jira.
The only shortcoming of this option is perhaps the additional effort, if required, to build tracibility and coverage which the test team should require and is readily available in TestManagement or ALM tools.

Test User StorySupportedOverview
1XWill allow for the easy capture of 1000s of tests.
2XOne could leverage the built in tools for version or backlog management to organize tests into test scenarios.
3XOne can easily modify and retire tests.
4XOne can build methods to easily link tests to other required artifacts of the process.

Method 6 – TestManagement/ALM

This is by far my favorite alternative.  The main reasons are
  • Many test management tools allow for customization of their pre-defined forms to fit your entire testing process and test case capture requirements as well as include the efforts of other stakeholders.
  • They inherently provide all (or most) the features a tester requires for managing and organizing tests into test runs.
  • Easy reusability of tests.
  • Can easily link tests to other project artifacts such as requirements, releases, sprints, test plans, configurations, etc even if these artifacts exists in a 3rd party tool.
  • One can leverage the test management reporting tools to quickly get test status, test coverage and other relevant information.
  • Can easily leverage the tools notification features  for reporting to all stakeholders.
  • Most importantly, it provides a “scriptless” environment to the test team allowing them to concentrate on what they do best, defining and executing tests.
It does require a bit of craftiness on the part of the test team and the automation specialists with perhaps the help of a system administrator to help customize the system to truly maximize its benefits.  It will also require the help of a test automation specialist or a developer to build a web service interface or leverage the test management’s bridge system to pass the test cases and the results to and from the test automation framework.

Test User StorySupportedOverview
1XAllows for the easy capture of 1000s of tests.
2XEasy to reuse tests within a number of test scenarios.
3XEasy to modify and retire tests without affecting the test run.
4XIntegrates perfectly with the TestManagement, 3rd party or ALM system.


There you have it, six alternatives for capturing and managing test case data that can be used with your TestFramework.   Now I know, sometimes cost is a factor when choosing a method for test case data capture and management.  So although I like the TestManagement option the best, I have had to use the Excel option myself because there was no additional budget for such a tool.  If you must use Excel, my only final suggestion is to invest the time and effort to customize the workbook and develop a process that makes using this option as efficient as you possibly can.

So I hope you found the information useful.  You can also refer to the articles One Click Sprint Management and Agile Test Planning for more information.

Happy Automation!

No comments:

Post a Comment