Monday, December 24, 2012

Big Bad Web Services!

Revised:  12-16-2012

Revision History
12-16-2012Develop your test casesAdded a bullet point about test case data modeling.


I hope you will find some value in this article.  I do try to stay away from endorsing specific tools so sometimes it is challenging to write a topic, like this one, without using a tool.  I hope the suggestions and ideas prove useful nonetheless :).


Web Services!  The first technology someone asked me to test that made me say “WTF is this!?” and “How the hell am I supposed to test this!?”  You see, when I was first introduced to Web Services, I was a normal black box tester with a little bit of test automation scripting skills.  I often referred to it as “baby programming.”

So here’s the quicky story.  I began working for a company that developed solutions for mobile phone and wireless device management.  It was my first client server solution that came with guess what!?....a Web Service API.  Now here comes the hilarious part.  They wanted our QA team to test it!  Our QA team?!  We were just a two man team of one manual tester and one test automation scripter.  First of all, I simply didn’t understand anything about Web Services or Web API or SOAP or anything like that!  In the beginning, I had not a clue how to figure out how to develop tests for it let alone run tests.  So for the first time in over a decade, I felt “green” and lost.  But not for long!  I found a way to conquer the big bad web services and I wish to share it with those still trying to figure it out.

What are web services?

So first things first, what are web services?  Let me share my layman’s definition with you.

Web Services provides a programmatic interface to your web application’s server functionality.  In its simplicity, a method of the web service asks for information or sends information (a request) and the server returns the requested information or a status (a response) to the requester.  Often many of the Web Service API methods provide the same functionality as your GUI.

So what is the benefit of using the Web Service API if you can perform the same functions in the web client?  The main benefit is automation.  An example may be, one can only manually add one data record at a time using the web client.  However, at the programmatic level the API can allow the exchange of data and information between two systems or devices or to perform operations in bulk.

See definition of web services as explained on Wikipedia.

Ok so this is just lovely.  We now have a basic understanding of web services. This still doesn’t tell me how to test it!

Ok, let’s get right to it.

The software testing process

Some of us are using agile while some of us are still using more traditional development processes.  Regardless of the development process you are using, at the very basic most of us testers have a strong desire to:
  • Plan our test approach
  • Develop our test cases
  • Organize our test cases into testsets or scenarios
  • Run the tests
  • Review the test results
  • Report errors with supporting documentation
  • Update reported incidents/bugs once failed tests pass.
Yeah yeah we know this but how do we translate this into an actual workflow to test this GUI-less thing called web services.  Let’s begin by identifying what is required for your test development effort.

What information will you require?

Some of our development organizations are a bit guilty of not supplying us with adequate information.  Normally, we could get by with exploratory testing methods to discover the features under test and using our super powers of analysis and knowledge of the system to figure a feature out.  For most of us testers, especially black box testers, we cannot use this method to learn a web service feature.  Perhaps white box testers or those who can see inside the code can “explore” it but not all testers possess the skill to do this.  So if you need help exploring the web services under test and its methods, you can use one of two methods to learn the web services:
  1. Documentation:  Agile loves user stories.  However, we need more than a cute fairy tale to help us out here.  We need documentation or a JavaDoc that tells us: a) what feature of the server (artifact) the group of methods service; b) method name; c) argument list with required parameters, d) what the method returns when called; e) nice to know what the types are for both the parameters and the return values; and f) of course the url to the web service api.
  2. Web Services Testing Tool:  These tools are fantastic for discovering web services however they can be a bit tricky to learn.  When I first used a web services testing tool, I remembered thinking “WTF is this!?” So don’t freak out!  The first thing you can learn is simply how to provide a url to the tool’s project which should create an interface to the web services.  The interface should list the web services and then a list of their methods.  You should find an XML representative of the request which should show an outline of the parameters, their type and which ones are required.
OK, my fellow Selenium enthusiasts, I know you can use Selenium for testing your web services however it is quite difficult to discover the services so you require documentation OR you can utilize any good open source web service testing tool to help you.
Ok so what other items do we need for web services testing?
  • A programmatic way to test
    • Here we have to decide if we will be:
      • Super Automationers camouflaging as developers using for example Eclipse or IntelliJ (or any IDE of your choice) and write your own code.
      • Bold Automationers coming out of your comfort zone and use a web service testing tool.
      • Comfy Automationers using the web service testing feature of a robust test automation tool.
      • The refusers!  It is okej.  You can still plan how to test and let your neighborhood friendly developer write the code :).
      • A language choice like .Net or Java.
      • A proper test environment with correct access and permissions

Plan your tests

So what is different with the planning process?  Nothing!  Don’t let their window-less nature throw you for a loop.   My only note is to plan for performance testing and security testing if possible.  We want to know what happens if a web service method is accessing a record at the same time someone is accessing it via the GUI or if the web service is performing bulk operations.  We also want to know how secure the server is as well.  All other steps you take to plan your test efforts remain the same.  For ideas on agile test planning, see the article Agile Test Planning.

Develop your test cases

Ok so now that we understand what the heck it is and what the heck we need.  Now we need to put our testing hats on.  Yeah programmatic doesn’t mean we get to escape defining the test cases.  Here are four suggested methods for developing test cases for web services:
  • The programmatic approach using classes and methods
    • You can keep it simple by using an IDE to develop tests where you are using your chosen open source automation tool or java and:
      • create a package containing classes.
      • In your classes, you have methods that serve as test cases.  These test cases will call your web service method, pass the arguments and test the results.  For information on structured test methods see the article Teach Me Ovie One KaDevie.  You will perhaps have several test methods for one web service method. You should test for permutations such as:
        • Required parameters only
        • Missing parameters
        • Invalid parameter type
        • Invalid data
        • Etc.
    • Last, develop a “launch test” or “driver script” to execute all of your test cases.  In the developer world, this would be called a “main” method.
  • Extend the TestFramework to include component functions designed to call the web service, pass the parameter list and receive the response.  Now you can test the response to verify if it meets your expected results.  The data is passed by your chosen test data management system like Excel, CSV files, or other preferred source.   I prefer this method of expanding my TestFramework.  We can make robust, reusable functions that allow us to extend our test effort to our heart’s content.
  • Use test case data modeling tools.  This method will allow you to "graph your tests" for each web service method and provide a pictorial way of viewing the web service method's permutations.  These graphs can still be managed and linked to associated test artifacts in a test management system.  They are a great way to save time with test case and test case data development and management.  The other cool benefit to using this method is the ability to drive the permutations randomly.
  • Web service test tool
    • If you elect to use a web service testing tool here are the key things you want to learn:
      • Creating a testing project
      • Providing the URL to your web services
      • How to create test requests
      • How to assert the test results
      • How to make scenario-based tests
      • Leverage java or groovy to make advanced tests
      • Learn how to do data-driven tests
      • A bonus if the tool will allow you to assert the results of your web services against the GUI.
    • For advanced test automation, some web service testing tools can be included with your TestFramework project, e.g. if you are using Selenium.  You can then leverage tools like Maven to manage the project’s dependencies.
    • Automation test tools
      • Some of the commercial testing tools provide features for testing web services.  Actually this is a nice compliment to those using the tools for other test efforts such as browser-based or window-based testing.  Most testers posing as test automation developers once they find one tool they are comfortable with, they often don’t want to leave that comfort zone let alone invest time they usually do not have into learning another tool.  The benefit here is web services can more easily be integrated into your overall TestFramework, Keyword Driven Functions or Test Case Scenarios.  For example, a) call a web service that adds a record; b) assert the return status of the call; c) verify that record can be accessed within the GUI.
      • Using the commercial tool fits various test automation skill levels.  One particular tool I used allowed the tester to interface with the web service and then provide code snippets for the method they wish to test.  This was nice as then one could easily modify the code to meet their needs or make it reusable.

Organize your test cases

From your test library, select and organize the tests cases for scenario-based testing.  You should be able to re-use the same test case within various test set scenarios to test different conditions.

If your web services test is basically a packaged set of java or javascript code, then you might do something rudimentary like run your GUI tests, then run your web services tests, then run another set of GUI tests to verify the changes posed by the web services.

If you are using an advanced TestFramework, then you have more freedom how to organize your test cases for more realistic scenario-based testing.

Again don’t feel intimidated.  It can take time to reach utopia.  When I began, I used JavaScript to develop test cases to verify the web services and executed them using java.  It was not a very resilient option.  I tried using a web services testing tool but honestly, I lacked the bandwidth to try to figure it out at the time.

Eventually using an open-source test tool, I learned to incorporate the methods along with my browser-based and window-based testing functions.  The utopia was one test case could leverage all the test functions for robust and realistic test scenarios for different test types.

Run your tests

Now that you have developed your tests, you want to run them repeatedly.  At least I hope so :).  How are you going to do that?  Well it depends on your chosen method.
  • Commercial or web service testing tools:
    • It is simple.   Click the “Play” button.  You start your testing tool, you navigate to the launch script or in the case of web services test tools you play the project or the selected test request.
    • These tools often provide a command-line feature that allows you to kick off your test run from a batch file or from another tool you are using.
    • TestFramework
      • If you can develop your test at this level then I don’t need to spoon feed you a kick off method :).

Get results with supporting information

Since we are using a programmatic approach, regardless of the tool, we can design our functions or methods to return one of the following statuses:
  • Pass
  • Fail
  • Skip
  • Block
See the above links for ideas on developing a method to return the desired test status based on the execution results of your function or method.


The most important thing we want to achieve is reporting our results in the same or similar fashion as we do all of other tests.  You still want to know which tests passed or failed your expected results or which tests were blocked from execution or which ones were skipped during a test run.  Additionally, you want to automate gathering the required supporting information of non-passed tests.  For web services, you want to capture:
  • The description of any exception that might have been thrown during a call to the web services.
  • The error logs generated by erroneous handling of the web service.  See the article I now pronounce you TestComplete and soapUI.
  • The logs generated by your test run.
The information gathered will either:
  • help the tester improve their web service test cases;
  • correct the web service test case;
  • provide the developer with valuable information to help debug and fix web service errors; or
  • provide your IT guy information to debug any environment or network related issues.

Update reported incidents from passed test

If we’ve done a great job developing our tests and reporting discovered problems, the tests should automatically pass when you run them on a newer build that corrects the reported problem.


There you have it, the beginnings to conquering the big bag web services.  As I said earlier, we can continue this discussion and we should!  There are a lot of things to talk about such as understanding more about
  • SOAP and REST and how to assert the results
  • JSON responses
  • WS-Security
  • Pre-emptive and NTLM Security
  • Security and Penetration testing
  • Load and Performance testing
  • And the list goes on :).
So have fun, let’s overcome the fear, roll up the sleeves and tame the big bad web services.

Happy Automation!

No comments:

Post a Comment