Applications Services Blog
Get the latest thought leadership and information about the role of Applications Services in an increasingly interconnected world at the HP Blog Hub.

Testing as a Service - Case Study #2

Benjamin Romberg, Applications Test Leader, Hewlett Packard Company (Asia/Pacific region)

Alex Martins, TaaS Capability and Business Development Leader, Hewlett Packard Company (Americas region)


check-in kiosk - compressed.jpgLast month we discussed one Testing As A Service (TaaS) case study. This month we’ll introduce a second case study, this one being one of our largest testing services clients in the airline industry.


A Direct Impact to Passengers

This client in the airline industry enhanced their self-service check-in kiosks to provide greater functionality for their passengers. Just checking in was not enough; passengers also wanted the ability to change flights, upgrade their seat assignments, buy promotional products, and so on. To enable this, the kiosk systems had to go through a major technology overhaul.


We were startled to learn that all testing for the legacy platform was performed manually by local airport agents recruited to test as much as they could prior to implementation. The agent - who was not a professional tester - had to test the kiosk software by performing basic customer functionality. This carried with it all the risks of an immature testing process - no standardisation of the test approach, no capture of testing metrics, poor traceability and ad hoc defect management. In addition, most countries have their own rules and regulations for ticketing and boarding, so the manual testing had to accommodate all the different rules for all the different airports around the world.


To make the upgrade even more challenging, the new technology platform was going to be built using Agile and the development team was adamant that they wanted to implement Continuous Integration from the first Sprint and later on get into a Continuous Delivery state. Relying solely on manual testing and using the immature testing approach was clearly no longer an option.


The New Way to Look at Quality

When HP implemented TaaS for the client we provided a mature and structured testing service which included test management, the development of formal manual test cases, automation of the majority of the testing suite, and then execution of the test suite. The billing to the client followed the standard HP deliverable-based pricing methodology, as outlined in the HP TaaS Catalogue (details here: Costing TaaS Deliverables).


A few facts about this implementation:




Test   Cases


Time to   run all tests (before TaaS)

70 hours   (1 dataset or 140 executions)

Time to   run all tests (with TaaS)

10 hours   (8+ datasets or 1120+ executions)

Cost   Reduction brought by TaaS


Return On   Investment

13   months

Testing   Tools Used



Prior to engaging with HP to implement TaaS, the client had a repository of 140 test cases, which were linear and manual, and a full cycle required two weeks to complete. These test cases did not contain test steps; they only had a title, which itself was quite generic. This meant that the test cases could not be understood without the required specialist knowledge pertinent to the test case flow. This also meant there was no guarantee that the test case was run exactly the same way each time it was executed; the test cases themselves were very mutable. Different airport agents would have different interpretations as to how the test case should be executed. In addition to relying upon the agent to interpret the test case as they thought best, there were no set test cases for failure and “unhappy” paths through the functionality; the airport agent had to be relied upon to execute breakage test cases - which meant in many cases these tests were not run at all.


There was no traceability between any defect discovered, the test cases and back to the business requirements, meaning that the test coverage was unknown. Thus the business had no guarantee that their requirements were present within the delivered solution! No regression testing was performed before deployment to Production - so any defects which were resolved were not retested to ensure that they were actually fixed, and that the fix didn’t impact any surrounding or dependent functionality. As would be expected, passengers and clients reported many problems shortly after a new release was launched.


Once TaaS was implemented, the existing test cases were re-engineered to contain detailed test steps outlining exactly how the test case was expected to be executed. This removed the dependency upon airport agents as testers and made the test cases repeatable and maintainable. The test cases themselves were aligned and grouped by business process. When gaps in test coverage were identified, extra test cases were written to ensure a higher level of coverage across the business functionality. With the switch to Agile development (using scrum and Continuous Integration, along with a focus upon automation), the testing suite could be fully executed once per day. Perhaps the most important improvement brought on by implementing TaaS was the Return on Investment. The client estimated that they were saving 13 months of testing effort every calendar year - a huge savings!




An Innovation to the Client

In addition to bringing a cost-effective way for the client to consume testing services, TaaS also enabled the client’s journey towards a more mature SDLC with state-of-the-art testing practices, processes and tools. Below are a few highlights of the innovations brought to this client.


1 - The business is now able to automate their own tests without requiring technical skills

By using the Business Process Testing (BPT) approach, the Business Users and UAT Testers can simply drag and drop the manual or automated test components or test steps created by the HP TaaS team to form a Business Process Test - and they can specify the test data set(s) before test execution or in run-time.


2 - Auto documentation

With the tests properly documented and duplications removed, the auto-documenting features of BPT significantly reduce the maintenance of manual and automated tests.


3 - Scalability and savings

Since this model is not built around individual contributors, it is highly scalable. Also, the TaaS consumption-based model allows the client to pay only for what they use – a highly cost effective approach.



The client greatly benefited from implementing TaaS in several ways:

  • Improved testing maturity,
  • Significantly lower rates of defect leakage out to Production where their customers were impacted, and
  • Excellent cost and time savings.


In out next blog we’ll cover a third TaaS case study, and then we’ll move onto a new topic - stay tuned!


Previous blogs by Benjamin Romberg and Alex Martins:


Previous blogs by Benjamin Romberg:


Previous blogs by Alex Martins:


Related links:



About the authors


Benjamin Romberg Photo.jpgBenjamin Romberg, Applications Test Leader, Hewlett Packard Company (Asia/Pacific region)

Benjamin has over twenty five years of experience in the IT industry, the last twelve as a Test professional. His experience includes work as a test engineer, performance testing specialist, automation specialist, test consultant and test executive. He has worked on a broad range of testing engagements from small web based applications to large core system upgrades. Benjamin has a Bachelor’s Degree in Genetics, a Post Graduate Degree in Software Engineering and an Executive Masters in Business Administration. He now leads HP’s Testing Practice in Asia Pacific and Japan.


Alex.jpgAlex Martins, TaaS Capability and Business Development Leader, Hewlett Packard Company (Americas region)

Alex is the Business Development leader for HP’s Testing Practice in Americas and also the TaaS Capability Leader globally.  His experience includes all aspects of Applications Testing, including management, development, consulting and delivery. Alex has built testing centres in multiple countries and created innovative Testing offerings for HP, such as TaaS.

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the community guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Showing results for 
Search instead for 
Do you mean 
About the Author
Benjamin Romberg has over twenty five years experience in the IT industry, the last twelve of which have been as a testing professional. He ...

Follow Us
Top Kudoed Posts
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.