Applications Services Blog
Get the latest thought leadership and information about the role of Applications Services in an increasingly interconnected world at the HP Blog Hub.

Case Study #3 - Performance Testing As A Service

Benjamin Romberg
, Testing Solutions & Capability Leader APJ, Hewlett Packard Company

Alex Martins, TaaS Capability and Business Development Leader, Hewlett Packard Company (Americas region)


Airplane computer.jpgThis post introduces a third Testing as a Service (TaaS) case study focused on how Performance Testing as a Service has enabled one of HP’s airline clients to successfully implement a complete hardware refresh and a major website overhaul, including updating their GUI and back end web server technology.


Original State

A large North American airline was going through a major three-year overhaul of their website. The project included a full hardware refresh, moving the site from a physical infrastructure to a completely virtualised infrastructure, hosted as a private cloud model. At the same time, the underlying website application code was to be refactored, the website user interfaces redesigned and the mid-tier platform migrated from ATG to WebSphere.


The client team didn’t have a deep understanding of their website traffic patterns, such as the most visited pages or the most popular and heavily executed transactions. They also lacked the appropriate tools needed to properly capture and analyse that information. The client used an open source performance testing tool that required a team of developers to maintain. However, this tool was not providing developers supporting it the detailed results needed to improve website performance. The tool was also being under-utilised, as the development team was using the tool mostly to perform smoke testing, not full end-to-end performance testing.


The Need

The client needed on-demand testing services to accommodate the peaks and valleys across their projected testing demand according to the ever-changing client priorities and environment availability. In addition, the client needed support to:

  1. Accurately and comprehensively determine the performance traffic patterns so they could better understand their website usage profile. From this information, the functions that support robust end-to-end Performance Testing could be scripted (these are the functions which generate most of the web traffic at the busiest times and are the most business critical).
  2. Define the right Performance Test Scenarios (Average, Peak, Stress, Endurance, etc.) based on their production traffic patterns as well as the growth predicted by the business team.


HP LoadRunner, integrated with dynaTrace, was utilised for code level root cause analysis. HP also developed an SLA Dashboard to automatically compare baseline transaction level metrics as well as server resource level metrics to current execution metrics (before vs. after) and flag the underperforming metrics. The load generators were initially set up within the datacenter to ensure the environment was performing as expected. Network virtualisation with Shunra was then utilised to validate the performance actual users would experience across different countries around the world.


Here are a few facts from the project and ongoing engagement:




Business Functions (before TaaS)


Business Functions (with TaaS)


Time to run all scenarios (before TaaS)

2 weeks

Time to run all scenarios (with TaaS)

2 days

Website traffic

2 million unique users/day

Number of territories and countries


Number of environments

27 QA environments, 1 Pre-Prod and 1 Prod

Testing Tools Used

HP LoadRunner, Shunra, dynaTrace


The Results

The client was able to successfully complete their project on time and within budget. Performance testing was successfully completed for the full production environment hardware refresh and platform migration to cloud. This included a before and after comparison showing that the migration and refresh improved overall performance. Several performance defects were detected by the daily regression testing on QA environments where new code was checked in each day. No post-production performance defects were reported in the twelve months that followed the implementation. HP continues ongoing Performance Testing support through the TaaS model, which enables the client to just pay as they go throughout their Agile Sprints and production releases.


Previous blogs by Benjamin Romberg and Alex Martins:

Previous blogs by Benjamin Romberg:

Previous blogs by Alex Martins:

Related links:

About the authors


Benjamin Romberg Photo.jpgBenjamin Romberg, Testing Solutions & Capability Leader APJ, Hewlett Packard Company

Benjamin has over twenty five years of experience in the IT industry, the last thirteen as a Test professional. His experience includes work as a test engineer, performance testing specialist, automation specialist, test consultant and test executive. He has worked on a broad range of testing engagements from small web based applications to large core system upgrades. Benjamin has a Bachelor’s Degree in Genetics, a Post Graduate Degree in Software Engineering and an Executive Masters in Business Administration. He now leads HP’s Testing Practice in Asia, Pacific and Japan.


Alex.jpgAlex Martins, TaaS Capability and Business Development Leader, Hewlett Packard Company (Americas region)

Alex is the Business Development leader for HP’s Testing Practice in Americas and also the TaaS Capability Leader globally. His experience includes all aspects of Applications Testing, including management, development, consulting and delivery. Alex has built testing centres in multiple countries and created innovative Testing offerings for HP, such as TaaS.

Showing results for 
Search instead for 
Do you mean 
About the Author
Benjamin Romberg has over twenty five years experience in the IT industry, the last twelve of which have been as a testing professional. He ...

Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.