- It’s Like Conducting an Orchestra!
You are a testing professional newly hired by a mid-to-large size company. Any industry. You
are responsible for the testing of all of the company’s internal corporate systems---HRIS, Finance,
ERP, CRM, CLM, Sales Management, and Knowledge Management to name a few. You have
just been tasked with providing the testing strategy and test management for a large-scale
commercial-off-the-shelf (COTS) implementation. At first you think, “Hmm. A COTS vendor
application that has already been developed, tested, boxed up, and released?!? This job is
going to be a piece of cake!”
Not so fast, buckaroo.
This is where I found myself earlier in my career. I took a role within the Corporate Systems – IT
division of a private mid-size company, where I was responsible for the testing of all corporate
applications, which were mostly COTS systems. The corporate systems department focused on
implementing new systems, upgrading existing systems, and integrating multiple systems. Not
only did I learn how to test the apps---mostly from trial and error, oops---but I also learned that
testing COTS systems in a corporate systems integrated environment is quite a bit more
complicated than I had originally thought.
Here are some of the lessons I learned as well as a high-level strategy that can be adopted to
test almost any COTS system. The strategy I use is to divide the COTS test effort into separate
“test tracks” where each test track is managed individually with its own team of test resources
and its own project plan. The individual test tracks roll up into your master test effort, including
the master test effort project plan. Yes, I said, “project plan.” It is my experience that in order to
manage test budgets, test timelines, and test resources on a large-scale, complex COTS test
effort, it’s important to view yourself not only as a tester, but as a Test Project Manager. Your role
is really like conducting an orchestra. While each musician learns his part, it is the conductor’s
responsibility to bring all the musicians together and ensure they play cohesively to produce
I’ve organized this article into three parts: Part I is about how testing COTS systems is different
than other test efforts; Part II focuses on the various test tracks I recommend for all COTS
implementations; and Part III provides some keen insights into how to better manage the COTS
PART I – The Instruments: A COTS System
Before preparing to conduct an orchestra, it is important to gain an understanding about the
instruments involved. For our purposes, we’re going to discuss one instrument, a generic COTS
Lesson #1: With COTS systems, you are not testing the actual product itself as if you worked for
the vendor company that developed it.
As we all know, software never arrives 100% defect-free. If you have worked for a software
vendor in the testing division, then you also know that there is a balancing act between the
quality of the system and time-to-market delivery. With vendor products, many times the timeto-
market aspect outweighs the value of a truly vetted application.
Download this white paper & save an additional $100.00 off your full STPCon registration!
Find the Promo Code in the white paper & Combine it
with the Early Bird Discount and Save Up
To $500.00 (new registrations only)!