Managing COTS Test Efforts, In Three Parts

Imagine this:

You are a testing professional newly hired by a mid-to-large size company. Any industry. You are responsible for the testing of all of the company’s internal corporate systems---HRIS, Finance, ERP, CRM, CLM, Sales Management, and Knowledge Management to name a few. You have just been tasked with providing the testing strategy and test management for a large-scale commercial-off-the-shelf (COTS) implementation. At first you think, “Hmm. A COTS vendor application that has already been developed, tested, boxed up, and released?!? This job is going to be a piece of cake!”

Not so fast, buckaroo.

This is where I found myself earlier in my career. I took a role within the Corporate Systems – IT division of a private mid-size company, where I was responsible for the testing of all corporate applications, which were mostly COTS systems. The corporate systems department focused on implementing new systems, upgrading existing systems, and integrating multiple systems. Not only did I learn how to test the apps---mostly from trial and error, oops---but I also learned that testing COTS systems in a corporate systems integrated environment is quite a bit more complicated than I had originally thought.

Here are some of the lessons I learned as well as a high-level strategy that can be adopted to test almost any COTS system. The strategy I use is to divide the COTS test effort into separate “test tracks” where each test track is managed individually with its own team of test resources and its own project plan. The individual test tracks roll up into your master test effort, including the master test effort project plan. Yes, I said, “project plan.” It is my experience that in order to manage test budgets, test timelines, and test resources on a large-scale, complex COTS test effort, it’s important to view yourself not only as a tester, but as a Test Project Manager. Your role is really like conducting an orchestra. While each musician learns his part, it is the conductor’s responsibility to bring all the musicians together and ensure they play cohesively to produce beautiful music.

I’ve organized this article into three parts: Part I is about how testing COTS systems is different than other test efforts; Part II focuses on the various test tracks I recommend for all COTS implementations; and Part III provides some keen insights into how to better manage the COTS test effort.

PART I – The Instruments: A COTS System

Before preparing to conduct an orchestra, it is important to gain an understanding about the instruments involved. For our purposes, we’re going to discuss one instrument, a generic COTS system.

Lesson #1: With COTS systems, you are not testing the actual product itself as if you worked for the vendor company that developed it.

As we all know, software never arrives 100% defect-free. If you have worked for a software vendor in the testing division, then you also know that there is a balancing act between the quality of the system and time-to-market delivery. With vendor products, many times the timeto- market aspect outweighs the value of a truly vetted application.



Download this white paper & save an additional $100.00 off your full STPCon registration! Find the Promo Code in the white paper & Combine it with the Early Bird Discount and Save Up To $500.00 (new registrations only)!