Through my many years in QA, I have experienced many epiphanies about quality, testing and what these terms actually mean to business owners and end users. After 20+ years working for both large multi-national and small companies across several industries, I have become intensely aware of how business goals and objectives drive the many dimensions of measuring business value and quality. The nature of the end product/application will dramatically affect how quality is defined and therefore as testers, one’s approach and understanding of how to relate and articulate this will be instrumental in ensuring results have accurately validated attainment. Product managers, business owners and consumers/ end users will all be involved in defining quality and value. Each will have their own perspective if they have been satisfied. In this article, the discussion will surround how to create a value-add testing organization. The scope will cover business objectives and goal alignment, quality dimensions in support of the business goals and how investments in testing have a quantifiable return.
Performance Pay and the Alignment of the Delivery Team
Companies that have disparate cross-functional goals have more operational issues, system outages, customer emergencies and obstacles in achieving their objectives than those with aligned goals. Goals will be different from company to company but generally, revenue targets, customer satisfaction rating, market share, solving business problems and challenges are some of the most common. Aligning goals to the success criteria of the end result attained in the marketplace is a great tool to raise the performance of the team. Instead of making some success be it an interim deliverable date defined by some project plan, the team measures performance on whether or not the end result attained the business value and objectives. When product/business owners, developers and testers are tied (at the wallet) to the end result, you will see an immediate increase in collaboration and teamwork. Similarly, when applied to an organizational goal to increase efficiency and lower cycle times. The delivery team examines how each could do their part in attaining this goal and looked for ways to assist their teammates in removing waste and indentifying efficiency enablers.
Keeping Your Eye On the Prize
When you hear the battle cry of your business and technology leadership saying we need faster, better, cheaper delivery of our software products and we must be innovative while doing so, it can be daunting. It truly is a very tough task to accomplish all of these goals at the same time. It can be done, but not all at once. Ask yourself do you understand how this request aligns with the company’s goals and objectives? Just as important is to understand and help your team understand the connection from quality to testing. If sales forecasts require that the product is available in the market place by a stated date in order to maximize the window of opportunity for revenue, then time to market is the most important criteria. Those activities that will accelerate delivery time will be sought out by the team. Finding that balance where increased speed does not create a quality issue is how the team will proceed and make trade-offs accordingly. At some point the team will need to assess when does speed to market cause lower quality that could impact this objective and possibly other objectives? Balance across objectives needs to be considered before a team proceeds with course corrections. Making sure the business owner is clear about the test effort in relation to the quality objectives will lower the chances of surprises when the effort (time, duration, cost) is needed for the level of quality required.
Dimensions of Quality
Aspects of quality or dimensions generally fall into the following categories: reliability, usability, maintainability, conformability and performance. The challenge here is that it is not normal to think of a test plan or strategy in terms of quality dimensions nor are business requirements categorized in such a manner. As testers, one needs to be able to articulate the tests in terms of these dimensions along with the effort to deliver it. A value add testing team will communicate and quantify the effort to achieve the quality dimension(s) yielding the commensurate business value. In the case where application reliability and accuracy are paramount to achieving business objectives, the testing organization needs to quantify and communicate the test effort required to deliver reliability. This robust test coverage will have an increased cost as compared to an application where reliability had less of a business impact. To achieve this level, the number of tests will increase ensuring the team has met a highly reliable application (i.e. range, boundary, min, max and error testing etc.). Business owners can now make a connection between the goals, quality and the test effort required. In the case where the testing phase gets crunched from eight weeks down to four, the team can quantify the effort that is necessary to attain that stated level of quality/dimension (reliability, accuracy, performance) to meet the business goal(s). The time reduced in testing might be so great that the product will not meet its goal to increase application reliability because the time to test is no longer sufficient to certify the products functionality. There is increased risk to the business goals; there will be untested code deployed into production that could negatively impact the business. Testing teams unable to articulate quality dimensions/objectives in terms of the test effort to support the business most often have been labeled a low value add team.
Justifying your Investments in Testing
Investments require the establishment of sufficient business rationale in order to determine the value that this investment is going to bring to the business and thus justify required expenditures and resource allocations. Call it a business case or business justification; we anchor our investments in testing to meet quality objectives in support of business goals. We measure success (meeting specifications) through testing/validation. The ability to articulate the expected benefits and associated cost of quality through investments in testing will be decided by the merits of a business case/rationale. These goals can be qualified (reliable, scalable) and/or quantified (5% increase in sales revenue/ market share) but regardless, one needs to identify the value and benefits the investments (testing) will and have contributed to the business. A qualitative goal such as, “the product has to be more intuitive to the customer”, is much harder to measure; but in the end, even this intuitive goal can be measured quantitatively. One can quantify/measure website traffic, usage, duration on site, increased sales/products sold, features purchased and increased advertising revenues. The marketing department has likely done some studies on qualitative goals and it would be a good source for the data one will need. Some general guidelines to follow:
- Understand the business rationale, quality goals, & success criteria – discuss this with your business owners and stakeholders but do not forget about marketing and finance to ensure a full understanding
- Know what you want from this initiative and how it will contribute to fulfill your success criteria
- Quantify as many of your objectives/goals as possible
- Identify all the processes in support of the goals and business rational
- Set thresholds and targets
- Use different ROI models for different purposes for the same project
- Numbers can say whatever you want; don’t use ROI calculations to confirm a bad investment
Why Metrics
Software testing exposes human failure and naturally people do not like to admit mistakes. A high performing aligned team will rally around these failures and hold themselves all accountable to the resolution bypassing the individual aspect. Some will go to surprising lengths to hide mistakes or take steps to remedy them before they are discovered. Establishing a measurement program is an investment in success as it enables proactive behavior driving early detection of risks to the business. Metrics provide the ability to identify, resolve, and/or mitigate risk issues before they surface. The act of measurement is not a goal in itself and needs to be integrated into the culture, the delivery methodology or the team. Metrics are used to compare the current state of your process/end results with past performance and/ or previous estimates. The results will identify trends (negative/positive) which enables the team to continue to adjust the course corrections. To be effective and drive improvements, metrics must not only be collected they must be used to understand if changes to delivery are yielding the intended efficiencies. Effective components include the following examples:
- Clearly defined software delivery issues affecting the business objectives and the data elements needed to provide insight
- Standardized processing of collected data
- Results need to be published on a regular basis otherwise progress will remain uncertain.
- Results must drive process improvements
In summary, if an organization’s and/or department’s goals are not in line or in support of business goals it is important to share business goals with your peers and understand how the synergized goals can be more powerful to the organization. Spend the time and find out some information about the product, its’ users, the market and what quality dimensions are important to each and how to measure them. Protect brand image and good will as bad news travels very quickly in the social networking era. When customers delay their purchases, there is an immediate impact to operating revenue. They might start to lose confidence in your software; this translates to lost opportunities and negatively impacts future revenue streams. Keep the team aligned; assess the impact of change to meeting your goals and your chances for success increase dramatically. Justify and quantify the test effort. If the business owner understands the relationship and you have quantified the effort, your ability to explain the cost, risk, benefit equation of quality & testing will go a long way to ensuring your team is adding value to the business.
About the Author
Paul Fratellone Paul Fratellone has a 27-year career in information technology
centralized in Testing & Quality Management. Performing every
activity/task of the QA professional from QA Analyst to Director,
he has experienced many challenges and opportunities to deliver
quality through testing. Through the years and challenges, Paul
has gained a keen insight in knowing how to tailor the process to
fit the maturity level while still delivering incremental efficiencies.