About STP / 877.257.9531
Log In Join Now
Community / Events / Training / Resources / ST&QA Magazine / Community Forum

Author



Rating

4


Published

Tuesday January 31st 9am

Total Software Quality

Test and QA Testing Software Test Professionals Conference Quality Assurance

With the increasing success of cloud deployment and SaaS business models, software has replaced many companies. Today's largest bookseller is not Borders, its Amazon, a software company. Netflix, a software company, left BlockBuster in the dust. As software usage becomes more mission critical in almost every aspect of life, understanding and more importantly, improving software quality also becomes critical. These days, agile methodologies have worked toward integrating QA with development rather than having QA as an afterthought through test driven development combined with scrum methodologies to form time-boxes for short iterations resulting in working prototypes very early in the development cycle. And as development cycles shorten, better testing techniques and test design with better tools for performance testing, automated testing, etc. have come forth. Structured testing has made significant strides in processes, methods and techniques (i.e. TMAP, TPI, etc.).

Even with all the effort and research in technology and processes, software project failure rates have not decreased significantly nor have there been any proven increases in software quality.

The problem of software quality continues to get more attention and effort, yet the results either stay the same, or even worsen. Sound familiar? Think about the problem with obesity and its cousin, diabetes. Over the last 10 or even 20 years, with diet after diet, and millions or even billions of dollars put into research, the problem of obesity in the USA continues to worsen. Diet alone, or even diet with exercise, are not enough if not done in coordination and in concert with other influencing factors. We at XBOSoft, are not experts on obesity or diet, but we do believe in a holistic and systematic approach to problems, and hence, our viewpoint on approaching the problem of increasing and improving software quality. The problem is that attacking any one single factor could have no effect or even result in negative outcomes in the long run. From the QA perspective, most initiatives on improving software quality today are focused on development and testing methodologies and working to integrate them to be better and more efficient. However, rather than solely focusing on development and testing, we propose using a more holistic view to assess and improve software quality.

This article proposes a holistic view of software quality. Following this introduction, we discuss the following:
  • Defining and understanding software quality
  • Other potential influences on quality
  • Modeling quality from a holistic point of view

Defining and Understanding Software Quality

The problem with quality is that it is an abstract concept. What is it that we are striving for? Its presence is difficult to define and see, yet it's easy to point out its absence. If a customer complains about quality, yet if all the requirements were completed, then does that mean bad quality? So before finding the solution, whether complex or simple, let's examine what quality is. Many have put forward their definitions of what quality is:
  • ISO 25010: the degree to which the system satisfies the stated and implied needs of its various stakeholders, and thus provides value.
  • Roland Petrasch: the existence of characteristics of a product which can be assigned to requirements.
  • Roger Pressman: conformance to explicitly stated functional and performance requirements, explicitly documented development standards, and implicit characteristics.
  • Philip B. Crosby: quality must be defined as "conformance to requirements" if we are to manage it. Consequently, the nonconformance detected is the absence of quality, quality problems become nonconformance problems, and quality becomes definable.

Understanding Requirements

There are numerous other definitions, but they all reference the satisfaction of requirements or conformance to requirements in some way. So when defining quality it is important to first understand requirements and to distinguish between functional and non-functional requirements. Put simply, functional requirements are what the system is supposed to do, such as retrieve a file, or complete a query. But as described earlier, requirements can be fully satisfied and the result can still be 'bad' quality, at least from the perspective of the end user. This leads us to non-functional requirements which refer to how 'well' the functions are accomplished. 'Well' is rather ambiguous and dependent on the eyes of the beholder.

Using Quality Models

To understand and quantify how "well" software meets user requirements leads us to model quality, in order to evaluate it in a quantifiable way. Modeling quality has been done by many, including ISO. McCall put forth a hierarchical model in 1977 combined with quality factors as shown in Figures 1 and 2.
Figure 1. McCall's Quality Model Structure
Figure 2. McCall's Quality Factors
McCall's work was then extended by others including Boehm, Dromey, and later ISO 9126. ISO 9126 (2001) was recently revised and replaced by ISO 25010 in March 2011 with 2 models, product quality, and quality-in-use employing the same hierarchical structure. These models are shown in Figure 3 and 4.
Figure 3. ISO product quality model
ISO's product quality model defines quality characteristics such as Usability by using sub-characteristics, i.e. Learnability. This is a common method where decomposing a characteristic into sub-characteristics enables us to better understand the meaning of the characteristic. In practice, depending on the domain and your end users, when modeling quality you only need to include a handful of characteristics and sub-characteristics, those most important to you. For instance, in the banking industry for software delivered to end users to manage their accounts, the top three characteristics would include usability, performance, and security. These characteristics would in turn be decomposed into their most important sub-characteristics for that domain.
Figure 4. ISO Quality in Use model
The ISO Quality in Use model was developed to clearly differentiate product quality from the product's effect in a real situation of use. As an example, a product could have 'good' quality at a product level, with very good performance efficiency, yet when 'in-use', satisfaction could be very poor. This relationship between the product quality and in-use quality is best understood through illustration in Figure 5.
Figure 5. Quality in the lifecycle [Figure C.2-ISO 25010]
It is important to note that process quality on the far left side of Figure 5 which also indirectly influences quality in use to the far right. Evaluating process quality is most often done using CMMI. CMMI is the successor of the capability maturity model (CMM), developed in cooperation of industry with funding from the US government, by the Carnegie Mellon Software Engineering Institute (SEI). The CMM was developed from 1987 until 1997. In 2002, CMMI Version 1.1 was officially released. At the time, its development was badly needed as the US government needed a structured methodology to evaluate vendors providing systems and software. Vendors who had reached certain levels of process maturity (i.e. CMMI level 3) were able to 'check off a box' and therefore were able to pass through one filter in selling products to the US government.

As you can imagine, any process model and certification system funded by the US government, or any government organization for that matter, involves copious documentation. Contrasting this effort, the Agile Manifesto, was developed over the course of a few days in 2001, and was intended to foster more interaction and less documentation with more adaptive planning.

What is most important in regard to process models and their usage is that although they implicitly improve product quality, they only have influence on product quality. It is possible, for instance, to have a well- documented process and great plan which is followed 100% with software delivered on time and within budget, but end up with poor quality software or software that does not satisfy end users. So, an organization that has a CMMI level-5 certification, may have well documented processes, but may still produce poor quality software, in the eyes of the end user.

Using Quality Models

Unfortunately, although much work has been done with respect to software quality modeling, these models contain guidelines but lack implementation specifics. So, it's up to us to tailor for our own needs and environment. In addition, looking at all these models it's difficult to know how to use, or even where to begin.

Firstly, in our pursuit of high quality software, we must recognize that the final goal is end user satisfaction, or quality in use. This is much different than aiming for product quality alone. Remember that good product quality could still result in poor quality in use and poor customer satisfaction (Note: here we use end user and customer interchangeably). The reason for this is that users are no longer isolated by themselves behind a desk when using your software application. Instead they are in constant communication with colleagues beside them and around the world, which have a great influence on their expectations of your software. In addition, with the popularity of web-based software, there is no longer a user manual, and most times, no formal training. On-line help is all there is. This creates different expectations regarding the learnability of the software. If the user cannot grasp your software in a few minutes (maybe even less than one minute for a mobile platform), they may quit, de-install, and find another easier to use application. Another big reason is that with SaaS business models, users are no longer buying your software, but in effect renting it. This means that they can try other software very easily and not be locked in to a huge capital investment with yours. Hence not only is the payment model different, but there is also a heavy service element. All of these factors combine to form the concept of user experience (UX) where the user's experience is not based on the product alone.

Given all this, it's clear that current quality models do not satisfy all of our needs in terms of evaluating software quality. Not that they can't be used, but they must be supplemented with other elements in order to fully understand, evaluate and improve software quality. We can however, use many existing concepts like hierarchical structure, and measurement and apply our own adaptations.

Developing a Holistic Model for Software Quality

In developing a more comprehensive model for software quality, let's keep in mind the end goal, developing high quality software that meets the needs of end users. The key words in the previous sentence, quality, needs, and end users, refer to the previous sections where we examined quality models, and how they are used to understand, describe, and therefore evaluate the satisfaction of user requirements. The end user can be anyone, for instance a developer, a tester, a manager, administrator, a doctor, accountant, etc. with each user playing a role in the product lifecycle as the product goes from conception to final release. Figure 6 shows a typical product lifecycle from the beginning where a company first conceives of a product to selling and producing it, and then finally maintaining and servicing it. And then, during the course of maintaining and servicing the product, the company runs into new opportunities for different or adapted products and the cycle begins again.
Figure 6. Typical Product Lifecycle
Those involved in each phase of the product lifecycle could have an influence, or impact on the final quality not only of the product itself but for the end users perception of quality. Now, applying the same modeling concepts from ISO (Figure 5), we propose a model that represents the Total Quality Lifecycle (TQL) of a product, from inception to usage by the end user, thereby modeling not only the product quality, but the perceived quality by the end user as shown in Figure 7.
Figure 7. Total Quality Lifecycle
Each phase in the TQL can therefore result in defects that in the end, affect customer perceived quality. Below are just a few examples:
  • Product sponsor: Those that initiate the development and sponsor a project or product can make errors in communicating requirements, expectations, and timelines to product management. Any mis-directions or errors in understanding customers' needs, which thereby lead to products developed with features and functions that don't satisfy needs of the targeted users can be considered defects.
  • Sales: Sales presents the product and its features and capabilities to prospective customers. If they present the product such that after the sale, customer expectations are not met, then this could contribute to a low perception of product quality. Misrepresentation in its various shapes and forms could be considered a sales 'defect'.
  • Requirements: When requirements are written and handed off to developers, if the developer does not understand the requirement fully, or must go back to ask the product analyst for clarity, this constitutes a requirements defect.
  • Development: These are the normal defects that we are familiar with which already have plenty of metrics and measurements. A developer writes code according to a requirement or user story, and there is an error in the code causing behavior to be different than the requirement. Deviants of these defects could be defects that were fixed and come back, or defects that are fixed that create other defects.
  • Testing: Testers also produce defects in their work by not documenting defects clearly so that developers can understand them or not thoroughly investigating an incident such that the defect cannot be reproduced.
  • Customer support/Technical service: Support and service representatives can create defects when the take calls from customers and give incorrect or incomplete information causing a customer to either not be able to solve their problem, or need to call back or both.
The next step would be to define measurements for each of these characteristics. For instance, one measurement of sales quality could be defined based on the customer's awareness of product limitations.

Conclusion

This article has presented an overview of an alternative yet holistic view on software quality, from the end user's viewpoint, or from the customer's perspective. Rather than a traditional view on quality where there is a focus on code quality and defects, we have proposed that in the end customer or user satisfaction is the most important and that quality should be measured from this point of view. Therefore, we proposed defining and modeling quality from this point of view, and some potential ways to measure different quality aspects. After determining a benchmark for these measurements, we can evaluate them according to the benchmark. It is important to note that just like grades in school, it is appropriate to assign indicators or grades on a scale.

Applying a more holistic approach to software quality requires a change in mindset. It's no longer the job of QA, but the entire organization to produce high quality software. It may seem complicated or too much trouble at first, but modeling, and implementing measurements, no matter how simple, can have high yields. With such a paradigm shift, the amount of low hanging fruit may surprise you.

About the Author:
Philip Lew - CEO, XBOSoft
Mr. Lew will lead session 902: Evaluating and Improving Software Usability on Thursday March 29th, at the Software Test Professionals Conference in New Orleans this Spring.

As CEO, Philip Lew guides XBO in its overall strategy, operations, and business development. Previously, he worked at a large IT services provider as their Chief Operating Officer and Senior Vice President. He is an Adjunct Professor at Alaska Pacific University and Project Management College teaching graduate courses in software engineering, IT project management, and IT Governance. Mr. Lew is a certified PMP.


Comments

You must be logged in to comment.
Retrieving Comments...


Advertisement






Friend SoftwareTestPro on Facebook
Follow @SoftwareTestPro on Twitter
Create or Join a Crew

Tweets You Care About


    



Explore STP
About STP / Contact Us / Advertise / Terms of Use

This site's content copyright © 2003−2010 by Redwood Collaborative Media. All rights reserved.
Software Test & Performance™, STP™, Software test professionals™ and Software Test & Quality Assurance Magazine™ are trademarks of Redwood Collaborative Media.