The “World Series” of Test Requirements

As I made changes within the quality control (QC) team, the program manager noticed the progress being made. Not only had I taken the burden off the subject matter experts (SME’s) during user acceptance test (UAT), I had taken some pressure off the program manager as well. He no longer had to worry about the QC team. With the QC team on the right track, we were making huge strides in creating a solid safety net for the ROSS project.

Since I had success in changing the QC test processes, the program manager had asked several times how to improve the requirements process. I knew the process needed improvement, but I had never managed requirements so I really didn’t have an answer. I could analyze requirements but that was the extent of my ability in that area. With each release, it became more and more apparent the requirements process needed change.

When the program manager came to me again to help with the requirements process, he not only asked me to improve the requirements process, he asked me to manage the team as well. I agreed on the condition I could hire two more requirements analysts with one to eventually become the lead. Doing so brought new ideas to the project and allowed me to rely on experienced individuals instead of starting from scratch on how requirements should be developed.

For me to give you a clear picture of why and where we took the requirements, it helps if I demonstrate where we were and the problems it was causing.

If I were to use ROSS business requirements it wouldn’t make much sense, so lets start with an example of business requirements that everyone can relate to and somewhat parallels a ROSS business requirement. When the US Forest Service fights a wildfire, they send in teams of firefighters. These teams of firefighters are identified and assigned to rosters, which from a high level correlate to a sports roster. A roster is created for the wildfire, but not all firefighters identified on the roster may end up going to the wildfire for one reason or another. Likewise, a baseball roster may have up to 15 players but only 9 get to play. As an example, let’s define the business requirements for managing a baseball roster and line up.

Creating a roster and line up for a baseball team might seem pretty simple. It can’t possibly be such a big deal, right? I can hear developers, software managers, CTO’s, requirements analysts from my past (and present) say, “Oh we can build that. It will be easy and I know exactly what the client wants.” Hmmm… let’s develop some basic business requirements for creating a baseball roster.

1. A baseball roster consists of 15 players.
2. The line up designates 9 of the 15 players on a baseball roster to play at any one time
3. The batting order defines the sequence in which the players will bat.
4. A designated hitter can be substituted for the pitcher in the line up.
5. A pinch hitter can be added to the line up.

1. Can the same person be on 2 different rosters?
2. Can any of the 5 players on the roster, but not in the lineup, come in to the game at any time as a pinch hitter?
3. Can the player identified as the designated hitter change during the game?
4. Can the player identified as the pinch hitter change during the game?
5. Can the designated hitter be one of the 9 players in the line up?
6. Are the positions associated with each of the 9 players defined in the line up?
7. Are there rules associated with these positions?
8. Doesn’t the pinch hitter make the line up equal to 10 players?
9. Can the batting order change during the game?

As you can see it was very easy to come up with twice as many questions as requirements and we have just scraped the surface! When it comes to developing business requirements for software, if they seem simple, there is a good chance they are too simple. For software to be developed, the business requirements have to define all the ‘can dos’ and ‘can’t dos’.

When I started on the ROSS project, the business requirements were being developed in a hierarchical, paragraph format, stored in a requirements repository for each enhancement. Using the baseball roster requirements listed, here is an example of what they might have looked like:

1. Baseball Rosters

1.1 All baseball rosters will consist of 15 players and players must be 15 years and younger in age but no younger than 7 years in age and can have a status of active or inactive. There can only be 12 active players at one time.

1.1.1 All baseball lineups will consist of players from the baseball roster of active players. These 9 players will fill positions consisting of catcher, first base, second base, third base, shortstop, left field, center field, right field and pitcher.

1.1.1.1 The 9 players that make up the lineup from the roster can have a pinch hitter who bats in place for them that bats for in place of them in the batting order.

1.1.1.1.1 A designated hitter must be a defined player on the roster and can substitute for the pitcher but can not be any of the 9 players on the lineup, can not be a designated hitter for 2 players, can never come into the game as one of the 9 players in the positions of catcher, first base, second base, third base, shortstop, left field, right field, pitcher, has a status of ‘reserve’ when hasn’t entered the game, a status of ‘active’ when in the lineup, a status of ‘played’ if the player was in the game and taken out and has to bat in the same position in the lineup order.

1.1.1.1.2 A designated hitter must be a defined player on the roster and can substitute for the pitcher but can not be any of the 9 players on the lineup, can not be a designated hitter for 2 players, can never come into the game as one of the 9 players in the positions of catcher, first base, second base, third base, shortstop, left field, right field, pitcher, has a status of ‘reserve’ when hasn’t entered the game, a status of ‘active’ when in the lineup, a status of ‘played’ if the player was in the game and taken out and can not be designated as the pinch hitter.

1.1.1.1.3 A designated hitter must be a defined player on the roster and can substitute for the pitcher but can not be any of the 9 players on the lineup, can not be a designated hitter for 2 players, can never come into the game as one of the 9 players in the positions of catcher, first base, second base, third base, shortstop, left field, right field, pitcher, has a status of ‘reserve’ when hasn’t entered the game, a status of ‘active’ when in the lineup, a status of ‘played’ if the player was in the game and taken out and can not return to the game if that player has been taken out.

1.1.1.1.4 A designated hitter must be a defined player on the roster and can substitute for the pitcher but can not be any of the 9 players on the lineup, can not be a designated hitter for 2 players, can never come into the game as one of the 9 players in the positions of catcher, first base, second base, third base, shortstop, left field, right field, pitcher, has a status of ‘reserve’ when hasn’t entered the game, a status of ‘active’ when in the lineup, a status of ‘played’ if the player was in the game and taken out and can not be any of the other players that were in the game as one of the 9 positions and taken out.

1.1.1.2 The 9 players that make up the lineup from the roster can have a pinch hitter that bats in place of one of the 9 players.

1.1.1.2.1 A pinch hitter must be a defined player on the roster and can be any of the players that are not any of the 9 players in the positions of catcher, first base, second base, third base, shortstop, left field, right field, pitcher, has a status of ‘reserve’ when hasn’t entered the game, a status of ‘active’ when in the lineup, a status of ‘played’ if the player was in the game and taken out and can not return to the game if that player has been taken out.

1.1.1.2.2 A pinch hitter must be a defined player on the roster and can be any of the players that are not any of the 9 players in the positions of catcher, first base, second base, third base, shortstop, left field, right field, pitcher, has a status of ‘reserve’ when hasn’t entered the game, a status of ‘active’ when in the lineup, a status of ‘played’ if the player was in the game and taken out and has to bat in the same position of the lineup order.

1.1.1.2.3 A pinch hitter must be a defined player on the roster and can be any of the players that are not any of the 9 players in the positions of catcher, first base, second base, third base, shortstop, left field, right field, pitcher, has a status of ‘reserve’ when hasn’t entered the game, a status of ‘active’ when in the lineup, a status of ‘played’ if the player was in the game and taken out and must be added during a dead ball point in the game.

Is your head spinning yet? Mine is. The following is a summary of the major issues we were having with the way the business requirements were developed:

1. The same information was repeated over and over again from requirement to requirement with just a small change denoting the real requirement. Requirements had to be compared side-by-side in order identify that requirement.
2. If one of the business rules was incorrect due to something as simple as ‘can’ vs. ‘cannot’ it was hard to catch as there was so much to digest in one requirement.
3. Since the requirements were in a hierarchy it was unclear if the parent requirement applied to every child requirement.
4. If a requirement was added or removed, it could change numbering of the requirements and cause a ripple effect in the traceability for the software requirements, design documents and test cases.
5. The SMEs would split up the requirements for UAT by requirement. For example, SME #1 would test requirement 1.1.1.1.1, SME #2 would test requirement 1.1.1.1.2, SME #3 would test requirement 1.1.1.1.3 and so on. If one of the rules stated in the requirement was incorrect, then, since it was repeated in every requirement, we would have three incorrect UAT test cases, which turned into 3 duplicate defects.

These issues with the business requirements rippled throughout the rest of the software development lifecycle.

We needed to revamp the requirements process and we took the route of use cases as the focal point for the business requirements. I had never worked on a project where uses cases were utilized. However, in doing research on them I discovered they could be used as the basis for developing solid test cases, which was the driving force for the changes needed to help the SMEs and UAT and also meet the needs of our internal clients; software engineering and QC.

With the requirements team in place, I pushed them to define the business requirements process. My role as the manager was to keep them moving. If they got to a point where they didn’t agree or couldn’t make a decision, I stepped in and made a decision for them and reminded them, we can always make tweaks/changes later if it wasn’t quite working.

Looking at the business requirements from a high level, 2 phases were defined;

1. Business Modeling, where business requirements are created to accurately capture the customer’s need and the boundaries of the change request.
2. Discovery, where the business modeling version of user requirements is decomposed into detailed user requirements.

While some would argue the business modeling phase is not needed, the argument for it on the ROSS project was that we had actually implemented change requests (some of large scope) only to find out, after many labor hours, that the change was really not what the field wanted or was going to use. Change requests were sometimes blindly implemented without documenting a clear reason as to why change was needed. It was important to make the SMEs think about the change request, analyze if it was even needed or made sense to implement in the software itself.

During the business modeling phase, we made it a goal to get answers to the following questions:

1. Did the business process need make sense as something that needed to be addressed through the software? Or should it be addressed by a manual business process?
2. What exactly is the business process currently in the field for this business problem?
3. Did the software already address the need and it was merely a training issue?
4. Did it contradict how the software already functioned? Were we going to implement an exception to the business rules already defined?
5. If it is determined that a software change is needed, what exactly is the scope of the change?

The business modeling phase focused on capturing ‘WHY’ the change request was needed and setting the scope for the change request. This information is extremely important in determining the ‘WHY’ for other change requests, to determine contradictions, or impacts to existing functionality. It also aids when a project starts a technology refresh effort, and you revisit the business requirements to determine if the functionality is even needed or can be simplified. Overall, it eventually drives the scope of the change.

Defining scope or avoiding scope creep, as we all know, is extremely important. We’ve all been there when scope has tripled/quadrupled while in requirements definition, code development, system test or at the worst time, UAT. Getting boundaries defined around a change request is probably the single most important part of the whole software development life cycle (SDLC) as it impacts everything. Did we completely eliminate scope creep? No, but over time we minimized it, and scope creep that did happen was small and manageable. We knew that if we executed the business modeling phase successfully, we would address the true business process problem rather than addressing a symptom of the problem or only getting it partially corrected. Once the business modeling phase was complete, the next step was to drive out the detailed user requirements during the discovery phase.

Our first step was to define the different types of user requirements, which gave structure to requirement components and allowed for consistency between the requirements analysts themselves. Looking at the requirements as a QC analyst, I knew it was unfair for those having to digest requirements to have to say, ”Well I’m reading Joe’s requirements today and his writing style is such and such, so I need to review the business requirements keeping that in mind.” The business requirements needed sufficient standards, structure, and definition, to ensure each requirements analyst would develop requirements consistently. Could you ever achieve 100% consistency? No, but we could try to make it as easy as possible for another person to absorb the requirements. The following are some examples of the definitions/standards we used:

Assumptions: Statements that describe important ideas or concepts that exist at the time of starting the project. Many times, they are used to document conditions that the project team does not have control over (e.g., Age of players will be verified with copies of birth certificates.).

Non-Functional Requirement: A requirement that documents quality attributes the system must have. These attributes are not system features, but they do influence how the functionality of the system is implemented and usually deal with some aspect of usability, performance, security, or are operational in nature (e.g., The software will load a web page within 5 seconds under a load of 100 concurrent users.).

Business Requirement: Requirements that describe stakeholder needs to support the business process. Business requirements provide the rationale for introducing new or changing functionality as well as a high-level description of what is needed (e.g., The baseball application will enable users to enter game day lineups from defined rosters.)

Business Rules: Requirements that control or influence the behavior of the system in order to enforce an operational business policy or practice. Business rules represent a constraint, calculation or condition. Using the baseball roster requirements an example may be: Once a player has been removed from the lineup they cannot return to the lineup.

Software Requirements: Requirements that provide the specific text to be displayed to users, or sent to external systems, to serve as an error, warning or confirmation. These define the design of the user interface.

Business Use Case: Steps that describe the interaction between the user and the system in order to achieve a specific goal. Use cases organize functional requirements by primary system entity and then user function.

Uses cases became the ‘meat’ of the detailed user requirements. As I stated before, use cases lent themselves to developing solid test cases, and if a requirement is testable – then it can be turned into code.

TThe second step was to evaluate and define the members involved in the requirements process. Before I took over the requirements process, the requirements were defined by the government lead SME and the requirements analyst or the government program manager and the requirements analyst. Then the requirements document was passed to the rest of the team for development and testing. Development, QC and the rest of the SMEs were not involved with the business requirements from the beginning. We didn’t get to hear why decisions were made; we didn’t get to chime in on whether the requirements made sense, if they contradicted each other, if they were incorrect, or if they correctly represented how the product was working at the time. Consequently, for every enhancement, an integrated product team (IPT) was identified that consisted of:

Government Subject Matter Expert (SME): Government employee who has the authority to make decisions on behalf of the ROSS Project Manager.

Lead Architect: Architect assigned to work with the IPT to ensure decisions are consistent with the current architecture and to provide expertise in architecture changes.

Lead Reports Developer: Software engineer assigned to technically design and implement the new and changing reports defined in the user and software requirements.

Lead Reports Quality Control (QC): Analyst/test engineer assigned to determine impacts to existing reports and the existing analytical reporting environment as well as identify new reports or new data to be added to the analytical reporting environment

Lead Reports Subject Matter Expert (SME): SME assigned to work with the IPT to determine impacts to existing reports and the existing analytical reporting environment as well as identify new reports or new data to be added to the analytical reporting environment. Lead Requirements Analyst: Requirements analyst assigned to work with the IPT to define the user requirements.

Lead Application Developer: Software engineer assigned to work with the IPT to develop the software requirements and detailed design using the user requirements.

Lead Application Quality Control (QC): Test engineer assigned to work with the IPT to draft and conduct system testing. Lead Subject Matter Expert (SME): SME assigned as the primary contact and liaison to the SMEs on project-related information.

Quality Assurance (QA) Engineer: Quality assurance engineer assigned to work with the IPT to ensure the IPT is following established process and work with the IPT to update appropriate training documentation.

The same person often filled multiple roles, but I put the responsibility of overseeing the enhancement and IPT (somewhat of a project manager role) on the requirements analyst. It was the requirement analyst’s responsibility to include the IPT in the business requirements analysis sessions from the start and throughout all phases of the enhancement (i.e., software requirements analysis, design, test case design, test case execution, UAT). If a change needed to be made to a requirement, all the stakeholders were included, which left no room for someone to not know what decision was made. This minimized the effect a change had on the whole team.

Last but not least, once the process changes were identified and approved by the Government SMEs, we needed to select a tool. Appendix A (located online) provides the criteria derived from the newly defined process and was used to evaluate requirements tools.

As this point the requirements team started to initiate changes. The first step was to reengineer existing requirements, which gave us a baseline to start with instead of blindly starting with a new change request. We were able to identify a majority of the “do’s and don’ts” around building the user requirements through use cases and business rules.

WWhile it seemed we were on the right track and gaining momentum, we actually made the requirements worse! Complaints started coming in about the requirements, that they were too complicated and not easily digested, I could relate as the use cases seem more complicated than what we had before and how was I to develop test cases from these use cases, etc. Yikes! Where did we go wrong? The user requirements definitely were way too complicated. I had business rules spanning several pages. I had use cases with over 40 steps, not counting alternate flows. I had to take action and fast. I went to my requirement analysts with the complaints. Their argument was the ROSS requirements were complicated and this justified the track we were on. My requirement analysts had engrossed themselves in the detail and weren’t keeping the big picture or the use case end goal in mind. I needed information and facts to make a decision. After researching countless articles about use cases on the web, I finally came across a gem from none other than Karl E. Wiegers, Ph.D. From his article, Software Requirements Analysis: Five Use Case Traps to Avoid, I found the answers I was looking for. We had fallen into several, here is my analysis:

Trap #1: Use cases that users don’t understand
The use cases we were writing were convoluted and hard to follow due to a number of reasons which are explained in the next ‘traps’.

Trap #2: Too many use cases
We didn’t fall into this trap, but were definitely on the other side of the spectrum covering way too much functionality in single use case as described in Trap #3.

Trap #3: Overly complex use cases
“The general guideline is that the number of steps in the normal flow of a use case should not exceed approximately a dozen.” When I reviewed the use cases we had developed, it became apparent we were covering multiple functions in a single use case (e.g., Using baseball rosters, we had both ‘Creating’ and ‘Updating’ a baseball roster in one use case.) The argument for doing this was basically that the use case steps were the same except for ‘add’ vs. ‘update’. However, it caused confusion and complexity. While I agree it’s basically the same steps to add or update, the rules around when you can update vs. add are different, causing the use case to be split into 2 different paths with multiple alternate paths. We went back to the basics with only one function per use case allowed, and focused the function on Create, Retrieve, Update, Delete. This approach narrowed down the scope of each use case and brought the number of steps back into reason with 12 to 15 steps per use case. theworld | series theworld | series

Trap #4: Describing specific user interface elements and actions
“Imposing premature design constraints in the use case can lead to a suboptimal design, unless you’re adding a new capability to an existing application where the screens already exist.” It wasn’t only the requirement analyst’s fault that we were on the wrong track. The SMEs contributed to the problem as well. Too many times, end users tend to focus on the ‘how’ instead of the ‘why’ which is only natural. When trying to define business requirements for an application (and especially an existing application) it is very easy to fall into this trap. You start to think about how the software is laid out and this tends to drive the business requirements. So we had fallen into the trap of trying to define the ‘how’ in our use cases. We needed to refocus our efforts on the ‘why’, not the ‘how’.

Trap #5: Not using other requirement models
“Avoid the temptation to force fit all of your functional requirements into use cases.” This is a trap we had fallen into and it was the trap that was the toughest to get out of. We needed to evaluate when a use case was appropriate to articulate a business requirement and when it was not. Sometimes a simple state diagram, flow chart, or decision diagram was just as effective and much easier to understand.

I went back to the drawing board with the requirements team. Instead of starting with a brand new change request, we again took existing requirements and reengineered them. We had sessions with the other groups including the SMEs, Development, and QC and reviewed the changes. The requirement analysts took the feedback and incorporated them into the process. It took time, patience and determination but we were starting to feel some momentum again. And this time, we were on the right track.

Let’s take the baseball requirements from above and re engineer them into business rules and use cases, and whatever other form might work to best articulate the business requirement.

Baseball Roster User Requirements

Business Rules:
 
BRULE1: Baseball Rosters must have at least 9 players but no more than 15 players.
BRULE2: A player’s age must be => 7 years and <= 15 years.
BRULE3: A pinch hitter can substitute a player in the batting order at any time the game is not in active play.
BRULE4: A designated hitter fills in for the pitcher in the batter order.
BRULE5: A designated hitter does not play a position.
BRULE6: A lineup that starts with a designated hitter must end with a designated hitter. BRULE7: Player Stat

The previous use cases are just scratching the surface when defining the baseball roster and lineup requirements. This was to demonstrate how we incorporated structure around our requirements. In addition to what is stated above, we used Logical Data Models, Entity Relationship Diagrams, decision flows, use case diagrams, defined roles, etc.

By changing the requirements process to involve all teams and going from hierarchical to use case format, we were able to reduce the number of defects across the board. The IPT teams of Development, QC and the SMEs were able to give input on the changes going into the product which in turn minimized assumptions about the requirements. The client has the final decision on the change, but made the change with input from all.

Looking at a few metrics, the As Designed, Cannot Reproduce and number of defects fixed overall decreased over time.

It all starts with requirements. Ambiguous, incorrect, and/or contradicting requirements cause problems throughout the SDLC. With more time spent at the front end of the process and getting the requirements right, we spent less time in Development, Test and UAT. The rest of the SDLC started to go more smoothly without the major ‘gotchas’ we had encountered when the requirements were developed in a vacuum. We also were able to effectively respond to changes as they arose since the IPT was involved from the start.

The goal is to make sure that all your clients, whether internal or external, understand the requirements. And all people have a different way of understanding. Some need visuals, some need diagrams, etc. As a requirements analyst, don’t try to force feed a standard format to all your clients, figure out how to represent your requirements in various ways so all can absorb and execute their part of the process successfully.

Download Appendix A

Previous ROSS Article