A Roadmap to Streamlining User Acceptance Testing

A Guide to Streamlining User Acceptance Testing (UAT) for $100 million dollar plus IT Projects focuses on the application of leading-edge testing management tools and techniques to streamline and manage the UAT phase of testing. Inspired by the author’s experience as a test lead for a $200 million dollar .NET implementation at a State Medicaid Organization, many of the steps outlined in this guide may apply to your market as well. The steps in this guide are designed to focus your organization on the critical success factors of UAT management while also introducing many of the critical elements often underutilized in leading-edge testing management tools.

By definition, the UAT
User Acceptance Testing - phase of testing provides your end users with an opportunity to conduct a series of tests that ultimately builds confidence in the system prior to Go-Live.

In a perfect world, the UAT phase is an opportunity for your end users to isolate lingering bugs from the system and to execute regression testing that may impact system usability, while also learning the new system. It should be straightforward and clear cut with little question on how to properly test the new system. However, the practicality of UAT unfolding by the book is farfetched – ask anyone who has been a UAT tester for a State-wide Medicaid Modernization Project, or another $100m+ IT project for that matter.

The reality is that your staff is busy and pulling them away from operational activities to fully test the modernized IT is a challenge in itself. Whether it is a modernized Medicaid system, or a newly implemented ERP (enterprise resource planning) system like SAP you’re implementing – the challenge of time-sliced human resources is very real.

In addition, knowledge gaps naturally exist as a result of the paradigm shift occurring across the technology landscape. The rapidly evolving IT landscape includes architectural changes such as Service Oriented Architecture (SOA), which inevitably impact the nature of software testing. Although your staff has deep experience in testing legacy and mainframe technology, they may need a refresher course in emerging technologies like Enterprise Service Bus (ESB) technology or XML messaging standards. Your staff may also be accustomed to using a homegrown test management tool, but the solution integrator implementing the new IT has chosen a different test management tool. Guess what? You need to anticipate this ahead of time with scheduled training or your UAT staff will be spinning their wheels.

The reality of the situation is that your staff will be responsible for much more than the isolation of lingering bugs not corrected in system, regression, or interface testing by the solution integrator. The UAT management learning curve is very real and can be a slippery slope without the necessary gear to climb it efficiently and effectively.

By applying the right people, processes, and tools to UAT from the start, your team will be ahead of the curve and well on their way to the top. A Roadmap to Streamlining User Acceptance Testing (UAT) on $100m+ IT Projects was inspired by real-world experiences. Each step includes a practical and simple hint to ensure UAT success!

1. Develop a comprehensive “UAT Desk Aide” that clearly outlines the people, processes, technologies, and timelines associated with User Acceptance Testing.

• UAT Test Plans delivered by the solution integrator can often be cookie-cutter in nature. Sure, it attempts to delineate roles and responsibilities, but a comprehensive desk aide for your UAT testing team gets personal and drives productivity.
• Your desk aide should outline all the staff on the project with the solution integrator counterparts. It should outline your key processes, such as the defect resolution (triage) procedure, the lifecycle of a UAT test case, and the ‘Top 10 Execution Guidelines.’ When a staff member within your organization is pulled from their day-to-day job to conduct UAT, this simple (yet effective) desk aide will dramatically cut the learning curve.

Real-World Hint
Statistics show that writing a “Top 10” list will get more attention than writing the same content out in a paragraph. With this in mind, develop a “Top 10 Test Execution Guidelines” list and include it within your UAT Desk Aide. Your testers will appreciate the thought that went into it and they are more likely to read it – and more importantly, follow it! Also, don’t be afraid to use the bookmarking functionality within Microsoft Word to ‘guide the reader’ when they reference the UAT Desk Aide in electronic format. By bookmarking sections within your UAT Desk Aide and then hyper linking to them in others, you can easily refer your testers back to important information. Simple things like this go a long way.

2. Candidly Discuss Testing Approaches with your UAT Testing Team

• It is common to have a wide variation of software testing knowledge across your newly assembled UAT team. You may have a mix of individuals that were taught different testing methodologies. Some are resistant to change, while others are open to new ideas. Expect the unexpected and have a plan to have open discussion with your team about the testing approach decided on by your executive management team.

Real-World Hint
Prepare yourself for resistance at the onset of UAT testing. Many of your testers are accustomed to their own way of testing. Smoke testing for example has been used by many legacy testers. The problem with this type of approach in the IT modernization arena is that it’s difficult to re-produce system errors if the tester fails to follow a pre-defined script. Another example is automated testing using advanced test management tools like HP’s Quick Test Professional (QTP). Some solution integrators leverage advanced testing tools like QTP for system modernization projects, whereas others do not. It is the preference of the executive management team who made the decision months before the start of UAT. If you accept the decisions early-on and openly discuss the concerns with your testers by framing it in a positive light, their resistance to change will subside and the focus will be shifted on testing productivity. Explain the pros and cons to every method of testing to your team when you’re met with resistance.

3. Collaboratively establish a framework to initially gauge and measure UAT testers’ l 3 evel of QAT expertise.

• Identify at least two individuals on your team with the aptitude to effectively gauge the knowledge-base across the entire UAT testing community. These resources should be recognized experts in the test management tool that your entire UAT team will be using to test the modernized IT system. Task these individuals with informally surveying your team to understand how efficient they are already in the test management tool. Once completed, this will be your baseline to build and measure progress against when you start conducting JIT (just in time) training sessions.
• Document every UAT tester’s functional and technical ability in a private offline Microsoft Excel Spreadsheet. Are they policy experts, technical testers, or generalists that played an important role in the management of your managed care department? It is very important to understand each individual’s strengths and weaknesses to deploy them in the right direction. Take for example the claims adjudication process within the new IT system. You will need a UAT tester with experience in the old system to understand if the test cases written by the Solution Integrator sufficiently test the ‘edits’ and ‘audits’ of your state’s Medicaid policy. Deploying a technical resource to test this functionality wouldn’t do you any good and could put the project at risk. That technicalminded tester should be one of your interface testers or help test the security of the new IT system.

Real-World Hint
Be careful with assumptions on the technical aptitude of your UAT testers. Just because you have a tester with many years of experience in the IT legacy environment, does not always mean that they will be equally effective in testing the modernized IT system. The paradigm shift in system architecture, especially the SOA concepts introduced in recent years, has left knowledge gaps in many seasoned IT personnel. An important step is to effectively communicate the paradigm shift to your testers by pointing them in the direction of white papers, articles, and tutorials on the shift in system architectural design. By explaining the benefits of moving to a different system architecture that aligns closely with federal funding while also driving IT flexibility, your staff is more inclined to jump on board quickly. Fundamentals are the key. Your testers aren’t the system architects!

4. Make communication and coordination the lifeblood of the UAT team

• You’ve heard it before: Communication is the lifeblood of successful projects. By embracing multiple channels (e.g. MBWA, Staff Meetings, Instant Messaging, and ECM) to facilitate communication, you will ensure that your lifeblood is nutrient rich and thriving.

Real-World Hint
Be careful with an overreliance on communication plans. Although it is necessary to define formal communication with various project stakeholders, it should never take the place of the intangible aspects of communication among your UAT testing team.

5. Bridge the gap between test management tools

• Let’s face it, not everybody prefers the same tools to finish a job. One of your project managers may prefer Microsoft Excel to track tasks, where another swears by Microsoft Project, and yet another is a bit more advanced and uses SharePoint lists with automated workflow. The point is that there are multiple paths that lead to the same destination. The same applies to test management tools like HP Quality Center, IBM Rational TestManager, or Oracle’s e-Test. Sometimes management decisions aren’t necessarily in the best interest of maintaining a ‘centralized repository’ of testing artifacts. Anything is possible though and there is a solution. To combat varying tastes in test management tools, you will be required to ‘bridge the gap’ between the two (or three) by building custom interfaces. As a UAT test manager, make sure you understand the implications of each interface, when they run, and how the different systems synchronize.

Real-World Hint
Developers and QA personnel (e.g. UAT Testers) don’t always agree on the use of one test management tool over another to track defects, test plans, test cases, etc. The bottom line is that your organization will need to decide on one system of record for test management. If your QA Team prefers HP Quality Center, but the solution integrator prefers JIRA (for example) for internal task tracking, it becomes increasingly important for you to build a bridge between the two applications to maintain transparency with the system of record. Without the bridge (interface), it will quickly become an administrative nightmare. If your UAT team is going to build custom test scripts in a tool like HP Quality Center, make sure they link each script (or test cases) to its appropriate business requirement. The linkage may be a critical component to ensuring that the interfaces to other ‘system repositories’ capture everything in the daily batch cycle.

6. Enforce strict documentation control and specify a central repository

• Lack of documentation control across repositories can create confusion among your UAT testers. Test cases written by your UAT testers who were not directly involved in the Joint Application Design (JAD) sessions with the solution integrator may not capture decisions made if the documentation was not updated appropriately. What this means is that your testers may be referencing outdated business requirements, design documents, or change orders. Furthermore, your new UAT test case reflects the outdated documentation, making the test results non-applicable to the modernized IT.
• Enforce strict documentation control by holding “overview sessions” on the documentation repository of choice, whether it is Microsoft SharePoint, or Documentum eRoom. All staff members should be trained on how to use the “check-in” and “checkout” functionality of the software. Version control needs to be closely monitored, so consider tasking one or two of your testers with the task of closely monitoring the process.

Real-World Hint
An obvious outcome of large IT modernization projects is the generation of hundreds, even thousands of project artifacts, including: design documentation, file layouts, training presentations, e-mails, memos, etc. Rolling the administrative aspect of managing this documentation under your Program Management Office (PMO) is a good idea only if the personnel in charge of ‘documentation control’ have an adequate understanding of modern enterprise content management (ECM) principles. By leveraging industry best practices such as those taught by AIIM’s leading-edge Certification Programs, you can effectively streamline the content management aspect of your IT modernization effort. UNICON Tech, employs certified ECM experts that help agencies rapidly develop design and deploy processes that streamline and enforce strict documentation control.

7. Train Your Testers to Use a Consistent Approach in Test Case Design

• A test case is only as good as the steps that make it up. It is common to have many different test case designers with their own style. One tester takes the minimalist approach, while another provides explicit directions within their steps. The risk with the inconsistency across test cases is that your modernized IT is tested at different levels. A well written test case will 100% absolutely ensure that the functionality of your system is working properly, whereas the test case designer who took the minimalist approach is leaving too much ambiguity for the actual tester. There should be no question in the tester’s mind in terms of what to do next. The test case should clearly spell it out – step by step. Pay attention to levels of consistency across your test cases and take corrective action immediately if you notice wide variations. Call a meeting with your testing team and reiterate the importance of consistency.

Real-World Hint
Build steps into your UAT test cases for the tester to validate/check the design documentation on the centralized portal (e.g. SharePoint, eRooms) used between your organization and the solution integrator. Whenever a change is approved through your Change Control Review Board (CCRB) process, the solution integrator should circle back to the documentation on the centralized portal to update the design documentation. This doesn’t always happen as planned. To catch this and improve your organization’s documentation quality, have your test case designers add steps into the test case to check the documentation as they are actually testing the modernized IT in UAT. It will add a few steps to each test case, but the long-term pay off is that the quality of your documentation is much higher after the modernized IT goes live into production.

8. Conduct just in time (JIT) training sessions on test management tools

• Some organizations provide software training on the test management tools weeks or months before the UAT tester is scheduled to begin testing the Application Under Test (AUT). The AUT in this context is your state organization’s modernized IT system, but what do we mean by the test management tools required for the AUT? Every IT modernization project includes a test management tool like HP Quality Center, IBM Rationale TestManager, or HP Quick Test Professional. What’s more is that state agencies and solution integrators often deploy ECM applications like Microsoft SharePoint, Documentum eRoom, or even proprietary knowledge repositories (e.g. portals) to hold project artifacts such as design documentation, test plans, change order requests, and requirements. Most of the UAT testers tapped from various parts of your organization have likely never used these tools to conduct software testing. In fact, most of them are accustomed to homegrown applications built over the years. Believe it or not, the UAT learning curve starts with the test management tools mentioned here. Prepare for the climb by grooming 2 or 3 internal UAT testers with focused training on the test management tools your organization anticipates using. The ROI in training a few of your UAT testers will pay big dividends as they provide your entire UAT team with JIT ‘over the shoulder’ support in the heat of UAT.
• The same concept of JIT training sessions applies to the UAT (the modernized IT system). Very few of us have photographic memories. Ensure that your solution integrator of choice is prepared to provide JIT training to your UAT testers in a timely manner. You will likely have groups of UAT testers that are responsible for testing various aspects of your modernized IT such as Electronic Data Interchange (EDI), Claims Processing, or Provider Enrollment. Each of these subsystems may have a different User Interface (UI) that requires navigational training. Your solution integrator should be on stand-by and ready to provide JIT training to your UAT testers right before they are scheduled to begin testing a particular subsystem. JIT is relative, but a good rule of thumb to follow is that training should be delivered to your UAT testers no later than 1 week before the scheduled date to begin testing a specific area. Experience has shown that anything further out isn’t effective. 

Real-World Hint
Remember that the Return-on-Investment (ROI) for training your organization’s testing staff on the test management tools needed for UAT can’t always be financially measured. Reframe the ROI in terms of energy invested now versus the avoidance of more energy invested in the future. The act of anticipating the future needs of your UAT testers by investing in a few to develop subject matter expertise will translate into less expended energy as UAT is in full swing. The “Power Users” that you develop upfront through formalized training will be critical to the ramping up of new UAT testers and the on-going over the shoulder support required for your team to be successful.

9. Manage By wandering (or walking) around (MBWA)

• According to BusinessDictionary.com, MBWA is an unstructured approach to hands-on, direct participation by the managers in the work-related affairs of their subordinates, in contrast to rigid and distant management. In MBWA practice, managers spend a significant amount of time informally visiting work areas (i.e. cubicles) and listening to the employees. The purpose of this exercise is to collect qualitative information, listen to suggestions and complaints, and keep a finger on the pulse of the organization.
• It is common for Test Managers and Test Directors to become distant and rigid in their management style. The reason for this can be a result of their personality, the job pressure, or a combination of both. Let’s face it, the modern day Test Manager typically oversees a core team of 10 to 30 testers and up to a 150 subject matter experts throughout the organization who provide 4 to 8 hours of expertise per week. Often times, the UAT test management position is stuck in the middle. Not only are you answering to the executive management team running the IT modernization project, but you are responsible for providing guidance, motivation, and overall direction to the entire UAT team.

To conserve energy, it’s sometimes natural to slip into a mindset of distant management and rigid structure – selectively replying to e-mails, occasionally swinging by your core tester’s cubicles or testing labs, and pointing to an outdated test plan for reference when your testing staff has questions. This is exactly what you do not want to do as a UAT manager.

Being and staying successful requires energy – a lot of energy. To keep your team successful and ensure the momentum of the UAT testing effort is moving in a positive direction, practice MBWA daily – if not hourly. Motion creates emotion –successful Test Managers do not lose sight of this concept. By walking around and actively engaging your testers, you build trust, confidence, and a productive working relationship with your team. If your testers only have your ear during the weekly staff meeting, they become guarded and aren’t willing to divulge information that can ultimately make or break the quality of your modernized IT. Many testers aren’t comfortable speaking in public forums anyhow – whether it’s the nature of their analytical minds, or the fact that public speaking is America’s number one fear. Remember that most of your testers aren’t accustomed to speaking in front of a large group. To combat this widespread fear, engage your testers on a more personal level by practicing MBWA. 9

Real-World Hint
When you’re not MBWA’ing (everybody has to sit down once in awhile!), embrace instant messaging collaboration technology such as Skype, AOL AIM, or Novel GroupWise Messenger to communicate immediately with your subordinates. The purpose of leveraging instant messaging technology is two-prong. First of all, it cuts down on e-mail traffic. Secondly, it makes your team more productive through increased collaboration. As an aside, you may consider providing coaching on the effective use of instant messaging technology.

10. Strive to keep it simple by distilling UAT execution raw data into dashboard traffic lights

• Software testing evolves and matures over time. Dashboards and real-time KPIs are the hot items in today’s testing environments. Through custom UAT dashboards, you’ll have a traffic-light display at your fingertips that will show the health of UAT execution against pre-defined criterion. One example of a business rule tied to a traffic-light might be the number of defects submitted in a given day for a specific business process area. Within the business rule of the trafficlight, you can define thresholds as it relates to green, yellow, or red. If the defect count is less than 2, you would see green – but if it is above 5, if you would see red. Anything in between is yellow. The possibilities are endless for what you can monitor across your UAT team. As a UAT manager, there isn’t much time to analyze the raw data of test execution. A UAT dashboard also makes it easy for you to analyze historical trends, in addition to analyzing metrics on a week to week basis. And if you are detail-oriented, you can always drill-down on a specific KPI to see the raw data.

Real-World Hint
As the complexity of information technology increases, it becomes even more important to implement dashboard KPI monitoring tools. From a UAT perspective, you can quickly monitor defects, requirements coverage, and test execution results from a trafficlight enabled dashboard. Many leading-edge tools contain dashboard functionality that contain pre-built KPIs and drill-down functionality to help you understand and resolve UAT project issues quickly. Make sure you are fully leveraging the functionality of the test management tool employed across your organization. Work with your solution integrator to customize a dashboard that includes the UAT business logic you require to make decisions quickly.