Management requires continuous adaptation, interaction with the environment, modifying the approaches for maximum return. That’s why experienced test leaders collect variety of methods in their “testing practices” toolbox, and pull them out when needed.

In an ideal world, there is always enough time for expert testing, and enough people who do it well. In reality, there are lots of constraints. For example, the ‘time to market’ pressure or changing requirements don’t allow time for sufficient testing activities. The team might not have a tester solely dedicated to a project or the ‘developers to testers’ ratio doesn’t allow for thorough skilled testing. The major architecture redesign, completely changed business workflow or new UI framework introduces so many ‘unknowns’ that the existing test automation suite may not be enough to mitigate the risks.

For situations like that, I use a special technique called a “group test”. During group test, the whole team that participated in building the product spends an hour running various scenarios on the single production-like environment. Here, I use the word “team” to identify a group of people involved in developing a specific product: business analysts, designers, programmers and testers. Organizationally, the team members may report into a single manager, or be a ‘virtual’ or a ‘project’ team: specialists from different discipline departments that are developing a product together.

The key “group test” principles are: doing it together during a dedicated timeslot on one shared environment (ideally as close to production configuration as possible).

The group test allows receiving rapid high-level feedback on the quality of the software. There are situations where you need to get this particular information fast. Also, it introduces diversified testing maneuvers: everyone uses UI differently and will come up with their own ways of completing a certain function or task. Participants become very creative if they don’t follow detailed ‘step by step’ test scripts, but instead are given some assignments to complete. Such practice appears to be very valuable, as even skilled testers will not be covering all possible scenarios or all UI paths. Another aspect of this practice is to receive a broad feedback on the software usability. Yet another aspect is to exercise multiple functions and usage of the program in parallel. Such practice may introduce very interesting unexpected combinations or racing conditions that are not apparent during single person or single component testing.

Yes, you can see that the group test is not a panacea; and it is not a replacement for the strategic skilled testing. The bugs will be missed. However, there are two surprising benefits from using this technique: the team building aspect and getting all team members to become “quality advocates” for the product. Here are my observations as to why. Rule #1 for the group test is – everyone participates. You really have to have a VERY valid reason not to. Team leads and managers are not exempt. During this very special hour, a communication channel is open: people exchange observations and issues, while at the same time exploring the application. By coming into contact with the system as a whole through hands-on activities, participants acquire better understanding of the product and the services it claims to offer. Such experience will eventually make each team member a quality advocate, as they will be more conscious in their later work of the goals such product was designed to accomplish. The bug advocacy becomes easier as well.

To Summarize:

What is it

  • A group testing activity where the whole team (and sometimes stakeholders and customer support) test various scenarios at the same time onone production-like environment

Benefits

  • Great for team building
  • Everyone is getting exposure to the whole product
  • Rapid feedback on the quality of the software
  • More eyes and hands on the UI
  • Whole system integration high-level testing

Limitations

  • It’s not skilled testing (the bugs will be missed)
  • It must be planned ahead and coordinated
  • You cannot do it all the time; people see it as a distraction

In my practice, I use group tests a lot. To me, it’s a good way to get the whole team to learn the product and become quality advocates for it. Oftentimes, developers only know pieces or components of the product. Group test is a good way to expose the whole offering to each team member. At various times, I mitigate the risks of the ‘unknowns’ through a series of group tests. At other times, I have a group test at the “feature complete” stage in order for everyone on the team to get exposed to the new features and go through validation of various workflows.

The group test is an event, and it needs planning. Here is my “recipe for
success”:

Put an hour for the group test on team’s calendar.

  • Deploy the build beforehand, and smoke test it. This way you will not waste time for many people later. If the build is no good, reschedule the group test. 
  • Compile the list of scenarios to cover what you need, and split them into areas depending on the number of people participating. This way, you will ensure the “must-have” coverage, and avoid duplication of effort. Usually, after getting through assigned scenarios, people start exploring around, and discover other things. 
  • Prepare the data you need for the group test. Production-like data is ideal in many cases. However, you need to decide how you anticipate using that data. For example, you would want to avoid the unrealistic scenarios, where the specific data access is limited to a single person in production environment, and you have ten people hitting it during the group test. In this case, you may want to replicate the data into several accounts, etc. 
  • Prepare the logins for everyone. 
  • You may want to have participants use the recorder tool during the test. There are few reasons for it. Reason #1: some bugs may not be reproducible in one click, and it could take a significant amount of time to discover the exact replication steps later. Reason #2: majority of the group test participants are not skilled testers or don’t have an extensive knowledge of the product. They may not know how they arrived at the bug. Having the tool recording their actions will provide a track of replication steps for developers. Prepare and distribute the instructions for the recorder tool beforehand. I suggest going to everyone’s desk to ensure the tool is working. Checking it during group test will take time away from testing. 
  • Right before the group test, send out the email to the group which contains: 
    • The login instructions (URL or Installer location, usernames, passwords) 
    • List of assignments for each participant
    • A reminder to turn on recorder tool at the login page 
  • Once group test ends, announce it and request a list of issues from every participant with as much information as they could provide (login ID, time when incident happened, replication steps, screenshots etc.). Go around and ask everyone personally if you don’t get a written response.
  • Give the list of issues to Product Owner and/or Tech Leads for prioritizing.
  • Repeat as see fit.

Here are some variations of the group test performed by other teams in my
company:

  • Creative ‘crowdsourcing’. One team developed a mobile application, and asked everyone in the company to install it on their own mobile devices and run through a few basic scenarios. They followed the ‘recipe’ above.
  • Internal marketing. The “company-wide” group test was a good way to re-introduce the major money-making product to all employees. The event required a lot of coordination; the SMEs and testers were allocated to each floor assisting with the installs and answering questions. Before the event, training sessions were held to introduce everyone to the basic concepts and functions of the software. Having a hands-on practice later on was fun and an educational experience for all.
  • System integration testing. A team, whose product has an upstream dependency relationship with many other products in the company, runs collective group tests for each major release, inviting other teams to participate and to run their own applications and scenarios in parallel. Oftentimes product support teams also participate in group tests using their monitoring tools, thus learning new features while practicing production-like troubleshooting.
  • Testing system recovery and user experience after some planned failures or under load. You can guess how the users would react to system malfunctioning; this technique allows having real user feedback.
  • Testing roles. Assign every participant a ‘secret’ role, such as “the novice user”, “the hacker”, “the gamer” (a user trying to manipulate the system for their own benefit), or any other personas who use the system to fulfill a certain goal. Such approach produces very interesting feedback, as the team may discover unexpected ‘holes’ and/or validate some business assumptions.

I hope there are some ideas to get your team started on how and where to apply this powerful technique. The team should certainly adjust it to better satisfy its own needs. Happy testing!


About the Author

Anna Royzman Anna Royzman is the test lead in a cross-functional product development team that creates and delivers game-changing software in the financial industry, where “quality” is as important as the “time to market.” With a wealth of experience in the testing and quality assurance field, she has developed unique perspectives on quality leadership during the past decade. Anna organizes discussion panels, leads SIGs, creates workshops, and speaks at conferences to promote the value of skillful testing and the whole team approach to quality. Anna is an AST member and passionate supporter of the context-driven school of thought in software testing. She writes about her latest ideas and personal insights on quality and testing on her blog.