Earlier I introduced the test challenge, along with this picture:
A simple look at the potential test cases:
7 basic events x 3 types (all, followed, conversations) x 2 networks (all or a specific one) = 42 test cases
Yes, 42 different cases. (Apparently, 42 really is
the answer to life, the universe, and everything.)
But that's just a start; that is just verifying that when I do things, they show up, and they also show up when filtered. It does not include evens done by others (+42) - and then events that should /not/ show up because they do not fit the filter. For example, when filtering by my conversations, we would want to check that all event types that are
not conversations
do not appear. Depending on how you count them, I would say that is at least 84 more test cases. So currently, 168 potential test ideas.
Then you have different browsers - IE8, IE7, IE6, FF3.5, FF3, Safari3, and Safari4. Depending on how much risk you are willing to take (hey, are FireFox 3.5 and 3.0 really all that different?) we've got somewhere between four and seven browser variants.
That is something between 600 and 1,000 test ideas.
But we are just getting started.
The tests above are basic confirmatory tests. They don't deal with special characters, internationalization, or boundary conditions. For example - say I send a message that is 140 characters of pure text with no spaces in it; I suspect the browser will not be able to chop off the word and that it will "spill over" the right side of the widget. (Turns out, it did.) Or can the widget handle UTF-8 - extended character set characters? Turns out, it could not. Also - what if someone else sends a signal or updates a page while I am looking at the widget? Do I see the change? (Turns out - eventually. The widget polls the server every 30 seconds. This turned into a load on the server for muliple users, and we had to increase he delay between polls.)
So, as of now, I'm only testing the basic functionality - not looking at the widget settings, or trying to move the widget within the container, or next/previous links - and I have a tremendous number of potential tests. Far more than I could realistically test in the three to six hours for which I am expected to test this widget.
And automation? Any time spent writing computer-assisted tests with something like selenium is time
not spent actively testing. In three to six hours I could only test a tiny fraction of those 164 test ideas. Yet if I don't write any automation, in two weeks, I'll have to test it manually again, plus have new features to test.
Bummer.
The next option is to go to management and say "We need to peel off three developers for the next two weeks to write automation. I know, I know, that will mean they don't produce working software next iteration, but I
read in a book that that iteration isn't done until testing is done."
How do you think that will go over?
Even if we got the funding to automate those 168 test ideas, they won't catch rendering errors (the text falls off the screen), and sadly, if the UI has changes, they will creates a significant maintenance burden.
Here's what Matt Actually Did
Before the developers began coding on the story, I had a chance to review it and work on the story-tests described in my
second blog post. These story-tests are important because they are concrete examples. The challenge I find with story-tests is in getting enough stories to provide good examples to guide development, increasing quality before "code complete" - but not too many. With too many story tests, I find that human nature takes over, and the developer skims over the tests, saying "oh yeah, it should do that" without physically doing the testing. (After all, the developer will think, that's the tester's job, right?)
After story review,
the whole team does a kickoff, to make sure everyone is on the same page about what the story should do. This was the final chance to clarify expectations, to put the story into our own words, bounce it around, and see if anyone says "oh, I didn't realize you meant that." (This overcomes
document under the door syndrome.)
Then the developers write the code. Time passes ...
Code complete; story is moved to QA. I build the environment (5 minutes). I spent about an hour doing exploratory testing, simulating three users - one in FireFox 3, one in Selenium, and one in IE7, and found the bounds issues above. I also explored logical concepts; for example, conversations only consist of page edits, and the 'code complete' version both allowed you to select signals and actually showed some. Then, I created a basic selenium RC test, demonstrating that signals and page edits do appear when a user creates them. I added to the test every iteration for the next few months, until it is now modestly robust. Then I logged in as a fourth user, who only overlapped with the IE7 user by one workspace, and tested in Internet Explorer Six. Thus I say the IE7 user's actions, but not other users - and only the profile actions and actions within the workspace we overlapped in.
Then we got the software up to our staging server, and ran the entire company on staging for a week prior to releasing the software to production. Thus I was able to test with real-world data, logging in as different users and asking "does this look right?", then exploratory test on staging.
A few other techniques to mitigate risk:
Create a staging clone and test update on it.
Log and monitor. Our software creates a wiki page with log output, sorted by either number of calls or duration. We can mine this output for most frequent operations (test those first) or slowest operations (mine for performance issues.) We can also log and monitor production, monitoring as a
kind of performance test. Assuming that staging has a much larger use per user than production use and that the activity widget will have a slow ramp-up adoption curve, this combination may be "just enough" load testing. (Perf testing is implied throughout; the difference between service level expectations and service level capability is likely a separate blog post.)
Actively engage the business users in the test process. We have a very simplified bugzilla form our employees can fill out in order to report bugs and a defined, non-painful test process. We train our business users; if Joe files a bug titled "the software is broke" with no text, we consider that a training issue, not a people issue.
Combine the test suite with the application. Our test are expressed as wiki-pages; the test suite "pulls down" the page and then executes it as a very high level programming language. In this way, we test the app while we are writing our tests! Moreover, if I know what the test suite does, I can log into the app after a test run - which is roughly 12,000 GUI clicks and inspections - and look to see if the activities of the wikitester user are documented properly; then I can filter to see if they appear.
Regularly rotate browsers. We do our work in different browsers and different browser versions; combining that with creating tests in our application creates a level of automatic coverage.
So that's it
The big games of testing, as I see it, are coming up with the test ideas, selecting the most powerful test ideas to run in the time allotted, and summarizing the results in order to provide information and make recommendations to decision makers. (And, when bugs are slotted to be fixed, selecting the appropriate sub-set of tests to run to gain confidence that the fix 'took' and didn't break anything else along that way.) In that way, testing is both an investigative activity and a risk-management activity.
Likewise, the challenge in the blog post is summarizing the strategy in a way that captures the essence of the work in a short period, without leaving important bits out, and, likewise, without putting everything in and putting your reader to sleep.
Still with me? :-)
I'll be talking more about this Oct 23rd at STPCon in Boston, in my session "
Testing the user centric web.
Still to come: Announcing the winners.