In graduate school, we learned a little bit about testing, and it was the classic stuff.
You've probably heard of it -- take the requirements, turn them sideways, create test cases, execute those test cases. The
interaction design folks and the
Acceptance Test Driven Development folks take this a bit further, combining requirements, test, and design into a sort of simultaneous process, so that what pops out is your requirements, examples of someone using the software (which is really a test turned a different direction), and maybe some of the user interface.
But what if we looked at testing in a different way, asking not "What is the (right) accountable way to test (according to some process)" but instead "how do we find bugs -- and how can we accelerate that process to try to make the most important bug-finding happen before we release the software?"
To do that, I'd like to tell you a story about a bug I found
today.
It all starts with a feature called the profile. Now a profile in Socialtext is very similar to a profile in something like Facebook -- it shows everything about one person in the system. Their name, contact information, tags, and what they've done lately. Here's an example for our demo server:
Notice that the profile events are all about the person. Seems pretty obvious, right? "Showing All Events from Me within (account)"
So this morning I was reading our internal blogs, mostly about the company face-to-face meeting last week. I read a comment by one person, who I'll call Sarah. Sarah impressed me at the face-to-face, so I opened a new tab to tag her. I did, and someonething felt ... wrong. I kept seeing my picture in her profile stream. This made sense, because I was the one tagging /her/, I had left a comment on
her blog ... but still ...
I went ahead and made my own blog post, then went back to the tab. Sure enough, my blog post was now on her profile too -- even though it had nothing to do with her.
On a hunch, I went into a test environment, created two browsers, logged in as userA, looking at my own profile, userB, editing pages. I waited three minutes and -- BLAMMO -- userB's edits were showing up in userA's profile.
It seems that the profile is restricted to the user you are viewing, but
new events are not restricted.
Overall, this is not a terrible bug; unless you have thousands of users in your account, it is unlikely you'll hang out on a profile page long enough to trip it. I mostly tripped it because I had the page open for an extended period of time in a different tab.
Would your test strategy find this?
Think about requirements in the form "User A's actions show up on their profile." How likely is it that the requirement say "... and User B's actions do not"?
Even if the requirements are written that way, do they cover simultaneous events, the kind that come in after the profile is being viewed? (Such as "... and as the user continues to view the tab, he sees user A's new events over time and no other events.")
In my experience, most folks just don't write requirements in this way. So test cases pulled from requirements would not be able to uncover this sort of defect.
Likewise, think about an ATDD-style "example": User A edits a page, sends a signal, tags a different user, and is tagged, all of these show up on his profile. User B does some independent actions, goes to profile, does not see UserB's actions.
Would that kind of example ever find this kind of bug?
Now thing about the typical automated-test tool: Selenium, WebDriver, Watir, take your pick - most of them work from a predominately single-browser-tab model. If you only have one browser tab active at a time, could you
ever find this bug?
Other Ways to Find Bugs
There are other ways to find bugs: You could do exploratory testing, you could
try to do an actual project like your customers would, you can do an extended beta program or roll out features to certain key power users.
That's it. No prejudice, no pretense, I just wanted to describe a bug I found today. Do you have bugs like that, or other categories of bugs, for which your test strategy turns a blind eye?
You can find out using some simple queries with your bug tracker, a little spreadsheet work and common sense, probably in
about an hour. If you do, I suspect what you find might make for good
fodder for a rollicking good brown-bag discussion.
If you do, let me know how it goes. Comments welcome.