It's great to test simple GUIs. You click,type,click, see the wrong answer in (browser), then go find a programmer, open up instant messenger, send an email, file a bug, or whatever you do.
But sometimes you get an answer back like "We think it's fixed. Get the latest build."
Sometimes the thing you were trying to test was the fix that was supposed to fix this very issue ... and it isn't.
So is it really fixed?
Who knows?
Should you get a new build and find out?
... have you ever spent all day getting new builds and finding out?
Okay, maybe not
all day. But I've certainly got the "try again" or "that's weird" replies enough that meaningful chunks of my time were wasted.
It's an awkward situation.
Your Options
You
could keep getting the bad builds, installing them, and seeing the software not work, until someone takes notice. At which time, someone else might just install the bad build to see the issue fixed.
Except, of course, it isn't fixed. Then the issue will quickly get resolved.
You could make a stink about the fact that,
no, frankly, it is not fixed. Perhaps the developer missed a commit?
That will solve the problem sooner - if you're right. It's usually possible that you are wrong; that this build fixes the problem, or else you needed to select slightly different combination of values on your installer, or the data file you ran in had a strange combination of events and yes, in fact, it does work the way it's suppose to.
The more complex the setup, the more complex the tear-down, the more complex the interaction of object, the more likely this is true. Which is the rub -- if you doubt yourself, it will take more time to re-test it again, only to come to (likely) the same conclusion.
Arg.
Here's one more thing to do
Get someone else independent to verify the problem.
It could be another tester; it could be a developer who happens to work early, like you, and has just a tiny bit of bandwidth. Just get one person who isn't you, and didn't personally build the software, to take a look at it and see if they think the problem is real.
If the other person finds the issue is fixed, well, great! That's why you wanted them to check. If they find it's a problem, then you can conclude that it
seems reasonable to believe that the defect still exists in the current build, and to escalate the problem. If, after you escalate it, it turns out that there really was some strange condition you didn't realize well, that's okay too, because
two people experienced the problem, it seems to be a reasonable mistake to make.
Sometimes, in software testing, if a problem actually exists or not is hotly contested -- to the point of the "oh yes it is" / "oh no it's not"
bugs bunny cartoon. (Look at 2:40 in for something like that - or, better yet, 4:09)
When that happens, I try to bring someone else in. Sometimes, we get it wrong. Most of the time, between the two of us, we get it right. When it happens a lot, I try to figure out what's going on.
Increasingly, I find those little techniques (sometimes obvious ones) to kick-start us out of an unproductive place are extremely important.
And they are things we don't talk about enough.
What are your techniques?
If you've got several and they are worth sharing, I know a magazine that's always looking for articles.
Just sayin'.