I get in to the office a little early, before everyone else. I will have some time to settle in before things get crazy. I’ll have some quiet time to plan and organize my schedule for the day before the mad rush at the coffee machine. The masses start shuffling in at 9am. By 9 or 10am everyone is in the office.
There is at least one cluster of people in each aisle asking each other “How was your weekend?”
I never understood what the big fascination is about what other people did on their weekends, but I play along.
The 9:15 team meeting starts. The project manager lets us know what’s coming up this week.
“We’re getting a new release into QA today. This is a minor release, but it has a feature that the business was asking for a long time and it’s our highest priority now because Compliance is requiring it. It needs to get out there by next week.”
OK, no big deal, I say to myself. We have been testing previous versions of this application for awhile, and we have the testing process down to a science. First we do the smoke tests, and then check the new features. When we’re done doing the functionality testing we move on to regression and performance tests. On a major build it usually takes 2 or 3 weeks, but this is just a minor release, only one change. We can probably knock it out in a few days. No problem, we’ll make the deadline for sure.
Meeting over, I go have a chat with Kathy, the build manager. She is in charge of the test environments, maintaining the change control system and compiling and integrating new builds.
Kathy says, “Oh, it’s just a minor release? OK. We don’t have to rebuild everything, just overwrite a few files with the new versions and reboot. It should be ready in about half an hour or so.”
My response, “OK great, just let us know when it’s ready in QA and we’ll get right on it.”
I go back to my desk. I take a closer look at this new feature that they’re adding. Any documentation? Release notes? I find an email that I didn’t have time to read before, that looks like it has some relevant feature information. Looking at it, however, it doesn’t make much sense to me at all. I need to have a conversation with the developer. I walk over to his cube.
Me: “Hey Eric, you got a few minutes to explain this new feature to me?”
Eric: “Uh, yeah, sure.”
Me: “OK, hang on a second.” (I take out my Blackberry, and turn on built in audio recorder.) “You don’t mind if I record this? Just in case I miss something I can refer back to it, without needing to disturb you again.”
Eric: “Oh, good idea. Sure. Anyway, see here, this is what the business wanted, so I implemented it like this….”
10:20
Eric’s explanation was very helpful. Now I have a good understanding of what the feature is supposed to do and how it works. And I have a whole bunch of test ideas scribbled on my pad that I can’t wait to try out. I already have a good idea of how I can attack this. Feeling jazzed, I walk back to my desk.
It’s been about half an hour but no email from Kathy yet. It’s probably taking a bit longer than she thought. I’ll cut her some slack before I nudge her – she’s generally reliable and she’ll probably sort it all out soon. There’s no sense in pressuring her for no reason. I go to the other testers in the QA lab to share what I learned about the feature from Eric.
“Hey guys, did you see that email from last week with the product information? It really didn’t make any sense to me but I just got this from Eric. This is what the business wants, and this is how it’s going to work. I bet we can break it if we do it like this.” I explain to them my testing ideas and ask them what else they think would be powerful tests that we could try.
Just because they are consultants, it doesn’t mean that their input isn’t useful. I can’t understand why other leads treat the consultants on their team like robots, just because they get paid differently than full time employees. It boggles my mind. One of the hallmarks of our test team is the informal group brainstorming we do to generate test ideas during the test cycle.
Raj pipes up, “Remember the last build? When we discovered late in the project that we needed to test it with user logins with the different permissions settings? Do we have to do that this time too?”
“Thanks for reminding me. Yes, I asked Eric that straight out and he told me we didn’t need to. His exact words were – you don’t need to test the permissions because it should work. But if we have extra time, let’s take a look at that anyway”, I say with a smirk.
Whenever a developer says you don’t have to test it because it should work, I chuckle to myself. I’ll take their opinion about what requires more or less investigation, but that phrase makes it sound as if we only should test things that shouldn’t work. But how could we tell if it works or not if we don’t test it?
Still no word from Kathy. I walk over to her desk. Eric and Kathy are hunched over her keyboard. Eric is muttering. I think he just dropped the F-bomb. Oh boy, it’s a little early for cursing, even for Eric.
“Hey guys, what’s up?”
“There’s a problem; the build is throwing an error. I asked Eric to take a look at it”, says Kathy.
They are investigating the problem, they know their stuff, and I have confidence that between them whatever is broken will get fixed because they are the right people to be working on it. So I go back to the team in the QA Lab.
The project manager walks into the QA Lab. “So, how’s the testing going?”
“Well, we’re still waiting for the build, but we already talked with development and we brainstormed on the test ideas we’re going to try.”
“What?! You didn’t start testing yet? I told you how important this feature is to the business. It’s our top priority. And you already wasted half a day!” I look at my watch. He’s nuts. It’s been only two hours, at most.
“Come on, whatever problems you have — fix it”, he continues. “And if you need help, I’ll pull everyone else off of what they are working on to help. I told you already, this is a top priority for us.” And he storms out.
Kathy walks in, apologizing. “Sorry guys, this was trickier than I thought, but it’s all OK now. We needed to reboot the server because a maintenance process that runs over the weekend didn’t complete in time. You’ll have the new build in QA in about 5 minutes.”
…and five minutes later, the new build appears in the lab. We start with our smoke tests.
We have a few automated smoke test scripts. One that is GUI based, and drives the front end. The other is a series of back end scripts that check basic
message flow between serverside components. We kick both of them off, and start playing around with the new feature.
“Hey, I think there’s a problem. I’m getting this strange error message pop up whenever I try to enter a new record. And the logfiles are showing the same thing.”
I have a hunch there’s a mis-configuration in the database. I call Vlad, the database programmer, on the phone.
“Hey Vlad, I’m testing the new version and I’m getting this error message when I’m trying to add a record from the GUI. Can you take a look?”
“Did you run the script?” he asks.
“What script?”
“You need to run the database script that adds the new fields in the database for these records.” he says. “I’ll email it to you.”
“Oh, I didn’t know about that. Um, actually, don’t email it. I’d prefer if you uploaded it to the shared drive, this way it will be documented and everyone will know about it,” I tell him.
“You know, I’m busy with this other project right now. I’ll upload it soon, but in the meantime you can just run the script. Here, I already sent it.”
“Well, alright, just make sure that the version you upload is the same version you just emailed me.” “Thanks, Bye.”
I run the script. Checking the database logfile, I’m actually surprised not to find any erorrs. So we bounce the database reader process and the GUIs, and start testing again.
Everything that needed to bounce is now up again. It is always a pain to bounce the processes because the reconnection takes so long to initialize. But there’s nothing we can do about that. That is just how it is, until management approves our order for new faster machines.
I navigate back to the record entry form on the GUI and try to enter a simple record. Nothing fancy, just the “plain vanilla” record. And again, I get the error message. I need Vlad.
I dial his extension again but get his voice mail, so I walk over to his cube. He’s not there, probably went to lunch. Great.
So I go back to my desk and open up the database script, and try to trace through exactly what it is supposed to do, and at the same time, checking the error messages that I got and to see if I can decipher any of it.
I think I’m beginning to understand what the database script is doing, when I see Vlad and Eric coming in from the elevator. I go to greet them. “Hey guys, I’m still getting that error message.”
Eric says, “What error message?”
Vlad fills him in– “He didn’t run the script.”
“But then I did run the script, and I’m still getting the error”, I say.
“Which script did you run?”
“Which script did I run? What kind of question is that? The one you e-mailed to me.”
“OK, let me take a look.”
He goes to sit at his desk, and I follow.
Vlad brings up the database editor, selects the QA environment schema, opens a few tables, and mutters.
“Your database was out of date. See? I gave you this script for this version, 5.4. But the 5.4 script had a dependency that the database was up to date. You didn’t update the database since 4.9. See?
You have to run all the database script packages in order. First 5.0, then 5.1, 5.2, 5.3, and then 5.4. You can’t skip any of them, because there are dependencies. Here, let me do it for you. I have it all open right here.”
“What? Wow. OK thanks. But that doesn’t make sense. How can we have had such an outdated database for so long? It’s nearly impossible.”
“I don’t know, but that’s what it is. Give me 10-15 minutes and you’ll be all ready to go.”
“OK, thanks a lot, Vlad.”
Vlad calls me. “You’re A-OK now. All set. And by the way, I uploaded all the scripts to the shared drive, like you asked.”
I’m beginning to think that the reason we didn’t run those scripts for the previous releases, was because he never uploaded it to the shared drive before. We didn’t run them because we never knew about them. Or maybe everyone assumed that someone else ran them. But if that’s true, that raises an interesting question – how did we pass those other releases, if in fact we weren’t testing in the correct database environment? I keep all these thoughts to myself, for now. I need to focus on this release now, before the project manager comes waltzing by spreading his special brand of cheer again. I want to jump in with those amazing tests from this morning’s session with Eric.
Again, we bounce the database reader process and the GUIs, and again navigate to the record entry form.
“Whoa! Double entries!” I can’t help but yell out. “Hey Raj, check this out. Click on this dropdown list, you see? It has everything listed twice! Oh, wait, it’s not just this dropdown…it’s everything! All the dropdowns have double entries. It’s as if the database script was run twice.”
“TWICE?” asks Raj, “It WAS run twice. Once when you ran it in the morning, and again now, when Vlad took care of the scripts.”
But that shouldn’t really matter too much, it’s just extra data in the database, and it’s just appearing in the front end, like you would expect it to. If I just select one of them, I should be able to just submit this record.
But no. Now I’m getting a different error message when I hit OK. It says that there’s a destination out of bounds exception.
I walk straight back to Vlad’s desk. He sees me coming. “Now what?”
“Hey, listen to this. I think we ran that script twice. We’re getting duplicate entries in the dropdowns, and I checked the database tables, the rows are there twice. And we’re getting a destination out of bounds error when we try to submit.”
He tries it on his desk, and things look exactly normal.
“Do you mind coming to the QA Lab? I can show it to you, it’s 100% reproducible.”
Vlad and I walk to the QA Lab, and Raj shows him the problem, and the new error message.
“Oh. You’re getting that error because it doesn’t know where to submit the record to; because there’s two possible places it could go. Congratulations, the database is corrupt.”
“OK, well, I guess we all need to rollback to the 5.4 script because it was run twice.” I suggest.
“I didn’t write a rollback script.” He says.
We all look at each other. “Oh, man. Now what do we do?”
“Let’s get everyone in here, we have a situation.” I say.
In just a few minutes we’re all standing around in the QA Lab. The project manager looks particularly agitated.
“How long will it take to write the rollback script?” he inquires.
“An hour or two, but I need to do this other project.” answers Vlad.
I’m just waiting for the project manager to flip out one more time and say that this is our top priority. I’m not disappointed. I try to hide my grin.
Anyway, I’m not very comfortable doing the test on this database, even if the script gets rolled back, because it’s already too messed up. It’s already in an altered state, and if we found anything, it would be hard to figure out if it was because of the database environment, or if it was a bug. Or worse, the altered database might mask true bugs. It just doesn’t sit well with me, and I would prefer that we test on a newly refreshed database. I explain all this to the team.
After some more discussion, we decide that Vlad will start working on the rollback script (the other project can wait…you know, this is a top priority, after all…) because just in case we’re going to need that when we roll out to production, we need to cover all our bases. In the meantime, we’ll ask Maria to refresh our database with the latest production copy. Maria is our DBA.
Refreshing the database doesn’t just mean copying over the tables. We aren’t allowed to have access to production data.
We call her.
Maria’s response,”Yeah, sure, I can do it. It normally takes about 2 hours to refresh all the tables. You know, it’s not just copying over the tables. Only the business is allowed to have access to production data. There’s sensitive information there that needs to be dummied out, otherwise Compliance will have us all fired. I can do it all, don’t worry, but listen, I need to leave early today, I have a dentists appointment. I’ll kick it off now and I’ll let you know.”
I go back to my desk. I’m starving, and realize I didn’t have lunch yet. I can’t believe it’s so late already. The cafeteria is closed already, so I go pick up a sandwich from the deli downstairs.
Back at my desk, eating, I catch up on some outstanding administration work i.e. checking emails, getting back to people.
Things have slowed down since the morning. I feel like we’re ready for the build, whenever it gets to us.
I start browsing some of my favorite blogs on the web. I see that James and Jon Bach are working on something new called thread-based test management. I wonder what that is. Is it like exploratory testing? I read on, continuing to learn.
My phone rings. It’s Maria, the DBA, calling from her cell phone. She already left for the day, but she wanted to let me know that the database refresh is complete.
I tell Raj, and together we go to the QA Lab, once again, bouncing all the components in the system. The GUIs come up, and we check the dropdowns. No double entries. It looks like finally, the test environment is ready for testing. Again, we kick off the smoke tests, but at this point, we’re not going to get to try our test ideas until tomorrow.
On the train ride home, I’m thinking about the day, and how we got nothing accomplished all day. It was a total waste.
Or was it?
- We found a build error, and now they should know how to avoid that again in the future.
- We found a database error.
- We found a hole in the process, where the database scripts were not being enforced to be put up on the shared drive.
- We learned about the feature, and why the business wants it.
- We came up with a list of good test ideas, ready to try out, in addition to the standard regression tests that we run.
- We had a great brainstorming session with the test team, which strengthened our attitude of teamwork.
- We reviewed our test documentation, and it’s ready for internal audit by Compliance, if they decide to audit our practice.
- We discovered two different showstopper errors.
- I did some deep digging into the logfiles, learning what the processes are doing on a more granular level than from the GUI perspective.
- All things considered, the working relationship between the teams is pretty good. Between Eric, Vlad, Kathy, Maria, and yes, even the project manager, we all worked well together under pressure.
- I “managed up”, explaining testing ideas to management.
- I even had some time for professional development, read those blogs, and keeping up to date with the industry trends. Not too bad for a wasted day.
About the Author
Bernie Berger Bernie Berger has been testing software at Wall Street firms for more than 15 years and has experience on various software projects related to the financial industry, including electronic trading and algorithmic strategies, FIX protocol and market data validation. He is a former director of the Association for Software Testing (AST) and is chairman of the AST Financial Services SIG. He founded and runs the Software Testing in Financial Services (STiFS ) workshops.