10:00 AM
“Holy smokes, I can’t believe I’m so late!”
I roll over and look at the clock. 10AM. 10AM?! I was supposed to be in the office an hour ago. What happened? I overslept? That’s so not like me.
I check my cell phone – battery is dead, no wonder, the alarm didn’t wake me up. And I didn’t hear any of the missed calls that I’m sure came in from work asking where I was.
I scramble to my laptop and frantically connect to work. Gee whiz, I completely blew off the 9:15 meeting. What poor form. I quickly log on, and send an email out to the team to let them know I’m delayed and will be in the office just as soon as I can. I quickly get ready and run out of the house.
10:30 AM
On the train ride in, I’m thinking about what has happened. Yesterday seems so long ago. I was up late last night, online, reading my favorite blogs and catching up on Twitter. I must have lost track of time and I finally hit the sack at around 2AM. But yesterday at work, yeah, that was a good day with a high-priority project. We found lots of issues, even though it wasn’t exactly according to plan. We’ll just have to catch up today. Coming in late today is not going to be one of my shining moments.
11:15 AM
I sneak into the office. I try not to draw too much attention to myself as I drop my jacket and bag, try to make it look like I’ve been there for a while. It’s not that I’m trying to get away with anything; I’m late, so what, it happens. But I just don’t like the extra attention.
Turns out that there was some huge traffic jam earlier today, and everyone else was late too. Half the office just came in only about half an hour ago, and the status meeting was canceled anyway. Whew, that was a close one. I remember something a college professor once said: “The first step of being successful is showing up on time.” Today was not that day.
The project manager stops by my desk.
“So I understand that yesterday was a complete waste,” he says not unexpectedly. I knew he would see it that way. “Are we still going to hit our target release date? I don’t have to keep telling you how important this release is.”
So much of software development – any development, really – is dealing with the wide range of human behaviors of multiple team members with varied backgrounds. Needing to validate his frustration, but at the same time standing up for our efforts yesterday, I say to the project manager in a tone a little more formal than which I normally speak, “Yes, I agree that we didn’t get as far as we would have liked yesterday, however, we did discover key pieces of information about the release that ultimately is making it a stronger product in the long run. But despite all that, I think there is still enough time for us to run our planned tests before the deadline, assuming there are no further crises.”
“OK good. As long as we make the deadline. Just get it done.”
I head over to the test lab to see Raj.
“Hey man, what happened to you, are you all right? Did you get stuck in all that traffic with everyone?”, he asks.
“Yeah, everything’s fine. Thanks. Let’s just figure out what we’re going to be doing today. We left yesterday and the automated smoke test finally kicked off, after all those set-up problems, right?”
“Yes, it took all day to set up the environment correctly, but now we are good to go.”
Between us, we plan out our strategy for the day.
“Let’s bring up the system and get the environment set up. We’ll kick off the full set of automated smoke tests. And let’s sit together and go through these new features,” I say.
11:30 AM
Raj and I are sitting next to each other in front of a desktop in the lab. Raj is at the keyboard, I’m next to him. I have my notes from yesterday’s meeting with Eric, the programmer.
We’re going to have a session of paired exploratory testing. Raj and I are a good team. We feed off each other when testing in a very productive way. Between us, we have a name for that feeling when we’re in the zone, when our creative energies bounce off each other creating a positive synergy. We call it being “on the jazz”. It’s a catchphrase from the TV show “The A-Team”, but it really fits. What usually happens is that I would suggest a test, and Raj tries it, then Raj notices something which triggers an idea that he tries. I’m right with him and then I suggest a third thing based off of Raj’s idea. And this cycle of trial and error continues to bounce between us until we find results that we think would be very interesting to report. We’ve done this plenty of times before. Working together, we “talk out” the tests that we want to do, which helps us discover new ideas or different variations of tests to try. Some people call it testing with “another set of eyes” but it’s much more than that. It’s the synergy that’s created between us that becomes really valuable.
We bring up the entry window, and, remembering the duplicate entries that we saw yesterday, I tell Raj to just quickly spot-check the dropdowns, just to make sure that our database is still OK. We are happy to see that it is.
There’s a new checkbox on the entry screen that the user is supposed to select to flag the entry as an “MMY entry.” “MMY” has a particular meaning to our users, but from the system perspective, it is simply a checkbox that triggers a flag in the database. And at the end of every day they need to generate a report of all the MMY entries for that day and submit it to the regulators. That’s why the project manager is so eager to get this in production already, because Compliance is down his neck for the industry-wide MMY deadline next week. Before, all entries with certain criteria were automatically considered MMY, but now the user can designate any entry no matter what criteria, as MMY, by clicking the MMY checkbox before submitting.
Simple tests seem to work fine. Clicking, unclicking, submitting. What about correcting an entry that’s already been entered? That seems to work fine too.
I tell Raj that things are looking OK but I think we’re thinking about this testing too narrowly. We’re only considering tests from the perspective of the checkbox; if it’s checked or not. We need to think out of the box.
“You mean out of the checkbox.” Raj says with a smile.
“Oh, ouch! That was horrendous,” I chuckle. “But do you know what I mean? Who cares if the checkbox gets checked or not. It’s just a stupid GUI object. I mean, no offense to the checkbox designer or anything, but do you really expect to find a bug where you click on it and it doesn’t get checked? We need to think about what it means, how it affects the downstream applications, how it would affect the different stakeholders, not only the users. Like the production support guys, for example. How would this affect them?”
“Well they are in charge of making sure that things go smoothly for everyone and if there’s a hiccup they need to smooth it out,” he says.
“OK, well, then let’s simulate a test scenario for them. Let’s see what would happen if we do all our regular checkbox tests while the system is being acted upon in a way that mimics what the production support guys would deal with on a typical day.”
“So you mean, let’s have a few workstations up. On one we’ll continuously insert entries, and on the other we’ll delete or edit them. And on the third, we’ll insert these special MMY entries.”
“Yeah, and how about this: in the middle of all these transactions, we’ll have a fourth workstation where we’ll play with the user permissions, granting and removing user entitlements for entering and deleting. Remember yesterday Eric said we didn’t have to test permissions? Let’s do it anyway.
That would be a really good simulation of real life conditions.”
“Well, not really,” interjects Raj. “What I mean to say is that it is good, but it could be better. What would be really the best is if we could get a copy of the production entry logs and replay them.”
“Ah! Perfect! You just touched on one of the classic debates in test design – whether it’s better to test with customer data, like the production logs, or constructed data, like the specific cases we wanted to try. There are pros and cons to both sides and like everything else it depends on your context. I can tell you about it later, but for now how about we do both. I’ll go ask Maria for a copy of the production entry logs and I’ll modify the Python script I was working on last week to pump those entries in on the server side here in the QA lab. While I’m doing that, you can put together a quick GUI automation script for the continuous entry of MMY entries for the second workstation. And while these are running, together we’ll manually play around with the user permissions, and we’ll see what happens.”
“Sounds like a plan.” says Raj. “But how about we get back together after lunch? I have to go to the Post Office.”
“Lunch? What time is it? 12:30 already! Wow, it feels like I just got in.”
“You kind of did,” Raj says with a smile.
So Raj goes off to the Post Office and I go back to my desk to call Maria, the DBA. I dial her extension.
12:30 PM
“Hey Maria, how’s it going?
…Yeah, it’s busy here too. Don’t ask. How was your dentist appointment yesterday?
…I know, it’s really hard to find someone you trust. Good that you took care of it, though.
…Anyway, listen, I was wondering if you could do something for me. Can I get a copy of one day’s worth of entries?
…Yeah, from Prod.
…Doesn’t have to be. It could be from whenever. We just want to pump them into QA for a test.
…What?
…Yeah, last week is fine, it doesn’t really matter.
…Yeah, I know, I know, you have to desensitize it. We wouldn’t want us QA folks taking a look at our customer’s sensitive data, now, would we?
…Hahaha, I know, tell me about it.
…Wait, but how long does it usually take to encrypt the client sensitive data?
…Thirty minutes? That’s awesome.
…Yeah, cool.
…Oh, one more thing, is it CSV or plain text?
…No, either is fine.
…OK cool that works.
…Where? On the same shared drive as last time? Yeah that would be perfect.
…Yeah I have access to that drive; I’ll just grab it from there.
…Thanks a million, Maria, this is a great help, really.
…OK, sure, thanks.
…Yup.
…Bye.”
1:00 PM
I’m not really hungry yet, but I better go get something from the cafeteria before they close. So I go down to get a sandwich. At the elevators I see Vlad and Eric. They seem to be arguing about something, as usual.
“Hey guys”
“Hey, what’s up”
“Not much. We were setting up a test for the MMY release.” I tell them about our test strategy.
“Sounds good. Let me know if you need any help with that.” says Eric.
“Thanks.”
1:30 PM
Come back to the lab, Raj is there already. So we sit back down at the computer. This time, I’m driving at the keyboard and Raj is next to me.
“I updated the Python script and I have the data file from Maria”, I say. “So now I just have to set it up here, and we’ll be good to go. How did you do with the GUI automation script?”
“It’s all done. It’s really just a simple script, nothing fancy, and it doesn’t do any validation. All it does is click on the submit button once every 5 seconds, so it would send in a new entry with the same criteria continuously. You have to manually set up the entry screen with the criteria, and then kick off the script to keep hitting submit.”
“Wow, I love how crude that is. Quick and to the point. But sometimes crude is all you need.
Why don’t you kick yours off first and we’ll just check that it would be OK to continue testing”
“Sorry? I didn’t get you. What do you mean by ‘OK to continue testing’? Let’s set up the whole test like we said before, I’ll run my script while you run your script, and at the same time we’ll modify the permissions.”
I smile. “Raj, you just highlighted another debate in the world of testing! When testing, is it better to start with the most complex case first, or, is it better to start with the simple case and build up to the complex case?” “I don’t know, I guess there are pros and cons to both sides here too. It depends.” “And what does it depend on?”
“I don’t know…your context?”
“Exactly! Like you said, there are pros and cons here too. When the first thing you do is start with the simple case, you are establishing that everything is OK to proceed with testing. Each next test adds a little more complexity to the testing so that if you find an issue, it’s easier to determine which level of complexity introduced the problem. But the downside is that this sometimes takes a lot of time to get to that point, and the simple cases are not likely to cause any problems anyway. So sometimes you might want to start testing right away with the complex scenarios.”
“OK fine, so let’s start with my script.”
“It’s your choice,” I smile.
Raj sets up his test, enters the criteria in the entry window, but doesn’t select the MMY checkbox, just to check the simple case before building up to the more complex case of MMY. He kicks off the script that clicks ‘Submit’.
“Hey, something is wrong. It’s submitting the entries, but I don’t see them coming through. That’s weird.”
“Really? That IS weird. I promise I didn’t set you up for that. But why wouldn’t these entries go through? It just worked this morning. What happened?”
“That’s what we have to find out. Let me check the logs.”
2:15 PM
Raj goes ahead to try to figure out what’s going on in the lab, while I go back to my desk to see if anyone else might have an idea. I’m going to call Eric and Vlad but first I’ll sort through my unread emails, just in case something urgent came in while I was in the lab.
Nothing too important. A reminder that I’m supposed to give a presentation at the company’s monthly QA Center of Excellence meeting tomorrow. Oh geez, I completely forgot about that. I’m not sure I’ll be able to do it anyway; this project is really taking up most of our time. I don’t even know what I’m going to talk about.
I walk over to Eric’s desk to see if he knows anything about the entries that are mysteriously not showing up.
“No, man, I have no idea. Maybe Vlad knows.”
Just then the project manager comes running in, agitated, frantic.
“IS ANYBODY SENDING TEST ENTRIES TO PRODUCTION HERE?!” he hollers.
“To production? Of course not!” I respond, shocked.
“Well, somebody is. We just got a call from production support that one of our customers got like 50 entries in a row with test criteria in them.”
My stomach drops. Those are our missing entries.
“In Prod!? How in the world did they get to prod? We entered those in QA.”
Vlad joins us. “What was that about prod?”
“We were doing a test in the lab,” I say, “just to enter a bunch of entries, and somehow they ended up in production. But this morning we also entered the records and it worked fine, in QA.”
“Uh, I was in the lab a little while ago, no one was there. I needed to compare my new code to what is currently in production so I changed the config to prod on one of the machines.”
“You what?” “Yeah.”
“So that explains exactly what happened. While we were at lunch, you changed the config, and then when we came back the script we ran was running over the production lines. That’s crazy. How did you change the config to prod? You know the production password?”
The project manager jumps in: “OK it doesn’t matter. We’ll talk about this later. But now we know what happened. I have to go back to prod support and let them know, so they can tell the customers what to do.”
“Sorry about that”, Vlad says. “I’ll go change the config back to QA.”
“Yes, please!” I say.
4:00 PM
Back in the lab, talking with Raj.
“Can you believe it? It was connected to Prod! It’s a good thing that we only submitted a few entries. That was bad enough. But could you imagine what would have happened if we would have submitted everything? If we would have replayed production data into production? That would have been a nightmare.”
“Yes, I agree. But my only question is how did we have access to production? Did we have production user permissions?”
“It’s a great question. I have no idea. I would have thought that it shouldn’t have been accepted,” I say.
I stop and think for a moment.
“Say, did you have the MMY flag ON in those entries?”
“No, they were the first batch, with MMY off. Why?”
“Because I bet that MMY is not backwards compatible! The production database is not set up to accept the MMY flag. We’re all extremely lucky. Listen, if the MMY flag would have been on, I’m sure we would have crashed the production system, and bad. Remember yesterday we needed to run all those database scripts to get the environment in order for MMY? Well, those scripts weren’t run in production yet. The database, as it is currently, wouldn’t know what to do with that extra flag.”
“Wow, you’re right,” says Raj. “But I still don’t understand how the user permissions allowed us to send entries to prod.”
“Yeah, user permissions…” I think for a second.
“Hang on a second! User permissions! I wonder if the users need a special permission to add MMY orders. You know, each user can be enabled or disabled for making entries in general, but they also probably need an additional permission to enter the order as an MMY. I better check.”
The project manager is out in the hallway, he just finished giving Vlad a lecture about keeping the environments in order. I join them. Nobody looks happy.
“Hey, do users need special permissions to enter MMY entries?” I ask them.
“Of course,” says the project manager. “You know that everything is permissioned.”
“Because I don’t think we have that capability.”
Project manager turns to Vlad, “Well, is MMY permissioned?”
“MMY? No, I didn’t add MMY to the permissions database. No one told me it had to be permissioned.”
Today is not Vlad’s day.
The project manager looks too tired to be upset. “Well, we need it in there. Vlad, just do it right away, come on. And we need it for this week.” And turning to me, then he says, “you’re just going to have to step it up tomorrow.”
Yeah. Tomorrow.
5:30 PM
Packing up my stuff to go. It was another insane but good day. We had a good initial exploratory test session this morning. Raj and I bounced ideas off each other all day. And we had different examples come up today about how there’s no one right answer to testing problems, like the customer vs. constructed data and the complex vs. simple tests to try first. Also, each of us decided to build an automated script to help us with our testing. What’s cool about that is that we decided on our own that it was the next important thing for us to do, and we made that decision while we were testing. And besides the drama of pumping entries into production, that showed us a big gap in our internal systems security. Plus, not to mention that in the course of discussion, we realized that an entire piece of functionality was missing from the build!
Yes, tomorrow is another day.
About the Author
Bernie Berger Bernie Berger has been testing software at Wall Street firms for more than 15 years and has experience on various software projects related to the financial industry, including electronic trading and algorithmic strategies, FIX protocol and market data validation. He is a former director of the Association for Software Testing (AST) and is chairman of the AST Financial Services SIG. He founded and runs the Software Testing in Financial Services (STiFS ) workshops.