... well, at least, I think it's sweet.
We have a few test challenges in this world -- but do we have many test examples?
I mean, think about it. One way to become a better writer, painter or sculptor is to study to read great literature or to study the masters. If you look around, there is open source software to read, and a few books that annotate and describe good examples of code.
But tests?
No so much.
When we do have examples they tend to lean toward one extreme or the other -- two simplistic or too complex. Anything with a real user interface, for example, is going to suffer from an infinite number of possible tests.
Don't believe me? Take a long hard think about how you'd test to see if a desktop calculator application had a memory leak.
That might make a good article, but I'm afraid it's beyond he scope of a blog post. And while it might be fun to read, it's unlikely that you could absorb the thinking process that goes on in defining the tests. Going from "empty sheet of paper" to "completed test plan" might be a big question mark. Even if you could do it for a calculator -- and follow the steps -- generalizing the ideas and using them on your next project would be a bit ... much.
Beyond the desktop we have Multi-user applications. These are even more complex because they add in race conditions, security and scalability and internationalization concerns.
Yet if you take away all that complexity, say having a single-user, batch-oriented tax-fee calculating application, you run the risk that your audience considers testing to be simply an exercise in figuring out logical paths with a little math. Worse, they might not even realize the math part of it and think it's just picking one of each option in some requirements document.
When I find something in the middle, something we can discuss and really cover well in maybe an hour; well, then, I get excited.
I am excited now.
No, it's not a perfect exercise, it's testing a batch program. This has real combinations and real different values. Yet it doesn't have that user interface, that infinite-possible-paths things we see in desktop applications(*). For example, I worked on a AutoDesk plugin once that would crash after you tried to undo three times in a row by a specific keystroke. Investigating those kinds of bugs is a great challenge, and important -- but a different type of thing that the example I've been looking for.
No, this is a real problem that lots of IT shops deal with every single day. I expect some of our fellow readers deal with this.
Please let me tell you about it.
The challenge
Every day, all over the world, programmers are writing software to extract data from a database, transform it if needed, and load it into some other system. Weather it is pulling all the new transactions out of the ERP system and into the data warehouse, or pulling all of the orders into the physical warehouse to ship products, or sending them off to a supplier to get parts to build the thing just ordered, Electronic Data Interchange (EDI) is all over American IT.
Oh, don't get me going about how the "real time enterprise" uses Service Oriented Architecture. I've done a fair bit of that, even
written about it, but that isn't this example. Instead, we're going to be a little more ... old school.
Every morning, at 10AM, after the data warehouse is refreshed and will remain static for twelve hours, we are going to run a extract-transform-load program. This process will select all of our current members of Cigar of the month club and dump them into a simple flat, text data file. We will zip this file up and, through an automated process, send it over to our outsourced customer service partner, that will unzip the file and load it into their systems. Then when a customer calls, they will first inquire that person's member ID number. If the person is a current member, our call center will allow them to get support place orders, and so on.
We are going to do a similar process every day to extract the members who made orders yesterday (on top of the monthly order) and send that off to our fulfillment vendor in Uruguay, Mr. Afficondio, who will unzip the file and prepare and ship the packages. We also send him a list of customers once a month for his monthly shipment.
Now there are a lot of possible issues here, things like security, privacy, better business process, or Mr. Afficondio's return receipt to us that the orders were shipped. For the purpose of this example, I want to focus on three things: The file to customer service and the two filetypes to Mr. Afficondio. Within those three things, I specifically want to know if they are fit for purpose: If customers can call in and get support, and if customers will get the right Cigars in the mail.
Most importantly for this exercise, I want to make sure the data is right and can be correctly imported into the 3rd party source systems. So we won't spent a ton of energy asking if UPS really does deliver Cigars from Uruguay to the united states, or the quality of Cigars. Let's try to focus at least a little on software testing, okay?
For purpose of simplicity, we have decided to use a separated-values format for the data. So the basic data for members is something like this:
Last name, First Name, MiddleInitial, Home Address, Phone,DateOfBirth, CustomerNumber,MemberSince,MemberUnitl
And for orders it is:
CustomerNumber,CatalogItem,DateOrdered
So that's the basic scenario, very similar to something I used to do at an insurance company so often that we developed patterns for testing it.
But before I describe that testing strategy and how we evolved it, feel free to ask the questions you would need answered to take your own stab at a test plan. And if you are comfortable, you can even leave a comment with your own answer to the question -- "How would you test this?"
My answer (and yes, I think we can have more than one valid test approach) will be up in about a week, just after TWiST number six.
Go on. Give it a try.
You know you want to.
--matt heusser
(*) - Yes, there are other infinities of possibilities, even with with plain text batch-processing. I think it's fair to say that the problem described here is a little more ...
manageable in blog-post form. Fair enough?