Writing Programs to Test Programs
As the software development cycle becomes more and more compressed by higher level languages and component-assembly approaches, it is becoming less and less possible to accomplish testing by hand. Internet-time delivery schedules demand rapid turnaround at every phase, but visibility makes quality more important than ever.
This trend is fueling continued growth in test automation: You have to fight fire with fire. What was built by new technology can’t be tested with manual labor.
On the other hand, though, you can’t just take your existing, seat-of-the-pants test approach and automate it. If your current testing is more exploratory than declaratory, you are going to dig yourself a deeper hole by trying to turn it into software.
Let me explain.
How To Do
Most of the most popular test automation tools are really just specialized programming languages. They are either unapologetic extensions of existing languages like Visual Basic, or they are vaguely reminiscent of others like C or C++. The only real difference between a test tool and a development tool – except that it is usually called a “scripting language” instead of a “programming language” – is that a test tool has special features to allow it to interact with another application during execution and log the results.
These tools can be implemented in two ways. The most easily recognized way is to “record” a script or program by performing the test manually and having the tool automatically generate the code. This approach, while easy in theory, results in a mass of unstructured, incomprehensible code that would give a real programmer hives. The other way, which is usually adopted after the first approach fails, is to just buckle down and start programming.
And this is where the trouble usually starts.
What Not To Do
A manual test process, it turns out, is quite tolerant of uncertainty. If you are sitting at the keyboard, you can react to the application’s state and behavior instantaneously, making decisions on the fly about what to do next. For example, your application allows customers to buy and sell stock. To buy stock, they must either have enough cash in their account to cover the transaction, or they must have enough borrowing power based on their account balance. If they have enough cash, the transaction is placed; if they have enough borrowing power, the application prompts for authorization to extend credit; if neither is available, an error is issued and the transaction is rejected.
Further, your test database is copied from production from time to time, and is also shared with development, training and support. As a result, you never know the cash or account balance of any given customer at any time. So, as you sit down to test a trade, you must react to circumstances presented by the state of the customer, and perhaps even select a different customer if the one you chose has no cash or borrowing power.
In this situation, which is more typical than not, someone armed with a scripting language will proceed to write a program that mimics this behavior exactly:
Select customer number
If account cash balance >= trade price, then
Confirm trade
If cash balance + borrowing power >= trade price, then
Authorize credit
Confirm trade
Else, add customer number +1 and try again
Let’s examine this for a minute. Does it occur to you that what is really happening is that the underlying application logic is simply being rewritten in a different language? In other words, if we looked at the code that is being tested, would it not have the same logic, more or less, but written in a development language?
In other words…isn’t this just writing a program to test a program?
But wait…it gets worse. If you are really being a thorough tester, you have to test for negative conditions as well, such as when the customer does not enough cash but the trade is confirmed anyway, or does not have enough borrowing power but credit is still authorized. For example,
Select customer number
If account cash balance >= trade price, then
Confirm trade
If account cash balance < trade price, and
Trade is confirmed, then
Log error
If cash balance + borrowing power >= trade price, then
Authorize credit
Confirm trade
If account balance < minimum margin and
Credit is authorized, then
Log error
Else, add customer number +1 and try again
Can you see what is happening here? It will actually take more code to write the test than the code being tested! Do you know of any situation where the test team is equipped with enough time, resources and skills to develop a system that is bigger than the system they are trying to test? And who will test the test system?
It’s crazy, isn’t it? Of course it is, yet every time I point this out, the hapless scripter simply says “but that’s the only way I can get it to work”. Said another way…that’s the only way you can reproduce the manual test process.
What To Do
Come to grips with the fact that automated testing is fundamentally different from manual testing. It is not tolerant of uncertainty. If you don’t know in advance which customers have enough cash or borrowing power to place a trade, then the problem should not be resolved in a script, it should be resolved in the test environment itself. Get a stable, controlled, predictable database where you can plan which customers meet which conditions, so you know how the trade should be handled. Otherwise, you’re doomed.
If you are not prepared to change your test process, you are better off sticking to manual labor. Automating uncertainty leads to yet more uncertainty – which failed, the application code or the test code? And as far as maintenance of the test code when the application changes…don’t even get me started.