In any endeavor I participate in, I have choices. In many of life’s transactions, I can spend money to have something done, or I can spend the time to do that same thing. For some things, it is worth it to invest the time to make up or offset the amount of money to be spent. At other times, there is a level of involvement and a required expertise that makes not spending the money a barrier to achieving my goals. There is a tradeoff; money for time, or time for money. The problems arise when we don’t give both of each of these areas proper consideration. In the world of software development, containing costs is important, but so is delivering a good quality product at the right time. Containing costs when a project is on time is a good thing, but losing time and not delivering a product when scheduled can cost both time and money, often in the lost revenue from dissatisfied customers. At the furthest extreme, the resulting loss of time could result in legal action for not being able to deliver on commitments. This article examines the tradeoffs between money and time, and demonstrates examples where time investment has meant financial gain, and where lack of understanding of time has negated or even negatively affected a company’s finances.
A New Project Gets Underway It’s time for another software project to begin. Each time I go through this process, I have a similar set of experiences. Each time I need to:
- Address physical infrastructure needs
- Determine where to set up machines
- Gather and install software applications to support the testing needs
- Consider the scope and types of testing that will be required
Physical infrastructure needs include buying and setting up server and client system and networking equipment. It may also involve internal or external hosting considerations, plus the adequate securing of these resources. When we consider physical machines, potential lab space for the devices and the need for adequate cooling must be considered. Our software application under development also has infrastructure requirements. SDK’s and IDE’s are needed for software development. We must configure and maintain file services, database services, web services, etc. We also have to consider deployment of the application(s) for testing and for general use. The specific testing areas (and the software testing tools that we utilize) must also be considered. Unit testing. Functional testing. Load testing. Performance testing. Para-functional and human factors/user experience testing. The number of areas to manage and maintain is dizzying!
I have witnessed various companies and their management teams wrestle with the goal to improve the testing and development process. These goals may be to speed up delivery, or add testing coverage for initiatives that haven’t adequately been tested. Many management teams also provided fanfare and cheering about the time savings due to some new enhancement. It might be a new tool to automate all of the regression testing. It could be a new server to run a battery of virtual environments. A robust versioning and backup system to keep all versions of files updated, tagged, secure and available would streamline the application lifecycle. These are admirable goals, and when implemented, the net result can be more productive testing. Work can be completed more quickly. All of these initiatives cost us something. To set up any of these enhancements, our teams have to spend something to bring them to fruition. With organizations squeezing budgets and economizing where possible, some businesses are reluctant to spend. During economically challenging times, many organizations opt to soldier on, using tools already in place (if any exist) and machines that are already at work in their current state. Money is not spent, so our costs are lower. What this “cost saving” doesn’t address is the time trade-off made to perform these tasks. The cost of testing is not exclusively about money, very often we must include time in that cost analysis as well.
Where Are We Spending Our Money?
As our development project ramps up, the resources available will dictate the total time needed to complete it. The number of people on a project certainly adds to the total price tag of a project, as well as the total amount of time a project may take. With a testing team, two testers working together will very likely complete more testing than one tester will. The combined test coverage is often much greater than the sum of individual tester’s standalone efforts. Having more powerful computers will certainly shave the time it takes to build applications. Likewise, having more machines makes it possible to deploy more test environments.
Support for multiple operating systems provides additional challenges. Microsoft Windows, MacOS, and HP-UX are examples of Operating Systems that have an up-front licensing cost for each installation. Linux and Free-BSD are examples of open source operating systems that are available for free. Beyond just the cost of the operating systems is the time needed to give each system thorough testing. Each supported platform adds a time multiplier as to how long it will take. There are also variations in database software, web server software, Office productivity applications, and other components that may be required to evaluate the functionality of the application under test. In addition there are variations in the way the software is written and deployed (pre-compiled code such as C++ vs. dynamic application code like Java). Each variation requires another environment be set up and tested.
Testing these multiple OS’s requires choices be made. We can purchase and maintain individual hardware machines. We can also utilize virtualization technology to reduce the need for multiple physical computers. The host server will require many orders of magnitude, more capacity and performance than the individual computers we would have virtualization replace. If a project requires six systems of equal capability, virtualizing those machines will require sufficient resources on a single server to be able to house and run those six environments. That host server will require enough disk space, RAM and system resources for the six virtualized environments to operate and respond as though they were standalone machines.
Testing tools range from free, open source options all the way up to very expensive proprietary commercial systems. The application under test often determines which tools, if any, will need to be used. There are many free and open source test tools for testing web applications. By contrast, most of the tools that test a compiled application from a black box perspective are commercial products (though there are also a number of open source options available that allow us to create tests and automate many tasks).
As I have seen over the years, not spending for these items or areas can provided a short term cost savings. We made do with the servers and machines we had on hand. We also managed to get by with a limit on software licenses (within reason and within the bounds of what is legal; I do not advocate piracy to cut costs at any time). I have set up test beds and test labs using free OS software, using a variety of Linux variations. I have also used both commercial and free virtualization software, where the host was running Linux, and the guest machines likewise represented a variety of free operating system options. These test environments also have been set up to use as many free or inexpensive variations of open source tools for testing purposes. There is no question that using these techniques has saved money.
What’s The Trade-off? The Trade-off Is Time
There is an adage that says “You can have a system that is cheap, fast, or of good quality. Pick two.” Price, time and quality rarely move up and down together. If one area gets incremented, others are decremented. When we spend less to perform a task, the time to accomplish the task often increases. The quality of the resulting output may also decrease. In my experience, companies want to bring costs down, while keeping the quality level as high as possible. To make both goals succeed (lowering costs and improving quality), in most cases we will have to allow time to increase.
Let’s use a build server as an example. If we chain a number of computers together to perform builds the overall time to complete the build process will diminish. What could shaving 25% of the time off, completing a build do for your development and testing teams? Think of the amount of time that could be applied to testing and fixing issues discovered, rather than waiting for the build to finish. The quicker the turnaround from development to testing, the quicker issues can be discovered, addressed and fixed. Quicker builds allows for better integration of new code. Quicker builds allows for a system that is up and running and more frequently available. With the effort to set up the scripts and tests needed, the quality of the build process can be improved. The productivity gains were produced by spending money to add server horsepower. Paying for the human brainpower to create scripts and processes makes for fewer errors in the build process.
To contrast, let’s consider what would happen if a new initiative of ours requires a server, but our budget doesn’t allow for it. We could share the workload on the existing build servers. We could pull one of the machines out of the build process and make it a standalone server. What happens? Cost savings are immediate; we didn’t have to purchase a new server. The quality of the builds may be unaffected, but the time to complete the builds has increased. If the system performs daily builds, the time we have lost because of the server change will be multiplied by the number of days for as long as the projects are active. The benefits of quicker turnaround for development and testing will not be achieved.
I have used test environments with open source operating system software and open source testing tools. On the surface, the environments appear to be totally free or have a very low physical cost to implement. In my experience, deploying, maintaining and administering these operating systems and applications requires a significant time commitment on the part of the development, test and IT teams. Open source software (and the communities that help with bug-fixes, documentation and implementation of new features) is on par with the development of many commercial applications. The ease of installation and maintenance of the installed software has likewise improved dramatically. Still, a high level of knowledge and skill is required to keep multiple open source applications and tools running smoothly. That knowledge development and maintenance requires time.
Several groups I have been a part of, especially at the beginning, had no formal tools in place. Most of the testing steps are performed manually. Many tasks are best suited to active manual testing, but when all our testing is manual, the time required to complete it can be significant. Multiply our manual testing efforts by the number of projects. The time commitment to do all of the needed testing rises exponentially. Few organizations have the luxury of long periods of time to make sure all testing is performed. What tends to happen is that testing time gets cut (or at best held to the required scheduled time). The total amount of tests get cut, which can lead to a lower quality product.
When we apply a risk based test approach, we aim to maximizing the potential of covering areas we consider most critical. Even with this method, consider how long it would take to cover all areas in the time limit. Where will the cuts be made? Should we eliminate human factors testing? Will considerations for user experience go untested or receive a lower priority? Will features that are desired but can’t be fit into the timeline be moved to later releases? If so, how long will it be until we will be subject to the hands of Kronos once again?
What is Time Really Worth?
Let’s say an average developer earns $50/hour. That includes all benefits. $50/hour is not unreasonable given wages, vacations, holidays, benefits, and sick days. What other aspects of their work day need to be considered? We do not have the ability to simply wake up, teleport ourselves to work, meet our objectives, and then immediately zip away to do other things. We commute and we spend money to make those commutes. When we work at home, we pay for infrastructure to allow that ability. We purchase clothing and food to support us while we are in the mode of working.
Some commentators have said that our full 24 hour day should be viewed in light of what we earn. To really determine the value of our time, we would need to take the total amount of time in a week, and divide it into an hourly figure. If we were to divide those hours in comparison to our average developer earning $50/hour, that individual is really earning $11.90/hour.
This is the concept of the “real wage”, and it was popularized in the book “Your Money or your Life”, written by Vicki Robin, Joe Dominguez, and Monique Tilford.1 By using this approach, the real wage is almost 80% less than what our mid-level developer earns while working. We can also look at a less extreme example. When all activities and expenses that immediately impact our work are considered, the difference in the real wage can be anywhere from 30%-50% lower than our earned hourly rate. When we realize just how much our real wages are, we feel prompted to do significantly different things with our spending and behavior.
The value of time becomes more critical as additional people come into the picture. Time is multiplied, and each non-optimized hour affects all of the individuals involved in a project. Going back to my previous build server example… imagine a daily build currently takes three hours. Adding another server could bring the time of the build process down to two hours. Put into monetary terms, if a team of 10 developers was able to save just two hours each per week, the team would then have 20 hours more per week to add value to the product. That is a saving of $1000/week or close to $50,000/year!
The “real wage” for an organization can be greatly increased by its ability to save time. It can also be greatly decreased by the time it loses.
Is Time Always the Enemy?
It is possible to lose money when time is not taken into account. Time can also be used to an organization’s advantage as well. Using the build server example once again, we will need to spend money to get the hardware and to achieve the time savings. What if we could get the same hardware for ½ the cost? Could we get double the hardware for the same cost? For the past 30 years, computer and networking equipment has roughly doubled its performance every 18 months. This doubling capability has also been closely tied to improvements in RAM, disk space, peripherals, networking devices and the cost of internal components of motherboards and expansion cards. We can see a general trend that machines effectively double in power every 18 months to two years, while staying relatively even cost-wise.
With this in mind, does the time not saved, and the subsequent lowering of the real wage of the organization, offset the potential of gaining double the power in machine performance in 18 months? If the answer is “no”, then waiting could potentially lower costs while still getting a better return on time. Consider what the cost/benefit analysis of buying a piece of hardware now versus later would be (specifically with the idea of the gained time for our example team providing a $50,000 cost savings each year).
What other steps could be put off until later? Consider what happens when automating tests. I have a natural tendency to want to get in early and start working with the system as soon as the UI or other elements are coded. By automating tests early, the ability to test the system early and often should yield a good return on the time spent to automate the steps. What happens if the UI changes? Additional fields are added (or removed), or just relocated to a different part of the screen. My automation efforts will have to be modified (best case) or completely redone (worst case). In this instance, is it better to wait until the UI is finished? The odds (and risk) of the UI being changed are now greatly lessened, and my automation efforts are less likely to require reworking or re-doing.
Balancing Short Term and Long Term Gains/Losses
When I look to balance and consider the costs associated with money and time, it’s important to look at both short term and long term views. There is always a return on investment when it comes to the time put into any process vs. the value of the output. Whether the return is positive or negative, and by how much, is up to the team and the organization. This may also be dictated by issues that we have little control over. Using automation as an example, there is a formula that Deon Johnson refers to as the Automation Return on Investment (ROI).2 The Automation ROI is determined by comparing cost savings, increased efficiency, and reduced risk. The equation we use for this is:
ROI = (Gains – Investment Costs) / Investment Costs
The larger challenge is for us to convince management that there is likely to be a positive return for our investment. It may cost our organization $100,000 to automate a series of tests. If the net result of that automation is quicker time to market and an increase in sales, that’s a strong incentive to make the changes. If the $100,000 spent for automation does not generate additional revenue to equal or surpass the initial investment, it can be argued that the return on investment is negative. In this situation, a manager would be unlikely to recommend continued automation efforts.
What I think needs to be considered is the window of time being used. One financial quarter may be far too short a time period to determine the value of a testing effort. Sadly, quarterly numbers are what drive many organizations. In this instance, a $100,000 deficit may doom a project. However, if a longer view is taken, that initial deficit will be erased. If an investment is made up front, and that investment can get a modest performance increase over an extended period of time, the return on that investment can be considerably higher than not making the investment. Compared to a longer time period (say, three or four years) the benefits and cost savings will become more apparent.
References
1. Robin, V., Dominguez, J, & Tilford, M. (2008). Your Money or Your
Life: 9 Steps to Transforming Your Relationship with Money and Achieving
Financial Independence: Revised and Updated for the 21st Century. New York:
Penguin.
2. How Is Automation Return-On-Investment (ROI) Calculated?, Automated Testing
Institute, http://www.automatedtestinginstitute.com/home/index.php?option=com_content&view=article&id=1097:faq-roi&catid=54:faqs&Itemid=84,
retrieved electronically on August 18, 2010.
3. Johnson, D, Test Automation ROI, Automated Testing Institute,
http://www.automatedtestinginstitute.com/home/articleFiles/articleAndPapers/Test_Automation_ROI.pdf , retrieved electronically on August 18, 2010.
About the Author
Michael Larsen I am a long time tester with experience covering networking equipment software, SNMP applications, virtual machines, capacitance touch devices, video games and distributed database applications/web services. I’ve also tended to work as a “Lone Tester” in many of these environments.
I am the producer of STP’s “This Week in Software Testing” podcast (hosted by Matt Heusser).
I am a black-belt in the Miagi-Do School of Software Testing.
I am a member of the Board of Directors and Chair of the Education Special Interest Group with the Association for Software Testing.
I am a founder and the principal facilitator for the America’s chapter of Weekend Testing.
I wrote a chapter for the book “How to Reduce the Cost of Software Testing”, published by Auerbach in 2011.
I can be found on Twitter at @mkltesthead and I blog at http://mkltesthead.com/.