Ah, the wonders of automation. There is a lot of controversy surrounding this beautiful world where machines can work for or even replace humans. Some people believe that automation can take the job of manual testers, whereas others are more rational and know that an automation script will only reproduce what it was programmed to do. If you are not familiar with the testing versus checking argument, I strongly suggest you read Michael Bolton’s homonymous blog post1 as you might understand the big difference between both activities. The world of automation is also full of myths and promises that, if you are not aware of, might become dangerous money blowing pitfalls. This article is intended to help you identify and avoid some of those pitfalls, which can happen in many different contexts. Can you relate any of them with your own?
Snake Oil, Silver Bullet, You Name It
James Bach originally published the Test Automation Snake Oil article2 in the Windows Tech Journal back in 1996 (and revised it in 1999). As a non-native English speaker, I must admit that I had to look up what he meant by the term snake oil which I learned is a term for mixtures highly marketed by traveling salesmen in the 19th century that claimed to cure all kinds of problems. Even though the article was written 15 years ago, it is still a must-read for every tester, automation engineer, developer and most importantly, manager. James carefully and thoroughly debunks a classic argument for automation given by a tool vendor from that time that is still used nowadays by many companies and managers. The myth: Computers are faster, cheaper and more reliable than humans; therefore, automate.
When managers fall for that vendor talk, we often face the following common pitfalls.
Top Down Decisions to Automate
Monday morning. The sun is shining and the birds are singing as you walk into the office. You check your e-mail as you usually do and notice that your manager scheduled a meeting to talk about automation efforts for your current project. You take a big sip of coffee and rub your eyes to see if you are still dreaming or really awake. You realize that you are in fact awake and the meeting will take place in about 30 minutes. You start to get nervous because you have no experience in automation and the reason you decided to be a tester was to never touch any code again. When the meeting starts, your boss says that he had just read an article about the wonders of automation and that your team would be able to deliver higher quality products faster if you had scripts for your test cases instead of four full-time testers. Your whole team becomes nervous; some are afraid of losing their job, while others think that’s an activity for the development team. The article was actually a white paper given by a big software vendor. Your boss was convinced that this was a smart move and there was no turning back.
What was about to happen was another well-known pitfall, the use of record and playback tools.
From my experience, record and playback tools are garbage. Yes, garbage. If you ever see it as a killer feature or a major selling point of any tool, watch out! I could easily call any of those tools “record, refactor and playback” instead. Don’t take me wrong, they might be good for learning, but you’ll probably spend a lot of time trying to clean up some “automagically” generated code that is barely understandable or useful. Let’s take Selenium IDE for example, a great open source learning tool and also very useful for quick and dirty automation. I have attached scripts written on the Selenium IDE to bug reports in order to help the developers reproduce and resolve the bugs more quickly. However the element locators generated by the Selenium IDE are usually very weak and confusing, especially for dynamic elements, requiring some effort to make them more human readable. So, if you are going through the hassle of debugging and cleaning up a mess, why don’t you write good code from the beginning instead of relying on the generated code? Automation code is expected to change and evolve, but please do yourself, and your team, a favor and try to avoid a nightmare from version one. Spending some minutes to achieve good design in the early stages will probably save you hours in the future.
Since we mentioned open source tools and coding, we are now comfortable to talk about the next pitfall:
Reinventing the Wheel
As we have seen in the beginning of this article, automation is nothing new. That being said, some problems you may face in your context have probably been already solved by someone else. Before you start cracking your knuckles to write an automation tool or framework from the ground up, take a step back and do a little research to see what is already available on the market or shared in public source code repositories (please pay attention to the license used). In case you need some very specific feature that is not available in any tool, there are plenty of open source projects that you could get involved with and join forces to make them even better.
Ask yourself if you really need to spend several hours writing a piece of code when you could just use something that is ready to be used and actively maintained by other people or companies. I have seen people starting monthslong projects just to be able to have ownership of a tool, instead of using well known and active solutions. There are clearly ego and pride problems involved, as software is written by people, and people still have emotions! Can you guess what happened in the long run? The tools, often with unmaintainable code, were used only by their authors and were eventually discontinued.
You might find it strange that the title of this article includes the word cost, yet I haven’t mentioned any figures throughout the text. Well, what is the most valuable and priceless commodity in life? That’s right, your time. If you were able to relate any of the aforementioned scenarios with your own experiences and take action to avoid or stop wasting time and money, I consider my mission accomplished. These were just a few examples to think about and I’m sure you can probably come up with or have already faced other scenarios. I do not think that automation is a waste of time. Bad choices, bad design and narrow minds, however, lead to a huge waste of time. Do what makes sense to your project, be it acquiring a new tool, investing into programming courses, or not using any tools at all.
References
1. http://www.developsense.com/blog/2009/08/ testing-vs-checking/
2. http://www.satisfice.com/articles/test_ automation_snake_oil.pdf
About the Author
Felipe Knorr Kuhn I’m a passionate software tester from Brazil.
My interest in software testing started when most of my time was still considered free time and I used to spend most of it playing video games (which also helped me to learn English :).
Unfortunately, getting a job in the gaming industry in Brazil is very difficult, as it practically didn’t exist until recently. So, I left that plan aside and started working as a full time tester in the corporate software industry.
I am currently working most of my time with test automation web applications, because I’m always going to be a programmer at heart.