About STP / 877.257.9531
Log In Join Now
Community / Events / Training / Resources / ST&QA Magazine / Community Forum

Author



Rating

4


Published

Tuesday February 14th 8am

The Testing Rut

Automation Development Testing

I think we can all recall that first time we truly understood test automation. It was a difficult thing for me to wrap my head around at first. Not because I couldn't understand the process of verifying inputs and outputs, but because I couldn't come to terms with writing code to verify that some other code I wrote was correct. Why would my tests be any more trustworthy a source?

As a developer, test automation affords a lot more. Test Driven Development (TDD) showed me how I could use tests to flesh out an API. BDD showed me how I could use tests to generate a behavior specification. Refactoring showed me how I could make use of all those wonderful tests I wrote to imbue a level of confidence around code modifications that I never had before. On the QA front, I could save hours of effort running through test plans when a new build was ready.

Our industry has fundamentally changed with the widespread adoption of test automation. Even if you question the merits of all the various methodologies surrounding it, experience with testing is a prerequisite for most development or QA jobs nowadays. For many of us, it's the only way we know how to work. And I believe we're all the better for it.

The problem is we've reached a local maxima and have done precious little to get out of it. In the past 15 years, while testing practices have evolved, so have languages, frameworks, hardware, execution environments, operating systems, and just about everything else. The core tenets of test automation transcend most of these differences, but little has changed in our testing practices to account for our new world. Many developers approach testing in a dynamic language like Ruby the same way they would a static language like Java. Or a procedural language like C++ and a functional language like Clojure.

Sometimes the similarities are blatant: the jUnit library has been ported to an untold number of languages. Other times the similarities are in spirit: tests are specified as a procedural series of steps that stage data, provide an input, or process an output. We've convinced ourselves that we're doing something different. RSpec certainly looks different from TestUnit. TestNG solved a real problem with jUnit at one point. But really, we're stuck in that same region of testing. There are not enough people looking at the process from afar. We've been optimizing testing with a myopic eye, rarely questioning if what we're doing still makes sense or if there could be a grossly better approach.

For my part, that's what I'm attempting to do with Web Consistency Testing. For years, the standard approach to cross-browser testing has been to open a page in two browsers and manually inspect for differences. I've yet to come across anyone that believed this was an appropriate solution, yet nothing better had been developed.

Web Consistency Testing automates that process. Five years ago, running a cluster of browsers would have been prohibitively expensive for most us. With plummeting hardware costs, that's no longer an issue. Changes in the environment have created a new opportunity.

I didn't just want to automate these page comparisons, however. I wanted to improve the way Web pages are tested in general. Rather than meticulously specifying inputs and verifying outputs, Web Consistency Testing provides a tunable testing engine that compares page representations. It is domain-specific: the engine would never work for native GUI applications. It is fuzzy: tests may pass even if two pages are slightly different, because we care how the customer is impacted, not whether we can line up pixels identically. It's wholly adaptable: you don't need to rewrite tests as your Web site or app changes, making it ideal for rapid development cycles. And it requires no programming: testing Web pages can be performed by anyone once you have a Web Consistency Testing system set up.

Maybe I'm on to something, maybe I'm not. My original notion for Web Consistency Testing was born out of my background in machine learning. As it turns out, the problem domain was much simpler than that and I've been able to make use of very simple geometry. As I spent time working on it, I discovered the same approach for cross-browser development could be used for regression testing or comparing different localizations. A lot of things that seemed hard or impossible from the outside became much simpler when looked at in the right light.

Even if Web Consistency Testing is ultimately the wrong way to go, it's at least a step in a different direction. Maybe along the way someone sees something that looks even better and we can then head down that path. But without taking that drastically different leap, without questioning the status quo, we're destined to remain stagnant and believing we've hit our global maximum not for any technically valid reason, but simply because we never bothered to try anything else.
About the Author:
Kevin Menard Kevin is the founder of Mogotest, a web consistency testing service that aims to change the way we test web sites and applications and simultaneously make testing available to all. He's been involved with many open source projects over the years, most notably the Apache Cayenne and Tapestry projects, Selenium, svn2git, and the rubber Capistrano plugin for cloud provisioning and deployment of Ruby applications. He is an Apache Software Foundation member and the current maintainer of the Selenium Grid project.

Come see Kevin lead session 804 Web Consistency Testing on Wednesday, March 28th at the Software Test Professionals Conference in New Orleans. Kevin's session is part of the Test Automation track. Learn more at www.STPCon.com.


Comments

You must be logged in to comment.
Retrieving Comments...


Advertisement






Friend SoftwareTestPro on Facebook
Follow @SoftwareTestPro on Twitter
Create or Join a Crew

Tweets You Care About


    



Explore STP
About STP / Contact Us / Advertise / Terms of Use

This site's content copyright © 2003−2010 by Redwood Collaborative Media. All rights reserved.
Software Test & Performance™, STP™, Software test professionals™ and Software Test & Quality Assurance Magazine™ are trademarks of Redwood Collaborative Media.