In the traditional approach test automation comes into play gradually upon the product maturing during its development. The automated tests are added during the subsequent development process phases, first as regression cases, once manual unit and functional tests are done, and then as performance and interoperability cases. An alternative approach is to start test automation in parallel with product development and then reuse the automated cases while moving to the subsequent process phases. Such an approach can shorten the products’ time-to-market without compromising its quality, since testability and error coverage are built into the product.
A few steps need to be completed before automation starts:
- system decomposition and abstraction
- error ordering and test design
- testability analysis and test harness integration
- test tool selection and test framework implementation
Once these steps are completed, the automation is straightforward and can be done in short (agile) iterations.
System decomposition and abstraction
A properly designed object is hierarchical; this reduces the size of independent components to simplify the design and, respectively, its verification. For instance, the seven layers of the network protocols allow the verification of each network layer (transport, application, etc) separately. A test for a particular protocol layer does not depend on the states, conditions and errors of other protocol layers. This leads to a sufficient reduction of the test size, while maintaining an appropriate level of error coverage. Let’s take as example a system that consists of 5 components, each of them having 10 possible states. Without knowing whether the components are independent it is required that 100,000 (10*10*10*10*10) combinations of states be analyzed. The fact that the components are independent reduces the total number of state combinations to be checked to only 50 (10+10+10+10+10).
Early in the evolution of test methodology this step was revolutionary. The only hardware model for test design generation was the combinational logic (a net of AND, OR, NOT, etc elements), which used the D-algorithm, the Boolean differences method, etc. This worked until the number of elements reached hundreds of thousands and continued to grow exponentially, according to Moore’s law (the number of components in integrated circuits doubles approximately every two years). The automated test generation systems became powerless. The solution came in the form of using a higher-level model for test generation in addition to the gate-level model: sequential circuits (flip-flop nets), sequential machines (state machines), register transfer level models with an instruction set, application algorithms, etc. This approach reduces the number of elements to be analyzed for test generation down to hundreds. As a result the error resolution is decreased. If the error resolution is insufficient, the next lower-level in the hierarchy can be analyzed, but here again (because of component independence) the number of elements is manageable.
In the software testing world the automation of test generation is rather “exotic”. Therefore independent software components have to be reasonably sized for manual test creation. This is typically true for modern software systems, which are hierarchical in nature, supporting vertical and horizontal separation. The vertical separation uses the following layers: drivers, operating system, middleware, proxies, applications, where a low level provides services to a higher one and is independent from it. The horizontal separation uses independent services on each layer that are accessible through APIs, for instance. This architecture makes the test design process and its automation feasible.
The question is what to do when the necessary level of system decomposition is not defined in the requirements. For example, the middleware implementation depends on the states of the higher layer application. Such a dependency is against the rules and, in most cases, is not documented. This makes the verification of all subsystem combinations and permutations difficult in a reasonable time interval. Therefore, a tester has to provide the system decomposition himself. The size of a subsystem a tester can cover during a release (or an Agile iteration) should be limited to tens of objects: states in a finite automata, “if” statements in an algorithm, objects and methods in an OO model, message exchanges in a ladder diagram, elements of GUI, etc. The approach is to “divide and conquer” and then to describe the solution in an estimation document (or test plan). This, in turn, should be reviewed by the systems engineers for consistency and to estimate the risk. The management either accepts the risk or decides to modify the architecture accordingly.
The outcome of this process is a hierarchical system made up of independent components.
A test is designed to uncover an error. Errors are well defined for formal models (like Rhapsody, Rational Rose RT). If a system component to be tested is described informally in business terms, the test needs to be defined by a subject-matter expert. However, the completeness of such a test cannot be proved. The completeness can be achieved only for a test built to cover all conventional errors of a formal model. There are approximately ten types of commonly used formal models versus a virtually unlimited number of informal business-oriented descriptions. Commonly used models (UML-like) are state machines, algorithms, message ladder diagrams, instruction sets, syntax, logical conditions, etc. As a result, a subsystem can be described by a few nested models. For example, a state machine has logical conditions to transfer between states and employs a ladder diagram to represent its output functions.
The outcome of this process is a hierarchy of formal models.
Error ordering and test design
A test identifies the presence of errors in a given model. Errors are possible distortions of the object’s model: its components’ syntax, structure and behavior. The selection of the level of system abstraction defines the error resolution that can be reached, for example:
- for a state machine model the errors are wrong transitions between states and incorrect state output
- for Boolean condition models errors are incorrect logical function invocations or constant values (1/0) for Boolean variables.
A test is a set of stimuli that forces an object to produce different responses when the object contains an error or when it is error-free.
M → E → T
A test set (T) has to cover the set of errors (E) defined for an object model (M). The advantage of using system decomposition and abstraction is that it eliminates the need to define a test case for each potential error, because for each model there is a pre-defined method to build tests that cover all errors. Most test design methods guarantee the error coverage and attempt to do it with the least number of cases. This is the right approach for regression test, where few errors are expected because the object passed the tests in previous runs/builds.
However, if a test is written for new or sensitive functionality, then occasionally it is desirable to have the maximum test resolution instead of the minimum number of test cases. In these cases a few additional steps have to to be used:
M →E → Eq → {Eq, <} → I →T
The set of errors (E) needs to be split into classes of equivalent errors (errors belong to class Eq if they cannot be distinguished from the system output). Classes Eq have to be ordered based on the masking effect: error e1 masks error e2 if it “prevents observation” of error e2 on the system output.
The outcome of this process is a hierarchy of error classes.
For each error class an identifier (I ) (consisting of one or a few test cases) needs to be built to distinguish the presence of this error class from all others.
The outcome of this process is a hierarchy of identifiers that are ordered according to the error classes’ hierarchy.
However, an unconditional test execution process is easier to implement (the sequence of execution of the test cases should not depend on the previous test results). Therefore, the tree of identifiers should to be translated into a sequence.
Testability analysis and test harness integration
Each test case must comply with the testability requirements, namely controllability and observability. Each test case should be able to be launched from an external interface (accessible by the tester) and the results of its execution should be available for a verdict. A test harness is a set of “instruments” that provides the ability to execute and analyze a test case. Based on the external interface, a test harness can be a set of CLI commands to access APIs or other test functions, or GUI hidden objects to access database values or to send transactions. This is a vital point in test automation. The development of test harnesses places an additional load on the developers (at least from the management standpoint). The success of test automation depends on strong commitment from developers to support the test effort. The testability analysis of the architecture should be done before the start of development in order to include the test harness requirements in the design specification. The availability of the test harness’ syntax makes it possible to start working on test automation in parallel with code development. The test cases can be written during the coding phase and their debugging can start with the private version of the code, before an official delivery. Most code errors are found during the debugging of the test cases. When an official delivery is made, the already automated tests simply have to be run one more time.
The experience shows that the overhead of test harness implementation is less than 5% because it partially replaces the unit test effort and some debugging instrumentation.
Outcome of this process is a hierarchy of harness objects.
They are ordered based on the hierarchy of the system elements (as CLI commands directories or GUI object screens). It is helpful to adopt standards for the test harness objects: syntax and format, confirmation and error messages, naming conventions. These standards simplify the test automation. Extensive support from Jim Pelech, the middleware development team lead, made possible the implementation of the test harness and subsequent test automation of all MW services in parallel with development.
Test tool selection and test framework implementation
Different test tools support different object interfaces: CLI, GUI, http, SNMP, Netconf, etc. They can be commercial or open source tools; they can support a general programming language or a specialized test one. Testers prefer having access to technical support and using a test language, which means they favor commercial tools with support of test frameworks. Most test tools do not have a test framework and consequently, each test organization has to develop its own variant.
What is the difference between a “capture/ playback” script produced by a standard tool and a test framework?
- Objects of a programming language are variables, arrays, hashes; objects of a test framework are test sets (TS), use cases (UC), test cases (TC), test actions (TA).
- Control statements of a programming language are “if”, “case” and “loop”; Control statements of a test framework are compare statements (check real result against expected result), repeat statements for TS/ UC/TC/ TA, etc.
- Functions of a test framework are predefined: setup, execute, compare, reset (tear down) phases.
- Parameters are defined for each test object TS/ UC/TC/ TA, as are runtime variables that can be captured and used in subsequent cases.
A possible hierarchy of test objects is the following:
- A test action is the single act of interaction between the test tool and the object-to-test. A test action, for example, can be the execution of a single CLI command, or a single interaction with the GUI, or sending a single http request transaction to the client. The verdict is “OK/error/timeout”.
- A test case is a series of test actions to move the object-to-test through the following phases: setup, execution, comparison, reset. A test case contains a single comparison act of the real result with the expected one. Theoretically, the first error found should stop the test execution because it can distort the subsequent verdicts, but if the test cases are independent the execution can continue after a single failure. Each test case has to start from an initial state (phase “set”) and the system should to be returned to the same state (phase “reset” or “tear down”) after the test execution, regardless of its verdict. A test case has only one compare statement and its verdict is “pass/fail”.
- A use case consists of test cases. A use case represents a subsystem from a functional point of view and is related to a specific requirement. A use case verifies the business function and can be used for requirements traceability purposes. The verdict is “full/partial/non functional” in relation to the system functionality.
- A test set is a grouping of use cases. A test set represents a system element (sub-system) from an architectural point of view. In the continuous integration process the test set to be run can be selected based on the subsystems that were updated in the latest build. The verdict is “accept/do not accept” the new build.
- A test (regression, functional, performance, etc) is a collection of test sets
For example, the LTE middleware is an embedded system that uses a CLI and contains 50 test sets, 250 use cases, 5,000 test cases, and 25,000 test actions. However, there are 50 unique test sets, 250 unique use cases, but only 50 unique test cases and 20 types of test actions. This is because the same test case is used in different use cases with different parameters.
To make the testware maintainable:
- All test sets are stored in an xml configuration file, since the list of test sets to run is changed daily based on the specifics of the build.
- All use cases are stored in the test management system. Their content changes on the release basis when new functionality is added. Use cases can also be stored as recursive test scripts (containing references to other scripts).
- All test cases are stored in libraries and called by the use cases with different parameters. They contain references to CLI command or GUI elements syntax and therefore changes to the interface will be made in one place.
- All test actions are statements of the test language. For GUI test framework test actions are also called action words. A GUI object requires one more level of the test framework – each test action refers to a virtual object that has its image in the object repository.
The outcome of this process is a hierarchy of test objects that allows the separation of the business function from the implementation.
The test objects provide project development feedback and metrics:
- The test set verdict is used to report the quality of the build and other metrics to product management.
- The use case verdict is used to report the current state of the system functionality to project management.
- The “fail” verdict of a test case is used for CR creation by testers.
- The “error” or “timeout” verdict of the test actions is used for CR creation by testers or developers.
The outcome of this process is a hierarchy of test reports that allows monitoring and controlling a project.
TweetAbout the Author
Gregory Solovey Gregory Solovey is a distinguished member of the technical staff at Alcatel-Lucent. He is a PhD in Computer Science (Thesis – “Error Identification Methods for High Scale Hierarchical Systems”), with more than 25 years of experience and over 60 publications and patents in the fields of test design and automation. He has extensive experience in establishing a test strategy and process, designing production-grade test tools, automating the test design, embedding automation test frameworks, and implementing continuous integration. He is currently applying and enhancing his approaches to develop a test framework for continuous integration of a hierarchical embedded system.