Tuesday December 20th 2011 10am
On Integrating Continuous Integration & Test Management
Test and QA
The process of transforming source code files into computer programs is called software build. When you create a computer program, depending on the size and type of this program, you may have to use many different tools (IDE, compiler, linker, analyzers, and so on) to build it. Build servers are programs that help you to organize many software builds using different tools and environments. In the process of creating a computer program you may need to test it too, and maybe use test plans, a structured tree with test suites and test cases or attachments as evidence of test execution. There are tools that help you to manage testing your computer program, the test management tools. In this article we will see how to integrate continuous integration and test management, as well as practices that help in doing so.
The first and probably more widely accepted definition of continuous integration is the one made by Martin Fowler. Circa 1999, 2000, Martin Fowler wrote an article on continuous integration, a practice that appeared with Extreme Programming (XP). The definition in this paper is as follows: “Continuous Integration is a software development practice where members of a team integrate their work frequently, (…) leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible” .
In short, we can say that CI is a set of good practices that include automation, committing many times in a short period of time, running tests, communicating all parts involved in the project, using a source code management system, using a clone of the production environment to execute your tests, among many others . Build servers (aka continuous integration servers) help you to compile, test and execute other tasks to build your software, and when used correctly can help you to achieve continuous integration.
Test Management Tools
When you think about tools for test management, probably the first thing that comes to your mind is a web system, where you create test plans, test cases and other artifacts used for testing other applications, systems like TestLink, QTP, HP Quality Center or even the brand new CaseConductor – result of a recent joint force of uTest and Mozilla.
However, there are different types of tools that can be used to manage tests. A team could use only e-mails to manage their tests, for instance. Or create many sheets with detailed explanation on what is being tested. I have seen teams working only with post-its and Kanban boards . These are only few examples of what could be used to manage tests.
Having a test management tool, be it a completely web system or a custom solution, the integration of such a system with a continuous integration process can bring significant benefits to the team.
How to integrate a CI process with test management
There are different ways to integrate a CI process with test management. They differ from each other in time, detail, direction and granularity.
You can execute your tests in your build server and only then update the status in a test management tool. Or you can execute a build that prepares a test environment and generate test data for your tests. The synchronism between the two points varies according to your environment, and there is no way to state when is the right time to execute one or another without knowing details about your set up.
When I integrated a test management tool into a CI process for the first time, it was a web test management system with a build server. The tests were executed in the build server and then only the execution status – either passed or failed – was sent to the test management tool. Later, after some requests of different teams, we included screen shots, log files, details on the environment and other data available during the test execution. The level of details sent to your test management tools, or stored in your build server, may vary according to the legislation, time to customize your solution, complexity of the project and other internal and external factors.
The build server can be used to trigger the execution, manual or automated, of tests. It is also possible to customize a test management tool so that it triggers the build server to execute a job that runs automated tests. External agents can also be used to orchestrate this integration. The decision on which direction to use – from the test management tool to the build server, vice-versa or using an external agent – depends on the context of the application. However, usually it is a good idea let the build server take care of handling multiple executions, unless the test management tool supports some kind of internal synchronism, as many teams could trigger the execution of a set of automated tests, this could clog the queue of executors in the build server.
One test case in a test management tool can be mapped to one test case in your test executor tool, within your build server. Or it could be mapped to a test suite, or to a test project. Maybe even only for a custom field of your test case. Or the contrary could happen too. You could have a way to execute all the builds related to a certain test case in your test management tool.
Jenkins and TestLink integration
Jenkins  is an open source (MIT) continuous integration server (ici!), former Hudson, created by Kohsuke Kawaguchi. It is a small Java application, packaged as a WAR with an embedded Winstone servlet. It does not need any database, as it saves its configuration in XML files. Although it is written in Java, it is language agnostic for building software. And it is highly extensible, with plug-ins for things like firing USB missile launchers , displaying custom images in a build page , running Selenium web tests, analyzing source code and so it goes.
TestLink  is an open source (GPL) test management tool, written in PHP and with a grand number of users. TestLink provides a XML-RPC API for external access with many available operations. Its structure (test plans, test cases, test suites and the other artifacts) is very well organized and it is also possible to import and export some artifacts like test plan, test suite and test cases.
The integration of both tools is done through a plug-in in Jenkins  that uses TestLink XML-RPC API to 1) retrieve automated test cases from TestLink, 2) execute automated build steps, 3) scan test results in Jenkins, 4) update test execution status and upload test attachments and 6) generate reports and graphs in both Jenkins and TestLink. The illustration below illustrates how it works.
Using this plug-in, your CI server – Jenkins – remains the same as it was when you used it for only continuous integration, except by the installation of the plug-in. And the only changes in your test management tool – TestLink – is the creation of a key custom field (used like a primary key in a database table to identify an entity) and mark which tests are automated.
Jenkins is extremely good at running builds in distributed environments, under high concurrency and parallelism. It can be through plug-ins, which have a stable API that lets the plug-in developers to create graphs, customize the UI, use other plug-ins’ code and so on.
Automation is a key concept for both, continuous integration and test management. Neither automation nor integrating CI and test management are silver bullets. You still have to weigh whether it is beneficial integrating both worlds or not.
Tips if you are thinking about integrating CI and test management
Before deciding to integrate CI with your test environment, there are some points that you may want to observe on both sides. The first one is malleability. You will have to deal with different tools and environments. Some of these tools and environments follow good practices, others do not. It can be extremely hard to build a generic way to integrate continuous integration and test management processes.
Another important point for both test management tools and build servers is reports generation. And this is important for both test management tools and build servers. Sometimes you will need to match what was executed with what was tested and looking at graphs and well formatted tables may be easier than looking at numbers or XML files.
The final common points are software license, good documentation and an active community. Depending on the license of the software that you picked up, you may not be allowed to change certain parts of it to attend your needs. Lack of documentation can discourage or delay your integration. And the community usually provides good feedback, sometimes giving good alternatives to what may look like an unsolvable puzzle, or feedback on parts of your integration. As well as answering questions faster than search engines (sometimes).
From the point of view of the test management tool, there are two important points to be noted. The first one is a good external API. There are different ways to create an external API, like with a XML-RPC server, SOAP or RESTful Web Services or a custom server application – for example, we could create a simple server program that listened for HTTP POST requests and returned some custom JSON stream.
In this article we haven't addressed the good practices to create external API’s, nevertheless, there are certain points that you must take into account when planning to rely on an API to integrate your CI process with a test management tool, like security, operations available and data structures returned from the server.
The data structures returned from the server lead us to the second point in a test management tool, a good internal structure. There are many artifacts used to test a system, like test cases, test plans, the hardware used, test data, available testers, and so on. There are, as well, many different ways to organize these artifacts. We could have a test suite with many test cases, or simply put the test cases a level under test projects. Or create a test case within another test case. A complex structure in your test management tool may turn integrating it with a CI process painful or even impossible.
Now let's have a look on the CI process. The first point is having a build server like Jenkins, Hudson, CruiseControl, Go, or similar. Should you prefer using scripts or different programs, then one piece of advice is trying to use only one language (Groovy, Korn Shell, Windows Batch, Perl, etc.) and try to create a simple build first, rather than building a complex and hard to maintain environment.
Even if you foresee a complex build, a good technique may be to start with a simple build and integrate it with your test management tool. Then repeat the steps incrementally, always trying to improve the whole process.
However, if you are using a build server, then there are some good points to outline here. The first one is modularity. A build server may miss one or two features that you need, perhaps parsing a custom format, or generating custom reports. The build server must give you a way to extend and customize it, according to your needs.
Parallelism and concurrency are important characteristics too. A project with more than one thousand tests is very likely to be broken into smaller projects and assigned to different teams. These teams may work building the solution in parallel. Thus, this build can execute the tests, or update the test execution status in a test management tool at same time, or using the same data in parallel. A build server that uses nodes or a master and slave model and can be horizontally scaled is probably a better choice to execute your builds and integrate it to a test management tool.
Last, but not least, you have to have fun. Accessing external API’s, generating internal documentation, customizing your CI server or test management tool, and seeing your build server executing your tests and reporting it back to your test management tool is grand! If you are not having fun either you have to work a bit more to overpass an annoying part, or you better review your test process, CI process, tools, internal culture and other factors that may be causing this bad feeling.
 Martin Fowler. Continuous Integration.
 Kenji Hiranabe. Visualizing Agile Projects using Kanban Boards. http://www.infoq.com/articles/agile-kanban-boards
 Jenkins Continuous Integration Server. http://www.jenkins-ci.org
 A Jenkins CI build monitor with a twist! http://www.youtube.com/watch?v=1EGk2rvZe8A
 Jenkins Chuck Norris Plugin.
 TestLink test management tool. http://www.teamst.org
 Jenkins TestLink Plugin. https://wiki.jenkins-ci.org/display/JENKINS/TestLink+Plugin