Selenium, a powerful tool for automated web application testing, drives several automated tools for mobile. We recently discussed mobile automated testing with Selenium (Appium and Selendroind) and we concluded that Appium makes the most popular tool. It’s not surprising, as Appium suits for testing native, hybrid and cross-platform apps as well as mobile web apps tailored for a variety of mobile platforms. What’s more, Appium is an open source tool enjoying a strong community support.

Though the tool fully uses the capabilities of Selenium for mobile testing, Appium does have some drawbacks. The most important are:

• Limited support of Android (API 17 and up).
• Unequal testing capabilities for platforms (some functions don’t work on iOS).
• The tool works slower with iOS and cross-platform apps.

These shortcomings make test automation engineers look for different mobile testing solutions. Let’s compare three popular tools with Selenium-based Appium to help you decide which one fits you best.

Calabash vs. Appium

Calabash is an open-source automated testing framework developed by Xamarin.

Diverse features of Calabash make the tool a full-range competitor to Appium. So how does it work?

Testing process

Test development
Calabash supports Cucumber, an open-source tool for automated mobile testing that enables to express an app’s behavior in natural language. Calabash Cucumber uses Gherkin, a simple business language, which allows non-technical specialists to contribute their perspective to the testing process. Gherkin framework provides for 60 languages, so the team can choose their native language to express their ideas clearly. Later on, test automation engineers write scripts in Ruby using the Gherkin-written specifications.

Though Calabash supports a very limited number of languages, they are very easy to learn. Typically, an automated testing engineer only needs 1-2 days to learn Ruby and proceed with scripting.

Test management.
Test management with Calabash isn’t very comfortable. For example, Calabash doesn’t have an built-in recorder to save test scripts. However, the tool provides a reduced recorder functionality. Though Calabash doesn’t identify elements automatically, it’s possible to send simple queries to console to get element locators. They are used to write scripts.

The tool automatically generates reports, but they may contain many false results in the case of iOS apps testing. Thus, manual testing is required to verify them.

This automated mobile testing tool connects to Xamarin Cloud, which provides impressive scaling opportunities. Not for free, however.

Testing speed.
In most cases, Calabash is faster than Selenium-based Appium.

Calabash Pitfalls

• Modification required. Calabash iOS requires linking Calabash server library to the app.
• Lack of stability. Though the tool provides for wireless testing, the connection often fails. If a test stops, testing engineers can’t run only skipped scripts: they will have to re-run the whole test suite. The tool doesn’t provide for safe plugging and unplugging of a device when its battery runs low. In this case, the test suite stops, and the automated testing team will have to run it again.
• Hardware issues. Calabash doesn’t perform equally well on every device intended for testing. For instance, Calabash doesn’t always work well on old Android versions.
• Farewell to Calabash. In 2017 Xamarin announced that they ceased active development of the tool. However, Calabash supporters continue to use it. Still, a tool that only enthusiasts support may be a bad choice, especially in large long-term projects.

Open-source tools catering for all mobile platforms usually are not equally effective for every platform they target. Therefore, it makes sense to consider platform-specific testing tools. Let’s compare some of them with Appium, our chosen benchmark.

Appium vs. Robotium

Robotium is a test automation framework for native and hybrid Android apps.

Testing Process

Test development.
Though this automated testing tool only supports Java, scripting in this language isn’t difficult. Testers can choose to create test cases using an IDE. This brings us to the nature of testing with Robotium, i.e. whether it is black box or white box. To be on the safe side, specialists prefer to say it’s grey box. Thus, partial knowledge of the source code is required.

Test management.
Free version of this tool doesn’t provide a test recorder. You can buy one for ≈ $500. Just like Appium and Calabash, Robotium doesn’t provide an in-built reporting system. However, the tool easily integrates with other reporting tools due to its open-source nature.

Testing speed.
Testing process with Robotium is faster than with Appium. Test cases are robust due to run-time binding to UI components.

Robotium pitfalls

• Modification required. To use automation API for all platforms, the app under test needs modification.
• No other app can be involved. Robotium doesn’t provide for activating an app in an app under test. If the app under test tries to use the camera, the test automatically fails.
• No support for Flash and Web elements
• No multiple tests. Robotium can only run one test at a time.

Though Robotium overpowers Selenium-driven Appium in testing speed and Android device support, it lags behind in other aspects (required modification of the app, the only programming language for scripting and more).

XCTest vs. Appium

XCTest is a testing framework by Apple. XCTest makes a part of XCode IDE. Both developers and testing engineers can use it, which improves collaboration within the team. The framework enables users to write unit, performance, UI and acceptance tests for iOS mobile apps.

Testing Process

Test development.
Scripting in XCTest requires the knowledge of Swift or Objective-C.
Test management.
XCTest has a recorder, which allows its users to develop test scripts very fast. The recorder provides test recording, generating the code based on recordings and mocking the user’s activities to run the tests. It is also possible to edit a piece of code in a generated script when the app features change.

The tool is supported by Apple. In 2017, XCTest has seen considerable improvements.

Testing speed.
Testing with XCTest is much faster than with Appium and other mobile automation testing tools. In fact, these tools rely on XCTest when testing iOS mobile apps, which complicates the testing process.

XCTest pitfalls

• Limited platform and language support. The tool works for the apps on iOS and employs only two programming languages: Objective-C and Swift.
• Occasional lack of stability. Test cases may work well on emulators but crash on real devices

XCTest is a fine tool for test automation of the apps specifically targeted for iOS. However, software developers usually try cover diverse devices and platforms. Therefore, XCTest loses to Appium in terms of platform and device coverage.

Conclusion: Who wins?

None of the automated testing tools discussed seems to be a universal solution for testing native and hybrid mobile apps on all available platforms. While Selenium-based Appium offers a comprehensive platform and language support, the testing process with it is time- and effort-consuming. Calabash provides for quick testing, but lacks stability, which increases time and effort needed for testing. The same goes with platform-specific tools: addressing one testing challenge, they introduce another one. Therefore, we advise choosing automated testing tools that fit the project best or to combine several tools that address project challenges.

Article Author: