User Acceptance Testing tutorial on Calabash: part 1: Getting Started
This is a blog post straight from http://blog.lesspainful.com/ that describes in overview, what Calabash is. I thought I'd give you an overview of Calabash in parallel with my investigation on Zucchini. I will also do my own posts in a week or two on the actual work on this, as well as Zucchini, but for now, this will provide a good sound understanding of the options you have for User Acceptance Testing.
AN OVERVIEW OF CALABASH IOS
Calabsh iOS is a new open source project for functional testing of iPhone and iPad apps based on Cucumber.
This post describes Calabash iOS in some technical detail. It will give you an architectural overview and compare Calabash iOS with some other functional testing alternatives (hence, it is fairly advanced and assumes that you are familiar with Cucumber and iOS development).
- For an introduction to Calabash and LessPainful (and some info on Cucumber), please read Calabash: Functional Testing for Mobile Apps.
- Instructions for how to use Calabash in your iOS project can be found in the Github Readme.
- For a tutorial on how to use Calabash iOS once setup, please see Getting started guide.
Calabash tests are executed with the Cucumber tool and consist of three parts:
- Your app, linked with the
calabash.frameworkwhich embeds an HTTP server in your app (of course, this is only in test builds of your app). The Calabash client library (see next bullet) makes HTTP requests to the server when it wants to do something with the current view.
- The Calabash client library and step definitions. Calabash supports two types of step definitions: built-in and custom. The built-ins define general and domain-neutral steps that are intended to give you a quick and easy start. They contain steps like:
I should see a "..." buttonand
I swipe right. The custom steps are written by you, and they’re specific to your domain or application. Examples could be:
I should be on the Login screen,
I login as Peteor
I add a track to my playlist. To implement a custom step you can use methods defined in the Calabash client library, calabash-cucumber, which is part ofCalabash iOS.
- Your feature files. The feature files describe the app use-cases you want to test. They are simultaneously a specification of the behaviour of your app and an executable test suite.
You can visualize this as:
Calabash iOS supports running on the iOS simulator as well as iPhones, iPads and iPod touches (whether jailbroken or not). The calabash server framework is distributed as a universal framework for your convenience.
The client library, calabash-cucumber, is installed using RubyGems which is as simple as running
gem install calabash-cucumber.
COMPARISON WITH OTHER FUNCTIONAL TESTING SYSTEMS
There are several options out there for functional testing on iOS. Apple provides its UIAutomation, there isZucchini, Frank by Pete Hodgson, KIF by square, FoneMonkey by Gorilla Logic, iCuke by Rob Holland, and probably a few I’ve looked at and forgotten again!
All the projects have their advantages and disadvantages, Calabash included. But we think that Calabash hits a sweet spot that none other the other options cover.
Frank is the library closest to Calabash; in fact, when we started experimenting with iOS support at LessPainful we started with Frank. Frank is a very cool project by a great person (that’s you Pete), and Calabash is highly inspired by it. But there were a couple of important reasons why we could not use Frank.
First, Frank is licensed using GNU GPLv3 and we wanted to use the less restrictive EPL. Pete actually didn’t want to license Frank as GPL, but since Frank is based on UISpec which is GPL'ed, he had to. We did not want the UISpec project as a dependency.
Second, when we started to look at Frank, it was a bit of a pain to set up (in the mean time, its actually gotten much better). It required modification of you app source, inclusion of static resources and source files. We wanted calabash to be easy to set up, so we made Calabash a universal framework.
Third, Frank is focused on running in the iOS simulator. We wanted something that ran equally well on simulator, iPhone and iPad. There is no fundamental reason that the Frank architecture shouldn’t work on physical devices (indeed Calabash has the same architecture), but many of Frank’s step definitions are simulator only.
Finally, we needed support for some advanced features that Frank just didn’t have. Frank’s touch emulation is derived from UISpec, which has some disadvantages: it only supports tapping, and actually doesn’t seem to work in all cases (for example on a UISwitch). We wanted support for gestures like swipe, pinch-to-zoom, backgrounding and rotation. We wanted support for querying and acting on UIWebViews.
Frank also has the cool Symbiote tool which let’s you explore your app from a browser. We don’t have Symbiote in Calabash, but instead we have a powerful console based tool for interacting and exploring your application (more on this in a later blog post).
KIF, ICUKE, FONEMONKEY
KIF is another cool project. KIF has an Apache 2.0 license which is ok, but two primary things kept us from going down this route. First, KIF is based on the same touch emulation code as UISpec, and has the same limitations (which actually goes back to a seminal CocoaWithLove post from 2008, Synthesizing touch events on iPhone). Secondly, KIF tests are written in Objective-C, and we specifically wanted Cucumber for the reasons outlined in the Introductory post.
iCuke actually looked very promising two years ago. Unfortunately its author, Rob Holland, has abandoned it, and we had a difficult time proceeding with it.
FoneMonkey, is GPL licensed and doesn’t support Cucumber. FoneMonkey is primarily based on a test recording approach, and we specifically didn’t want recording of test scripts since it often leads to large, unreadable and unmaintainable test script . We do actually support a different form of recording that doesn’t suffer from this problem, but that’s a topic for another blog post.
UIAUTOMATION AND ZUCCHINI
Apple’s UIAutomation framework is actually quite advanced in supporting various gestures. It’s also simple to set up since it works without modification to the app (runs via instruments). There are a couple of reasons why we decided not to use UIAutomation.
First, UIAutomation is not open source – this means we have to wait for Apple to make bug fixes, and we can’t extend it. It is also very poorly documented by Apple, and not that much information available on the net.
Third, UIAutomation is hard to get working in a continuous integration setup (in fact, when we first looked at it, CI was completely impossible). Extracting tests results and screenshots in a CI friendly way is hard.