Xcode7, Calabash and Code coverage


Recently I had an opportunity to learn the basics of Calabash and dive in to code coverage measurements related challenges while running Calabash tests.
I hope that sharing my learnings will be helpful for others who are interested in Calabash as well as code coverage while running it.
These were my goals:
1. Create a simple iOS application.
2. Add Calabash to this project
3. Write UI test cases in gherkin and run the tests.
4. Measure the code coverage and create a HTML report with the coverage result.

To give you a head start on this, I have created a sample application called ‘Minion’. It is in github.

1. Creating a simple iOS application.
The idea was to create a simple application that mimic at least some behaviour of real life apps.
Minion is a weather checking app with just two screens. The first one is a login screen with hardcoded credentials. After the successful login, user can enter a valid city name in the textfield and weather summary of that city will be displayed.

2. Adding Calabash to project
This may look complicated at first with many steps, but believe me the steps are trivial.
There are two types of steps. The first set of steps are done within Xcode where you set some configurations and create a new target.
The other set of steps are about setting up the environment in your workstation. You would download and install few frameworks and ruby gems here.

  1. Creating a Cal target: Follow steps from one to three in this page to perform the first set of steps. All the steps are explained here with relevant screenshots.
  2. Download calabash-ios framework: CD to the project directory. Execute the command:calabash-ios downloadContinue from step 4.2 till the end in the above link.
  3. Generating features folder:Execute the command:  calabash-ios gen

Please note that  all the Xcode configuration related steps explained above are already performed in Minion.

Once over, proceed with environment set up related steps.

  1. Install Homebrew if you do not have it already.
    /usr/bin/ruby -e “$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)”
  2. Install rbenv:
    brew install rbenv
  3. Install Ruby.
    brew install ruby
  4. Install Calabash-cucumber
    gem install calabash-cucumber
  5. Install Xcode command line tool if it is not there
    xcode-select –install
  6. Install lcov for code coverage
    brew install lcov

If you find yourself in trouble, please feel free to refer to this, this and this links.
3. Writing UI test cases in gherkin and run the tests.

Each of the two screens in Minion got negative and positive test cases.
For the sake of simplicity all the gherkin statements used in this project are pre-defined. If you are interested you can see more pre-defined steps here and here. Please note that in real applications, you may need to write custom ruby scripts.

4. Measuring code Coverage when Running Calabash tests in Xcode 7

Measuring code coverage in Xcode differs with its version. For Xcode 6 and prior versions, there are specific build options which can be configured under Xcode build settings.
The build settings ‘Generate Test Coverage Files’ and ‘Instrument Program Flow’ can be turned on prior to building the app. When the app is built, .gcno files will get generated. After the app is executed, .gcda will get generated. After this point there are open source frameworks like Slather or lcov which can generate reports based on .gcno and .gcda files.

But for Xcode7, Apple introduced a new format, .profdata replacing .gcno and .gcda files. And .profdata will get generated only while running unit tests and instruments based UI automation.

Problem statement at this point can be summarised as below:
If your project is in Xcode7 and you have an Calabash based UI automation suite, how would you measure the code coverage?

I could not find an answer for this until I discussed this issue with TeresaP of Calabash team. She was gracious enough to spent some time on this and figured out a solution. These are the high level steps in her approach.
1. Force Xcode to generate .gcda and .gcno files using .xcconfig file since those options are no more in Xcode build settings UI.
2. Using custom ruby scripts, process each of the .gcda files in to a single .info file after each test cases. Combine all the .info file and generate HTML report using lcov tool.

Here is her blog page describing the steps. Here is the gist of ruby files used for this solution. You need have these files copied locally (Home>Features>Support) in case you are using your own application. Also you need to chanage @app_name in code_coverage.rb and add the name of schema in env.rb.

Once done, all you have to do is running the command:

cucumber CODE_COVERAGE_BUILD_PARENT_PATH=/YourApp/Home/Directory/

This will trigger the tests. Once it is finished, you should see the HTML reports in CodeCoverage folder.

Xcode7, Calabash and Code coverage