This article introduces an application of Walter, a deployment pipeline, to software tests.
Specifically, how Walter is applied to the integration tests of my personal products, Serverspec and Specinfra are covered.

TL;DR

• Application of Walter to the integration tests of Serverspec and Specinfra are described.
• Combination of Walter and Wercker is nice. Especially parallel executions by Walter in Wercker is useful.
  • Walter parallelly executes tests on CoreOS, CentOS 6.5, CentOS 7.0, Ubuntu 14.04 and FreeBSD 10.1 with Docker

Walter + Wercker

As a dogfooding trial, I apply Walter to the Wercker CI for the integration tests of Serverspec and Specinfra.
The following shows the wercker.yml file for integration testing Serverspec.

box: mizzy/serverspec-base@0.0.6
build:
  steps:
     - script:
       name: Run walter
       code: ./$WORKING_DIR/walter -c ./$WORKING_DIR/pipeline.yml
  after-steps:
     - wantedly/pretty-slack-notify:
     webhook_url: $SLACK_WEBHOOK_URL

The below shows the pipeline.yml called from wercker.yml.

- name: Make $HOME/.ssh directory
  type: command
  command: mkdir -p $HOME/.ssh
  only_if: test "$WERCKER" = "true"
- name: Put SSH publick key
  type: command
  command: echo "$DIGITALOCEAN_SSH_KEY_PUBLIC" > $HOME/.ssh/id_rsa.pub
  only_if: test "$WERCKER" = "true"

We can run the tests in my laptop without pushing the changes to the GitHub repository. It is handy 🙂
When the processes are different between local and Wercker, we can use the only_if feature which allows us to change the behaviors by the environments.

Of course, we can execute the tests with Wercker v2 locally. Unfortunately Wercker v2 is a bit hassle, since it needs Docker nevertheless the integration tests of Serverspec and Specinfra do not need Docker.. Therefore I feel Walter is handy since Walter does not need any VM systems.

Paralell Execution

The tests are run in each VM which is created for the OS on DigitalOcean. The tests are executed in parallel with the following Walter configuration.

- name: Parallel builds each OSes
  parallel:
- name: Build CoreOS
  type: command
  directory: $WORKING_DIR
  command: vagrant up coreos --provider=digital_ocean && ./apply-itamae-and-serverspec-to-docker.sh
  only_if: test "$WERCKER_GIT_REPOSITORY" != "serverspec"
- name: Build CentOS 6.5
  type: command
  command: vagrant up centos65 --provider=digital_ocean && bundle exec itamae ssh --host centos65 --vagrant recipe.rb 
     && DIGITALOCEAN=true rake spec:centos65
  directory: $WORKING_DIR
- name: Build CentOS 7.0
  type: command
  command: vagrant up centos70 --provider=digital_ocean && bundle exec itamae ssh --host centos70 --vagrant recipe.rb && 
     && DIGITALOCEAN=true rake spec:centos70
  directory: $WORKING_DIR
- name: Build Ubuntu 14.04
  type: command
  command: vagrant up ubuntu1404 --provider=digital_ocean && bundle exec itamae ssh --host ubuntu1404 --vagrant recipe.rb 
     && DIGITALOCEAN=true rake spec:ubuntu1404
  directory: $WORKING_DIR
 - name: Build FreeBSD 10.1
   type: command
   command: vagrant up freebsd --provider=digital_ocean && DIGITALOCEAN=true rake spec:freebsd
   directory: $WORKING_DIR

In the above settings, each Walter stage starts VMs (CoreOS, CentOS 6.5, CentOS 7.0, Ubuntu 14.04 and FreeBSD 10.1) in DigitalOcean in parallell,
and then executes the provisioning by Itamae and run tests with Serverspec.

When the test run in sequence, the tests are not finished within the limit (25 minitutes). By contrast, Walter with the above settings finish the all tests within the limit.

We can see the Wercker log here. The following is the image of the log. As wee see, it is a bit difficult to understand the results.
Specifically when some tests are failed in a environment (OS), understanding that the failed tests were run in which OS is not intuitive.

I overcome the problem by the Slack notification feature provided by Walter.

Slack Notification

Walter provide messenger notification. The results of the stages are sent to a Slack or HipChat channel.

When tests of a stage failed, the stage is shown with red as follows

In the above case, we can see the tests of CoreOS are failed.
Clicking the bottom link and move the the build page, we can see the failure is from CoreOS tests.

Future

Currently I am satisfied with the combination of Wercker and Walter. As a next step I will use Walter with Jenkins or other CI systems and add the features to Walter if needed. In addition, I have a plan to provide the Walter server with which users can build CI system without Jenkins or other CI services.