Walter, a deployment pipeline was released almost exactly one year ago. Although it has been slow going since then, the development is still ongoing. This blog entry explains how Walter has been improved since its release. These improvements can be classified into several types. This post will explain each of them in the following section.

• Environment variables
• Support properties used in stages (directory, only_if)
• Support cleanup pipeline
• Reuse of already-defined Stages
• Reuse of Stage results

Environment variables

Environment variables can now be used on Water pipelines. Environment variables are written in the pipeline settings file using $VAR. All environment variables at the execution of Walter commands are available. For example, the following command outputs the details of the environmental variable GOPATH.

pipeline:
  - name: print GOPATH
    command: echo $GOPATH

With the following settings, the safety can be improved by describing information that you do not want to register to a repository, such as the keys required for HipChat, as an environmental variable.

messenger:
  type: hipchat2
  room_id: $HIPCHAT_ROOM_ID
  token: $HIPCHAT_TWO_TOKEN
  from: $HIPCHAT_USER_NAME

New Properties

Walter users add lists of stages to the pipeline blog and describes their processing. Normally, only commands (command) and stage names (name) are specified in Walter’s stages. For example, in the following stage added to the pipeline below, the name “show the files” and its command (ls -la) are specified.

pipeline:
  - name: show the files
    command: ls -la

Property: only_if

only_if property executes the stage only when the specified conditions are satisfied. In the following example, the command specified using “command” is executed only when the value of the environment variable, WERCKER is set to TRUE.

pipeline:
  - name: Put SSH public key
    command: echo $PUBLIC_SSH_KEY > $HOME/.ssh/id_rsa.pub
    only_if: "$WERCKER" = "true"

Property: directory

directory specifies the directory where Walter executes the command. For example, the following Is stage displays a list of the /usr/local files.

pipeline:
  - name: show /usr/local files
    command: ls -la
    directory: /usr/local

Cleanup Pipeline

The newly-added cleanup pipeline defines the post process. Specifically, cleanup is executed after the process described in the pipeline has finished.

The following pipeline block changes the input.data file to Apache Solr input format before registering the data. The cleanup pipeline operates after the pipeline has been finished. In the following example, the data created for Solr inputs is deleted.

pipeline:
  - name: create data
    command: create_solr_data.rb -i input.data -o output/solr-data.xml
  - name: register data
    command: $SOLR_HOME/bin/post -host $SOLR_HOST -port $SOLR_PORT -c car-instance -o output/solr-data.xml

cleanup:
  - name: cleanup xml data
    command: rm -f output/solr-data.xml

Reuse of Already-Defined Stages

There have been requests to reuse stages already defined in Walter configurations. Mechanisms to import files that include stage definitions (stage definitions files) are now supported.

An example of a stage definitions file is shown below. The stage definitions file describes the namespace and stages. The namespace was added so that names are not collision even when multiple namespace definitions files are called.

The def blocks are added in the stages block, and the stage defined. In the following example, the whomai stage is defined.

namespace: mystages

stages:
  - def:
    name: whoami
    command: whoami

The stage definition file described above is saved using the name stages.yml. The require block imports the stage definitions file from the stage definitions file. The defined stage is called using call. The called stage is described using the namespace::stage name format.

require:
  - stages.yml

pipeline:
  - call: mystages::whomai

Reuse of Stage Results

The results of previous stages in the pipeline can be acquired using three special variables (__OUT, __ERR, __RESULT).

__OUT: Stores the result (“true” or “false”) output to standard outputs.
__ERR: Stores the results output to the standard error outputs.
__RESULT: Stores the command return value.

The variables are used as maps. Assigning a stage name as the variable key acquires the stage results.

For example, in the following example, the second stage uses the output results from the first stage. Specifically, the input markdown document (markdown.md) is checked during the first stage with an OSS proofreading tool RedPen, and then in the second stage, whether or not an error was output to the standard error outputs of the first stage is checked. If an error was not output during the second stage, a PDF is created.

pipeline:
  - name: Check document with RedPen
    command: redpen -c redpen-conf.md document.md -f markdown
  - name: Convert markdown to PDF
    command: markdown-pdf -o document.pdf document.md
only_if: test -z "`echo __ERR["Check document with RedPen"] | grep ERROR`"

Misc Improvements

• Slack communications are now supported
• Communications with the local HipChat (proprietary) server are now supported
• Communications with Slack are now colored

Future Work

Ongoing development of Walter is scheduled for the future. In particular, this winter, Walter is scheduled to support server functions. We also want to enable the stage start conditions to be specifiable. Currently, when the previous stage finished, the next stage ends up starting immediately, but specifying the start conditions will enable settings such as wait until the necessary port is ready to be set.