Automatic Java Artifacts Upload to Nexus Repository with Walter and Git Hooks

I have been developing a tiny tool, RedPen. The tool is for proofreading input texts written in Markdown or LaTeX formats, and written in Java. As this tool are released, the jar files are uploaded into the Nexus repository.  For uploading, we used the Maven release plugin.

Although the settings of Maven release plugin (“pom.xml”, electronic signatures, etc.) is bothered, the tool itself is convenient. We can upload the artifacts to the Nexus repository just running the mvn release command at a release timing. However, I have recently feel that the release process with the plugin is troublesome.

This article introduces an automatic release process covering adding version numbers and upload artifacts to the Nexus repository with Walter, a tiny deployment pipeline and Git Hooks.

But first, I would like to introduce the release process with the Maven release plugin.

Release process with Maven release plugin

When we run the release plugin, we need to input misc information interactively. The following is the status for the preparation of a release plugin running mvn release:prepare.

➜ mvn release:prepare
[INFO] Scanning for projects...
...
[INFO] Checking dependencies and plugins for snapshots ...
What is the release version for "redpen"? (cc.redpen:redpen) 1.7.5: :

Then, we are requested to input the release version, though the release plugin suggests the release version guessed from the current snapshot version (1.7.5 in the above example). If the suggested version is not what we want, we input the version number manually.

Next, the plugin requests a development version number, and the we input it manually if necessary.

What is the new development version for "redpen"? (cc.redpen:redpen) 1.7.6-SNAPSHOT:

After that, the plugin commits the new version number to local git, and then the build is executed. When the build succeeds, the plugin notifies that it will push the version number change to the remote repository.

[INFO] Working directory: /Users/takahi-i/IdeaProjects/redpen
Username for 'https://github.com': Password for 'https://github.com':

As we see, the plugin requests the GitHub account name and password. In addition, even after mvn release:prepare succeeds, we still need to input interactively even for the next release command,release:perform.

Automatic uploading artifacts to Nexus with Walter

As we see the release plugin is quite tricky due to its interactive input operation.

Prior Examples

Recently I heard that we can use mvn deploy command for the release purpose and the command is simpler than Maven release plugin.

Applying Walter

We can apply mvn deploy Jenkins server for the release automation, and off course we can use external CI services such as Wercker or Traivs for this purpose. But just for the release process of a tiny project such as RedPen, both solutions are a bit costly. In addition, analyzing the log files in remote servers is bothered when the release process in such services does not work.

Therefore, I apply Walter for the automation of release process. Walter automates the release process and the process is run locally. Please refer here for details of Walter. Below is the configuration file for the automation of the RedPen release process.

pipeline:
  - name: Set Version
    command: mvn versions:set -DnewVersion=$REDPEN_VERSION
  - name: Add Files to the Next Changes
    command: git add pom.xml; git add **/*.xml
  - name: Commit Version Changes
    command: git commit -m "Set version in pom.xml to $REDPEN_VERSION"
  - name: Delopy to Sonatype
    command: mvn clean deploy -DperformRelease --settings ~/.m2/settings.xml > /dev/null
  - name: Create Release Tag
    command: git tag -a redpen-$REDPEN_VERSION -m "RedPen release $REDPEN_VERSION"
  - name: Flush Next Release Version Number
    command: echo $REDPEN_VERSION | awk -F. -v OFS=. 'NF==1{print ++$NF}; NF>1{if(length($NF+1)>length($NF))$(NF-1)++; $NF=sprintf("%0*d-SNAPSHOT", length($NF), ($NF+1)%(10^length($NF))); print}'
  - name: Echo Next Version
    command: echo __OUT["Flush Next Release Version Number"]
  - name: Set next version
    command: mvn versions:set -DnewVersion=__OUT["Flush Next Release Version Number"]
  - name: Git Add
    command: git add .
  - name: Commit New Snapshot version
    command: git commit -m "Update versions in pom.xml for next release"

The first stage sets the version that was given from the environment variable. Next stage registers the changes to git. The artifacts are now ready to be uploaded on Nexus. Then the fourth stage (Delopy to Sonatype) uploads results to Sonatype with “mvn clean deploy”.

After the deployment, Walter assigns the development version for the next release. The development version name in Java projects commonly has a “-SNAPSHOT” suffix. The stage, “Flush Next Release Version Number”, calculates the next version number, and then set the version with the “mvn versions:set” command.

The release process is then completed just running the “walter -c release.yml” command (off course we need to close and release the stating repository by manual operation in Sonatype).

Combination Walter and Git Hooks for Further Automation

The release processing is able to be completed with a single command. However, one problem still exists. RedPen has the ability to check its versions with a command and server. The version is set by overwriting one line of the RedPen.java file.

Unfortunately I have an experience releasing RedPen without increasing the version number. For the problem, I applied Git Hook to solve this problem with Walter. When a specific git command is run, Git Hook automatically runs an assigned script.

I set Git Hook script to run the release command automatically with Walter only when the commit message is “Bump version…” at the time of git commit. Specifically I put the following Git Hook script file in ./git/hooks/post-commit.

#!/bin/bash

LATEST_MESSAGE=`git log -1 --pretty=%B`

if echo $LATEST_MESSAGE | grep -q "^Bump version " ; then
  export REDPEN_VERSION=`echo $LATEST_MESSAGE | grep -o "[0-9.]\+"`
  echo "Extracted Version: " $REDPEN_VERSION
  ./walter -c release.yml
else
  echo "Not changed version"
fi

With this file, when the commit message shows “Bump version” the release processing will run (the commit message that changes the RedPen version is Bump version). This trick prevents accidental release without increasing version number.

Future work

In the RedPen project, a sample server on Heroku is replaced to a the latest every time a commit is pushed to the remote repository. Although I am currently using Jenkins for the replacement of servers, I will try the replace of the process in Walter as the next step.

Advertisement

New Features of Walter

Walter, a deployment pipeline was released almost exactly one year ago. Although it has been slow going since then, the development is still ongoing. This blog entry explains how Walter has been improved since its release. These improvements can be classified into several types. This post will explain each of them in the following section.

  • Environment variables
  • Support properties used in stages (directory, only_if)
  • Support cleanup pipeline
  • Reuse of already-defined Stages
  • Reuse of Stage results

Environment variables

Environment variables can now be used on Water pipelines. Environment variables are written in the pipeline settings file using $VAR. All environment variables at the execution of Walter commands are available. For example, the following command outputs the details of the environmental variable GOPATH.

pipeline:
  - name: print GOPATH
    command: echo $GOPATH

With the following settings, the safety can be improved by describing information that you do not want to register to a repository, such as the keys required for HipChat, as an environmental variable.

messenger:
  type: hipchat2
  room_id: $HIPCHAT_ROOM_ID
  token: $HIPCHAT_TWO_TOKEN
  from: $HIPCHAT_USER_NAME

New Properties

Walter users add lists of stages to the pipeline blog and describes their processing. Normally, only commands (command) and stage names (name) are specified in Walter’s stages. For example, in the following stage added to the pipeline below, the name “show the files” and its command (ls -la) are specified.

pipeline:
  - name: show the files
    command: ls -la

Property: only_if

only_if property executes the stage only when the specified conditions are satisfied. In the following example, the command specified using “command” is executed only when the value of the environment variable, WERCKER is set to TRUE.

pipeline:
  - name: Put SSH public key
    command: echo $PUBLIC_SSH_KEY > $HOME/.ssh/id_rsa.pub
    only_if: "$WERCKER" = "true"

Property: directory

directory specifies the directory where Walter executes the command. For example, the following Is stage displays a list of the /usr/local files.

pipeline:
  - name: show /usr/local files
    command: ls -la
    directory: /usr/local

Cleanup Pipeline

The newly-added cleanup pipeline defines the post process. Specifically, cleanup is executed after the process described in the pipeline has finished.

The following pipeline block changes the input.data file to Apache Solr input format before registering the data. The cleanup pipeline operates after the pipeline has been finished. In the following example, the data created for Solr inputs is deleted.

pipeline:
  - name: create data
    command: create_solr_data.rb -i input.data -o output/solr-data.xml
  - name: register data
    command: $SOLR_HOME/bin/post -host $SOLR_HOST -port $SOLR_PORT -c car-instance -o output/solr-data.xml

cleanup:
  - name: cleanup xml data
    command: rm -f output/solr-data.xml

Reuse of Already-Defined Stages

There have been requests to reuse stages already defined in Walter configurations. Mechanisms to import files that include stage definitions (stage definitions files) are now supported.

An example of a stage definitions file is shown below. The stage definitions file describes the namespace and stages. The namespace was added so that names are not collision even when multiple namespace definitions files are called.

The def blocks are added in the stages block, and the stage defined. In the following example, the whomai stage is defined.

namespace: mystages

stages:
  - def:
    name: whoami
    command: whoami

The stage definition file described above is saved using the name stages.yml. The require block imports the stage definitions file from the stage definitions file. The defined stage is called using call. The called stage is described using the namespace::stage name format.

require:
  - stages.yml

pipeline:
  - call: mystages::whomai

Reuse of Stage Results

The results of previous stages in the pipeline can be acquired using three special variables (__OUT, __ERR, __RESULT).

__OUT: Stores the result (“true” or “false”) output to standard outputs.
__ERR: Stores the results output to the standard error outputs.
__RESULT: Stores the command return value.

The variables are used as maps. Assigning a stage name as the variable key acquires the stage results.

For example, in the following example, the second stage uses the output results from the first stage. Specifically, the input markdown document (markdown.md) is checked during the first stage with an OSS proofreading tool RedPen, and then in the second stage, whether or not an error was output to the standard error outputs of the first stage is checked. If an error was not output during the second stage, a PDF is created.

pipeline:
  - name: Check document with RedPen
    command: redpen -c redpen-conf.md document.md -f markdown
  - name: Convert markdown to PDF
    command: markdown-pdf -o document.pdf document.md
only_if: test -z "`echo __ERR["Check document with RedPen"] | grep ERROR`"

Misc Improvements

  • Slack communications are now supported
  • Communications with the local HipChat (proprietary) server are now supported
  • Communications with Slack are now colored

Future Work

Ongoing development of Walter is scheduled for the future. In particular, this winter, Walter is scheduled to support server functions. We also want to enable the stage start conditions to be specifiable. Currently, when the previous stage finished, the next stage ends up starting immediately, but specifying the start conditions will enable settings such as wait until the necessary port is ready to be set.

Serverspec CI environment with Walter + Wercker + DigitalOcean

This article introduces an application of Walter, a deployment pipeline, to software tests.
Specifically, how Walter is applied to the integration tests of my personal products, Serverspec and Specinfra are covered.

TL;DR

  • Application of Walter to the integration tests of Serverspec and Specinfra are described.
  • Combination of Walter and Wercker is nice. Especially parallel executions by Walter in Wercker is useful.
    • Walter parallelly executes tests on CoreOS, CentOS 6.5, CentOS 7.0, Ubuntu 14.04 and FreeBSD 10.1 with Docker

Walter + Wercker

As a dogfooding trial, I apply Walter to the Wercker CI for the integration tests of Serverspec and Specinfra.
The following shows the wercker.yml file for integration testing Serverspec.

box: mizzy/serverspec-base@0.0.6
build:
  steps:
     - script:
       name: Run walter
       code: ./$WORKING_DIR/walter -c ./$WORKING_DIR/pipeline.yml
  after-steps:
     - wantedly/pretty-slack-notify:
     webhook_url: $SLACK_WEBHOOK_URL

The below shows the pipeline.yml called from wercker.yml.

- name: Make $HOME/.ssh directory
  type: command
  command: mkdir -p $HOME/.ssh
  only_if: test "$WERCKER" = "true"
- name: Put SSH publick key
  type: command
  command: echo "$DIGITALOCEAN_SSH_KEY_PUBLIC" > $HOME/.ssh/id_rsa.pub
  only_if: test "$WERCKER" = "true"

We can run the tests in my laptop without pushing the changes to the GitHub repository. It is handy 🙂
When the processes are different between local and Wercker, we can use the only_if feature which allows us to change the behaviors by the environments.

Of course, we can execute the tests with Wercker v2 locally. Unfortunately Wercker v2 is a bit hassle, since it needs Docker nevertheless the integration tests of Serverspec and Specinfra do not need Docker.. Therefore I feel Walter is handy since Walter does not need any VM systems.

Paralell Execution

The tests are run in each VM which is created for the OS on DigitalOcean. The tests are executed in parallel with the following Walter configuration.

- name: Parallel builds each OSes
  parallel:
- name: Build CoreOS
  type: command
  directory: $WORKING_DIR
  command: vagrant up coreos --provider=digital_ocean && ./apply-itamae-and-serverspec-to-docker.sh
  only_if: test "$WERCKER_GIT_REPOSITORY" != "serverspec"
- name: Build CentOS 6.5
  type: command
  command: vagrant up centos65 --provider=digital_ocean && bundle exec itamae ssh --host centos65 --vagrant recipe.rb 
     && DIGITALOCEAN=true rake spec:centos65
  directory: $WORKING_DIR
- name: Build CentOS 7.0
  type: command
  command: vagrant up centos70 --provider=digital_ocean && bundle exec itamae ssh --host centos70 --vagrant recipe.rb && 
     && DIGITALOCEAN=true rake spec:centos70
  directory: $WORKING_DIR
- name: Build Ubuntu 14.04
  type: command
  command: vagrant up ubuntu1404 --provider=digital_ocean && bundle exec itamae ssh --host ubuntu1404 --vagrant recipe.rb 
     && DIGITALOCEAN=true rake spec:ubuntu1404
  directory: $WORKING_DIR
 - name: Build FreeBSD 10.1
   type: command
   command: vagrant up freebsd --provider=digital_ocean && DIGITALOCEAN=true rake spec:freebsd
   directory: $WORKING_DIR

In the above settings, each Walter stage starts VMs (CoreOS, CentOS 6.5, CentOS 7.0, Ubuntu 14.04 and FreeBSD 10.1) in DigitalOcean in parallell,
and then executes the provisioning by Itamae and run tests with Serverspec.

When the test run in sequence, the tests are not finished within the limit (25 minitutes). By contrast, Walter with the above settings finish the all tests within the limit.

We can see the Wercker log here. The following is the image of the log. As wee see, it is a bit difficult to understand the results.
Specifically when some tests are failed in a environment (OS), understanding that the failed tests were run in which OS is not intuitive.

Wercker log

I overcome the problem by the Slack notification feature provided by Walter.

Slack Notification

Walter provide messenger notification. The results of the stages are sent to a Slack or HipChat channel.

Wercker results

When tests of a stage failed, the stage is shown with red as follows

Failed stage

In the above case, we can see the tests of CoreOS are failed.
Clicking the bottom link and move the the build page, we can see the failure is from CoreOS tests.

Future

Currently I am satisfied with the combination of Wercker and Walter. As a next step I will use Walter with Jenkins or other CI systems and add the features to Walter if needed. In addition, I have a plan to provide the Walter server with which users can build CI system without Jenkins or other CI services.