Hello Jenkins Pipelines

Building continuous delivery pipelines and similarly complex task orchestrations in Jenkins using traditional job types and plugins can prove to be an awkward and hacky process. Here at Under Armour we have tried many things to achieve a simple Jenkins setup, code review workflows and automation practices. Until now, it has mostly been fraught with difficulty from Jenkins, leaving us with many hacks or workarounds, and nothing has stood out as a stellar solution.

Enter Jenkins Pipelines, formerly called Jenkins Workflow. It’s a Jenkins plugin that allows you to program your build pipeline via a Groovy-based DSL. If you have ever used continuous integration services like Travis CI or Circle CI, Pipelines will seem very similar, though maybe less structured and a lot more flexible (even scarily so). For example, a developer could fill up your Jenkins slaves with running Docker containers that have exited successfully but were never removed.

node('docker') {
  scm checkout
  sh 'docker run postgres:94' # <== note missing `--rm` so the container will hang around
  # ...
}

Multi-Branch Workflow projects group up all your jobs and branch runs into a folder which in turns reduces your job count bloat.

Multi-Branch Workflow projects group up all your jobs and branch runs into a folder which in turns reduces your job count bloat.

Jenkins Pipelines also make job configuration super simple with Multi-Branch Pipeline. This job type will automatically create a folder in Jenkins, index all of the branches in your repository and generate a job for each one that contains a build script. When that job runs (after detecting SCM changes), it will load the repository branch’s Jenkinsfile and perform the instructions inside. This means that developers can update and control how their project is handled by Jenkins on a per commit basis, even down to the granularity of having different pipelines per branch, if they so choose.

“But wait, now I can’t provide reusable functionality to all my Jenkins jobs because each repository has custom code for how to interact with Jenkins?! How is that better?” Don’t worry: you still can, it’s just a bit different, and it’s opt-in.

Each Multi Branch Workflow project has its own view which makes it really easy to see the build status of each branch in your project.

Each Multi Branch Workflow project has its own view which makes it really easy to see the build status of each branch in your project.

Say we have two developers, John and Jane. John doesn’t want to have to think too much about his build infrastructure or even how his project is built and tested, all he needs is for Jenkins to call python setup.py install. Jane, on the other hand, loves the cutting edge. She’s always looking to try new tools out and loves to experiment on her project. Jane is the kind of developer who needs to understand all of the small details and doesn’t use existing library code unless she deems it worthy.

With the Jenkins Pipeline file, both of these developers can be satisfied because this setup allows each developer to write their own build script for the custom needs of their project and from their own prerogative as a project owner.

Now, say we see a pattern of builds posting messages to a Slack channel, but we don’t want each developer to worry about creating webhooks/credentials or the security of it all. Jenkins Pipelines has a solution for this, it’s called the Pipeline Global Library.

(root)
 +- src                     # groovy source files
 |   +- org
 |       +- foo
 |           +- Bar.groovy  # for org.foo.Bar class
 +- vars
     +- foo.groovy          # implements 'foo' variable function
     +- foo.txt             # help for 'foo' variable function

Jenkins Global Library file & directory structure

Every Jenkins installation with this plugin installed now has a git repository to which you can push code, giving you the ability to provide first-class pipeline steps that look and act exactly like other pipeline steps, and you can even provide documentation to boot!

It also means that for someone like John, you can build out whole templates for him to use—which, let’s be honest, were probably derived from something awesome you saw Jane do in her Jenkinsfile, but have now been vetted and made for use by the whole team.

# vars/statsd.groovy

import groovy.transform.Field;

@Field def namespace = "${env.STATSD_METRIC_PREFIX}.pipeline."
@Field def delim = ":"
@Field def gaugeToken = "|g"
@Field def counterToken = "|c"
@Field def timingToken = "|ms"

def emitMetric(metric) {
  node('swarm') {
    sh 'echo "' + metric + '" > /dev/udp/localhost/8125"
  }
}

def timing(name, value) {
  this.emitMetric(this.namespace + name + this.delim + value + this.timingToken)
}

def gauge(name, value) {
  this.emitMetric(this.namespace + name + this.delim + value + this.gaugeToken)
}

def count(name, value) {
  this.emitMetric(this.namespace + name + this.delim + value + this.counterToken)
}
# vars/statsd.txt

<p>Produces a metric into the statsd cluster</p>
<h3>Methods:</h3>
<ul>
    <li>timing</li>
    <li>gauge</li>
    <li>count</li>
</ul>
<h3>Args:</h3>
<ul>
    <li>metric name: The name of the metric you'd like to publish.</li>
    <li>value: The numerical value to pass through to the metric.</li>
</ul>
<h3>Example:</h3>
<code>
  statsd.count("my.cool.metric", 10)
</code>

We have started to build out our internal Groovy workflow CI library with many common tasks for our developers to run, including posting Slack messages, defining whole build flows for certain project types, posting status events into Kafka for consumption, as well as creating run-book like scripts that can be configured to run manually for any number of triggers. This has turned our Jenkins setup into an easy-to-use, easy-to-introspect component architecture, similar to AWS Lambda.

As we upgrade each project to use this system, our developers are learning more and build and test cycle times are getting shorter as they can easily make improvements to their pipeline with a just a few Git commits. We’re also watching for useful patterns (the fruitful results of Jane’s tinkering) that we can turn into easy-to-configure pipeline steps for everyone with the Pipeline Global Library.

We highly recommend you take the time to dig into the suite of Jenkins Pipelines plugins. The documentation is slowly growing as they continue to flesh it out, fix bugs and add requested features. As of the 1.5 release, documentation has been separated out into GitHub repositories for each of the Pipeline’s components, to give better clarity on what each plugin does for the ecosystem.

automationcijenkins