Jenkins Pipelines Without The Headaches (Mostly)
How we keep builds moving, even when jenkins has other plans.
Why We Still Keep Jenkins Around
We’ve tried a lot of CI/CD tools over the years, and somehow jenkins keeps surviving the annual “should we replace this?” meeting. Not because it’s trendy, but because it’s flexible, runs almost anywhere, and has a plugin for the weird thing our legacy app needs at 2 a.m.
Jenkins shines when we need to integrate oddball build steps, talk to crusty on-prem systems, or run inside networks that don’t want to meet the internet. It’s also easy to start small: a single controller and a couple of agents can carry a surprising amount of workload. The flip side is that the same flexibility makes it easy to create a fragile snowflake if we don’t put a bit of structure around it.
The core lesson we’ve learned: jenkins works best when we treat it like any other production system. That means versioning configuration where possible, keeping plugins on a diet, isolating workloads onto agents, and having a plan for backups and upgrades. If we do that, it’s a steady workhorse. If we don’t, it becomes a haunted house where jobs randomly fail because somebody updated a plugin “just to see what happens.”
If you want the official starting point (and a reminder of how big the ecosystem is), the upstream docs are still the canonical reference: Jenkins Documentation. We’ll build on that with the stuff we actually do day-to-day to keep things boring—in the best way.
Controllers, Agents, And Not Setting The Controller On Fire
A jenkins setup is basically two roles: the controller (schedules jobs, stores config, provides UI/API) and agents (do the actual work). The fastest way to make jenkins miserable is to run builds on the controller. It’s tempting early on—everything “just works”—until one runaway build eats disk, RAM, file descriptors, and your afternoon.
Our rule: controllers coordinate; agents build. We keep the controller small, stable, and as stateless as jenkins allows. We also separate agent pools by workload. For example: one pool for Docker builds, another for Android builds, another for “this needs a Windows license and nobody likes it.”
A few practical guardrails that save real pain:
– Dedicated workspace storage on agents, with cleanup policies. Workspaces grow like sourdough starter.
– Labels that mean something (“linux-docker”, “windows-dotnet”), not “agent1”.
– Concurrency limits per agent type so we don’t melt shared resources (artifact repos, databases, test environments).
– Pinned toolchains (JDK, Node, Maven) via agents/images rather than per-job snowflakes.
If you’re using Kubernetes for ephemeral agents, the Kubernetes plugin can work well, but keep the pod templates versioned and reviewed. For a broader “what is CI/CD supposed to be doing anyway?” refresher, Atlassian’s overview is a clean read: CI/CD explained.
The vibe we want: predictable compute for builds, predictable controller behaviour, and no “oops, the controller disk is full because a job wrote a 30 GB log”.
Jenkinsfile: The Only Snowflake We Allow
If there’s one hill we’ll cheerfully defend: pipelines belong in source control. A job configured by clicking around in the UI is a job you can’t reliably reproduce, code review, or roll back. A Jenkinsfile is at least honest about what it’s doing.
Here’s a declarative pipeline we’d happily ship as a baseline. It builds, tests, archives artifacts, and keeps logs readable. It also avoids cleverness—because cleverness is how we end up debugging Groovy at midnight.
pipeline {
agent { label 'linux-docker' }
options {
timestamps()
ansiColor('xterm')
disableConcurrentBuilds()
buildDiscarder(logRotator(numToKeepStr: '30', artifactNumToKeepStr: '10'))
}
environment {
APP_NAME = 'example-service'
}
stages {
stage('Checkout') {
steps {
checkout scm
}
}
stage('Build') {
steps {
sh 'make build'
}
}
stage('Test') {
steps {
sh 'make test'
}
post {
always {
junit 'test-results/**/*.xml'
}
}
}
stage('Package') {
steps {
sh 'make package'
archiveArtifacts artifacts: 'dist/**', fingerprint: true
}
}
}
post {
failure {
echo "Build failed. Time to pretend we didn't touch anything."
}
cleanup {
cleanWs(deleteDirs: true, disableDeferredWipeout: true)
}
}
}
We keep the “shape” of pipelines consistent across repos, even if the guts differ. Same stage names, same options, same artifact handling. That consistency is underrated: it makes onboarding easier and makes the UI readable.
When we need shared logic, we prefer shared libraries but keep them small. If you’re new to Pipeline syntax, the upstream reference is the one we all end up on eventually: Pipeline Syntax.
Credentials And Secrets: Let’s Not Leak The Crown Jewels
Jenkins can handle credentials decently, but it won’t stop us from doing something daft. Our goal is simple: keep secrets out of repos, out of console logs, and out of build artifacts. Then rotate anything that might’ve been exposed, because eventually something will.
A few policies we use:
– Prefer short-lived tokens (OIDC where possible) over long-lived static secrets.
– Store credentials in jenkins’ credential store, but back it with a real secret manager when we can.
– Never echo $TOKEN. Not even “just to test.” Especially not to test.
– Masking helps, but don’t rely on it—some tools reformat output and break masking.
Here’s what “reasonable” looks like in a pipeline: use withCredentials, scope it tightly, and don’t let secrets wander into the environment longer than needed.
stage('Publish') {
steps {
withCredentials([string(credentialsId: 'nexus-token', variable: 'NEXUS_TOKEN')]) {
sh '''
set -euo pipefail
curl -fsS -H "Authorization: Bearer $NEXUS_TOKEN" \
-T dist/app.tar.gz \
https://nexus.example.com/repository/releases/app.tar.gz
'''
}
}
}
If we’re building containers, we also avoid baking secrets into images. Use build args cautiously, prefer runtime injection, and use your platform’s secret mechanisms (Kubernetes Secrets, cloud secret stores, etc.).
For teams that need a north star on secret management practices, HashiCorp’s Vault docs are a solid reference even if you don’t use Vault itself: Vault documentation.
Bottom line: in jenkins, secrets should be boring—scoped, audited, rotated, and never printed like a trophy.
Plugins, Upgrades, And Other Ways To Ruin A Weekend
Plugins are jenkins’ superpower and its favourite foot-gun. The plugin ecosystem is huge, and that’s great until we realize our instance is a Jenga tower of unpinned dependencies.
We manage plugins like we manage production dependencies:
– Install the minimum set we need.
– Review plugin health (maintenance, release cadence, known issues).
– Keep a list of “tier 1” plugins we trust and “probation” plugins we limit.
– Test upgrades in a staging controller before touching production.
When it’s time to upgrade, we focus on three things: Jenkins core, plugins, and Java. Changing all three at once is a great way to create a mystery. We do staged upgrades: core first (or Java first if required), then plugins in batches, validating critical pipelines as we go.
Also: backups. Not “we should do backups,” but “we restored from backup last month to prove it works.” For jenkins, the gold is typically:
– $JENKINS_HOME (jobs, config, credentials, build history)
– Configuration-as-code files (if used)
– Seed job definitions and shared libraries (in Git)
The official security and maintenance guidance is worth reading at least once a year, even if only to feel mildly guilty: Jenkins Security.
If we do plugins and upgrades with care, jenkins is stable. If we treat plugin updates like phone app updates (“just tap update all”), we’ll be writing a postmortem with the phrase “unexpectedly incompatible.”
Speeding Things Up: Caching, Parallelism, And Smarter Builds
Most pipeline pain isn’t philosophical. It’s “why does this build take 27 minutes?” Jenkins won’t magically make builds fast, but it can amplify whatever habits we already have—good or bad.
We usually get big wins from:
– Dependency caching (Maven/Gradle/npm/pip) on agents or shared cache volumes.
– Incremental builds where the build tool supports it.
– Parallel test execution split by packages, test suites, or historical timing.
– Early failure: lint and quick unit tests first, integration tests later.
In jenkins pipelines, parallelism is straightforward, but we keep it readable and we limit fan-out so we don’t DDoS our own infrastructure.
We also like “change-aware” behaviour: if docs-only changes happen, skip expensive stages. If only a single service changed in a monorepo, build only that service. This is less about jenkins and more about being considerate to everyone sharing the CI system.
One underappreciated trick: keep logs clean. Excessive logging slows builds, bloats storage, and makes debugging harder. We prefer structured test reports, targeted debug output on failures, and a sane log retention policy.
If you’re pushing artifacts, make sure your artifact repository isn’t the bottleneck. Sonatype’s Nexus docs (or JFrog Artifactory equivalents) are good references for tuning and best practices; here’s a general starting point for repository concepts: Nexus Repository Manager.
The goal isn’t “fast at all costs.” It’s “fast enough that people don’t bypass CI because they’re impatient.”
Making Jenkins Auditable: Configuration As Code And Job Seeds
Even with Jenkinsfiles, the controller still has configuration: credentials (metadata), agent definitions, security settings, shared library config, and so on. If we rely on hand-configured UI state, we’ll eventually drift into “nobody knows why it’s set like that, but don’t touch it.”
This is where Jenkins Configuration as Code (JCasC) can help. We don’t need to codify every last knob to get value; codifying security realm, authorization, agents, and global libraries gets us most of the way. The big win is repeatability: if we need to rebuild a controller, we can.
A minimal (illustrative) JCasC snippet might look like this:
jenkins:
systemMessage: "Built by humans, maintained by caffeine."
numExecutors: 0
security:
globalJobDslSecurityConfiguration:
useScriptSecurity: true
unclassified:
location:
url: "https://jenkins.example.com/"
tool:
git:
installations:
- name: "Default"
home: "/usr/bin/git"
We often pair JCasC with a “seed job” approach (Job DSL or similar) to generate multibranch pipelines consistently. That way, new repos get the same defaults: log rotation, timeouts, webhook triggers, and sane naming. Less artisanal clicking, more predictable outcomes.
This is also the point where we draw a line: if someone wants to create a one-off freestyle job “just for now,” we ask them to write it as code. “Just for now” is the most permanent timeline in DevOps.
Done well, configuration-as-code makes jenkins feel less like a pet server and more like a reproducible system we can reason about.
