Jenkins K8s Jobs¶

TL;DR — Scope: Jenkins pipelines that deploy to Kubernetes (backend microservices, cron jobs). Merging to dev, rc, or master triggers a Jenkins pipeline that builds a Docker image, pushes it to ECR, and runs kubectl apply against the target EKS cluster. Two migration paths exist: Jenkinsfile-K8S (kops server switch) and Jenkinsfile-K8S-MIG (migrations run directly in Jenkins).

Scope: K8s-bound jobs only

This page covers Jenkins jobs that target Kubernetes (Jenkinsfile-K8S, Jenkinsfile-K8S-MIG, Jenkinsfile-K8S-SST). Frontend pipelines (Jenkinsfile-UI-SST, Jenkinsfile-UI, etc.) deploy to S3 + CloudFront and are documented in jenkins-ui-jobs.md.

Overview¶

Continuous Delivery to Kubernetes is orchestrated by Jenkins (self-hosted — URL and access details in databases-and-services.md). Jenkinsfile templates are kept centrally in the ops-manifest repo and consumed by each service via the Remote Jenkinsfile Plugin (RJPP) — see ops-repos.md for the full Jenkinsfile catalog.

Branching strategy¶

Branch	Target environment
`dev`	dev
`rc`	rc
`master`	sandbox
(prod)	Manual Jenkins job — detects commit diff between sandbox and prod, operator picks services

Merge strategy between environments (varies by team)¶

Some teams: full merge from dev → rc → master via PR with merge commit.
Other teams: selective cherry-pick of commits between environments.

Image build per environment

A new Docker image is built per branch/environment rather than promoting a single image across environments. The image that runs in RC is not bit-for-bit identical to the image that reaches sandbox or prod.

Microservices pipeline — detailed flow¶

The pipeline runs on a kops server (one per environment: dev, rc, sandbox, prod) via SSH. Every deploy follows this sequence on the kops server (confirmed from build logs):

1. git pull ops-scripts                         # fresh pull every deploy
2. [if migration enabled] build ephemeral migration Docker → run migrations → discard
3. docker build service image (from repo Dockerfile)
4. docker push :latest + :{commit-hash} to ECR
5. kubectl apply yamls/custom/{env}/ (or yamls/ flat) — see deployment-yamls.md
6. kubectl apply configmap
7. kubectl rollout status deployment/{service}  # blocks until done

Migration step (conditional)¶

Only services that have migrations enabled run this step. Two mechanisms exist — see the next section.

Docker build¶

Builds the service image using the Dockerfile from the repo. Requires SECURITIZE_READ_NPM_TOKEN as a build arg — see secrets.md.

ECR push¶

Two tags pushed per build: - :latest - :{full-commit-hash} (e.g. 9ae3bb998f7d73f4ffb0a1faef4f10e4d4cdbd47)

ECR repo format: {aws-account}.dkr.ecr.us-east-2.amazonaws.com/securitize/{service-name} — account depends on the target environment (see aws-infrastructure.md).

Manual rollback

The commit-hash tag means you can identify exactly which commit is running in each environment and roll back by re-tagging an older :{hash} as :latest (ask DevOps). See rollback-and-incidents.md.

Apply yamls + configmap¶

kubectl apply runs against the yamls in the service's ops-scripts/k8s/{service}/ folder. The script deploy_service_jenkins.sh (on the kops server) auto-detects flat vs custom/{env}/ layouts — details in deployment-yamls.md.

Rollout wait¶

Active wait via kubectl rollout status deployment/{service} — blocks the pipeline until all replicas are updated.

Two migration mechanisms¶

Services with DB migrations use one of two Jenkinsfile templates:

Option A — `Jenkinsfile-K8S` (kops server migration switch)¶

A switch in migration.sh on each kops server decides whether a given service runs migrations.
If enabled, an ephemeral migration Docker image is built from a per-service Dockerfile stored on the kops server (not in git).
Runs npm ci + npm run build + npm run migrate with DB creds passed via --build-arg.
Container is discarded after migration.

When this is used: when a service was created without persistence and later the team adds a database — DevOps sets up the migration Dockerfile + switch on each kops server manually.

Node version upgrades affect this Dockerfile

The migration Dockerfile has its own FROM node:<version> and lives only on the kops servers — it is not updated when you bump Node in the repo. If the service uses this mechanism and you upgrade Node, coordinate with DevOps to update the migration Dockerfile on all 4 kops servers (dev, rc, sandbox, prod). See bitbucket-ci.md — Node version for the full list of files to update.

Example migration Dockerfile:

FROM node:20.11.1 as builder
COPY . .
ARG TYPEORM_TYPE
ARG TYPEORM_HOST
ARG TYPEORM_PORT
ARG TYPEORM_USERNAME
ARG TYPEORM_PASSWORD
ARG TYPEORM_DATABASE
ENV TYPEORM_TYPE=${TYPEORM_TYPE} \
    TYPEORM_HOST=${TYPEORM_HOST} \
    TYPEORM_PORT=${TYPEORM_PORT} \
    TYPEORM_USERNAME=${TYPEORM_USERNAME} \
    TYPEORM_PASSWORD=${TYPEORM_PASSWORD} \
    TYPEORM_DATABASE=${TYPEORM_DATABASE}
RUN npm ci; npm run build
RUN npm run migrate

Option B — `Jenkinsfile-K8S-MIG` (Jenkins-based migrations)¶

Runs npm ci + npm run build + npm run migrate directly in Jenkins via nodejs().
DB credentials injected from AWS Parameter Store: /secrets/{env}/ops + /secrets/{env}/{team}/{service}.
After migration, calls deploy-service-{env} (same as K8S).

When this is used: when the dev knows from the start that the service will have persistence. Service is scaffolded with the K8S-MIG template — no DevOps setup needed on kops servers.

Node version upgrades: Jenkins must have the version installed

Migrations run via nodejs(), which uses Node versions installed in Jenkins Global Tool Configuration. If you upgrade Node and the new version isn't installed in Jenkins, migrations will use the team default or fail. Coordinate with DevOps to install the new version — same constraint as UI pipelines (see jenkins-ui-jobs.md — Build flow, step 3).

Build commands vary by team¶

Team	Commands
FR	`npm ci` → `npm run migrate`
ISR, TA	`npm ci` → `npm run build` → `npm run migrate`
Default	`npm install` → `npm run build` → `npm run migrate`

Potential redundancy warning¶

Avoid enabling both mechanisms

If DevOps enables the kops server switch for a K8S-MIG service, migrations run twice — first in Jenkins (effective), then again in the ephemeral container (no-op). Wastes time. Only one mechanism should be active per service.

Operational details (kops servers)¶

kops servers are not in git

Migration Dockerfiles live on the kops server, edited manually by DevOps (with manual backups before changes).
One kops server per environment — any change to migration Dockerfiles or migration.sh must be replicated across dev, rc, sandbox, prod manually.
deploy_service_jenkins.sh lives at /var/local/buildscripts/deploy_service_jenkins.sh on each kops server.

The deploy-service-{env} Jenkins job orchestrates everything by SSHing into the corresponding kops server.

Frontend pipelines¶

Not covered in this page. Frontend Jenkins pipelines (Jenkinsfile-UI-SST, Jenkinsfile-UI, etc.) deploy to S3 + CloudFront rather than Kubernetes — see jenkins-ui-jobs.md for the detailed flow, or frontend-architecture.md for the high-level paths overview.

Production deploy (K8s services)¶

A Jenkins job shows K8s services pending deployment to prod (detects commit differences between sandbox and prod).
The process is manual: a TL or dev selects services and deploys them.
No fixed release cadence — deployments happen on demand (no release train or schedule).
Deploy state tracking lives in the ops-deployments repo — see ops-repos.md.

Frontends also use a prod deploy queue

Frontend prod deploys use the same ops-deployments queue infrastructure. Details are documented alongside the frontend pipeline flow in frontend-architecture.md.

Tags¶

cd #jenkins #deploy #kubernetes #ecr #migrations #kops #sst #cdk¶