Ops Repositories¶

TL;DR — The main ops-* repositories in the securitize_dev Bitbucket workspace manage the entire operational surface: K8s configs, Jenkins pipelines, Terraform IaC, ArgoCD GitOps, deploy state tracking. All are private and owned by the DevOps team (teams.md).

Overview¶

The ops-* repos cover every layer from "AWS Terraform state" up to "deploy queue orchestration." They connect in a clear dependency chain shown below.

Relationship map¶

ops-terraform-modules  →  ops-infra          (reusable modules → real AWS infra)
ops-infra              →  ops-k8s-infra       (creates EKS clusters → GitOps inside clusters)
ops-manifest           →  all services        (Jenkins templates via RJPP)
ops-deployments        →  ops-scripts         (deployment state → orchestrates deploys)
ops-scripts            →  everything          (central operations hub)

ops-scripts¶

URL: https://bitbucket.org/securitize_dev/ops-scripts
Size: ~227 MB (largest OPS repo)
Status: actively maintained

The central operations hub. Manages the full lifecycle of ~231 microservices.

K8s / microservices only — not for frontends

ops-scripts manages only services deployed to Kubernetes (NestJS/Express microservices, cron jobs, etc.). Frontend deployments do NOT live here — frontends deploy to S3 + CloudFront via Jenkins using Jenkinsfile-UI-SST (modern, SST/CDK from the repo's infra/) or Jenkinsfile-UI (legacy). See frontend-architecture.md.

Structure¶

ops-scripts/
├── k8s/                  # K8s configs per service (~231 services)
│   ├── {service-name}/
│   │   ├── configmaps/   # Per-env ConfigMap env files (see "Per-service ConfigMaps" below)
│   │   ├── yamls/        # deployment.yaml, service.yaml
│   │   │                 # (flat, or custom/{env}/ — see ../ci-cd/deployment-yamls.md)
│   │   └── env/          # Optional — legacy .env files, often empty
│   ├── apac/             # APAC-region services (legacy layout — flat deployment.yaml + env/)
│   ├── global/           # Global ConfigMaps/Secrets shared across services
│   │   └── {global-name}/yamls/{env}/configmap.yml
│   └── template/         # Template for new services
├── scripts/              # Operational scripts (~32 scripts)
│   ├── deploy_service.sh
│   ├── build_service.sh
│   └── create_service.sh
├── RBAC/                 # RBAC scripts per environment (dev, prod, apac, sandbox, rc)
├── k8s-addons/           # Cluster add-ons (Helm + Terraform)
│   ├── alb-controller/
│   ├── datadog-operator/
│   ├── prometheus-operator/
│   └── nats/, stan/, ...
├── helm/                 # Helm charts (Kong, Prometheus, Datadog, Logz.io, …)
├── aws/                  # CloudFormation, Lambda, Cognito, ACM configs
├── jenkins/              # Jenkins pipeline configs
└── kops/                 # Cluster creation scripts

Technologies¶

Bash, Kubernetes YAML, Helm, Terraform (k8s-addons), AWS, Python.

What it does¶

Lifecycle of ~231 microservices: creation, build, deploy.
Cluster RBAC per environment.
Cluster add-on management (Helm + Terraform).
AWS integrations (EKS, CodeDeploy, Lambda, CloudFormation).
Integrations with Datadog, Prometheus, Logz.io, Kong, NATS.

Per-service ConfigMaps¶

Most services (~226 of 231) follow the modern layout. Per-env ConfigMap values live in k8s/{service}/configmaps/ as plain .env files, one per target environment. All filenames in the table below are relative to k8s/{service}/configmaps/:

File	Target environment
`properties_dev.env`	`dev`
`properties_rc.env`	`rc`
`properties_sandbox.env`	`sandbox`
`properties_prod.env`	`prod`
`properties_apac.env`	`apac` (only services that run in APAC)
`properties_qa.env`	`qa` (limited — not every service has it)
`properties.env`	shared/base — role is service-specific; not covered by the standard Jenkins apply job
`properties_demo.env`	demo — role is service-specific; not covered by the standard Jenkins apply job

Applied via the OPS-k8s-configmap-update Jenkins job, which takes service_name and environment parameters, reads k8s/{service_name}/configmaps/properties_{environment}.env, applies the ConfigMap, and restarts the deployment.

Production apply is Tech Lead only

properties_prod.env changes are not blocked at the repo level — anyone can edit and merge the PR. Applying the change to the prod cluster requires a Tech Lead: only TLs have permission to view and trigger the prod variant of the Jenkins apply job. A non-TL cannot run the prod apply directly; coordinate with your team's TL. This is distinct from ingress / DNS changes in ops-k8s-infra, which for prod still require a DevOps ticket (see the prod ingress is not configured via this repo note further down).

Layout exceptions (do not follow the modern layout):

Exception	Layout	Notes
`k8s/apac/{service}/` (~24 services)	Flat — `deployment.yaml` + `service.yaml` + `env/` + optional `properties.env` at top level	APAC-region legacy. No `configmaps/` folder.
`k8s/assets-proxy/`	Flat — `env/` + `yamls/` only	Legacy.
`k8s/hello-world-2/`	Helm chart — `base/` + `{env}/values.yaml`	Helm-based.
`k8s/global/`, `k8s/ta-global/`	See "Global ConfigMaps / Secrets" below	Shared artifacts, not per-service.

Tooling that targets the modern layout (for example, the configure-service-configmap workflow) does not handle these exceptions — they require manual operation.

Global ConfigMaps / Secrets (`k8s/global/`)¶

A sibling to per-service configs, k8s/global/ holds ConfigMaps and Secrets shared across multiple services (e.g. ta-global, redis-global, keycloak-global). Each global folder follows this layout:

k8s/global/{global-name}/
├── yamls/
│   ├── dev/configmap.yml
│   ├── rc/configmap.yml
│   ├── sandbox/configmap.yml
│   ├── prod/configmap.yml
│   └── apac/configmap.yml    # only where applicable
├── env/                       # .env / .prodenv
└── properties/                # properties.env

Applied via the OPS-global-yamls Jenkins job:

Parameter	Description
`environment`	Target cluster: `cluster-eks-dev`, `cluster-eks-rc`, `cluster-eks-sandbox`, `cluster-eks-prod`.
`global_folder_name`	Subfolder name under `k8s/global/` (e.g. `ta-global`).

The job switches kubectl context to the chosen cluster and runs kubectl apply on every .yml under k8s/global/{global_folder_name}/yamls/{env}/.

Only ConfigMap and Secret are applied

The job greps for kind: ConfigMap or kind: Secret in each YAML file and applies only those. Other kinds (Deployments, Services, Jobs, etc.) placed in this folder are silently ignored.

Per-service secret references¶

Secret values are not stored in this repo — they live as K8s Secret objects in the cluster. What lives in this repo are the references: each service's deployment.yaml declares which secrets it consumes via secretRef (for envFrom) or env.valueFrom.secretKeyRef (for individual keys).

To see which secrets a given service consumes, open that service's deployment.yaml under ops-scripts/k8s/<service>/yamls/ and look for secretRef / secretKeyRef entries. Those references can point to:

Global secrets shared across many services — documented in secrets.md.
Per-service secrets — a dedicated K8s Secret object for the service (its values are not in git; creating it is a manual step when scaffolding a new service, see service-creation.md).

YAML patterns used per service (flat vs custom/{env}/) are documented in deployment-yamls.md.

ops-deployments¶

URL: https://bitbucket.org/securitize_dev/ops-deployments
Size: ~25 MB
Status: actively maintained

Deployment state tracking and queue management for all services.

Structure¶

ops-deployments/
├── master/               # Main deployment state
│   ├── AO/, ATS/, BC/, CA/, CORE/
│   ├── FR/, INVT/, ISR/, JP/, OPS/
│   ├── SFS/, TA/
└── pipeline-OPS-prod-deploy-queue   # Groovy Jenkins pipeline
    pipeline-OPS-apac-deploy-queue   # Groovy Jenkins pipeline

Technologies¶

JSON, Groovy (Jenkins DSL).

What it does¶

~269 JSON files (one per service), organized by team.
Each JSON tracks which commit is deployed in sandbox, prod, and apac.
Records QA results, timestamps, and state transitions (synced / queued).
2 Jenkins pipelines managing the deploy queue (prod and apac).
Single source of truth for deployment state across the whole platform.

ops-manifest¶

URL: https://bitbucket.org/securitize_dev/ops-manifest
Size: ~3.9 MB
Status: actively maintained

Jenkinsfile template library used by all services via the Remote Jenkinsfile Plugin (RJPP). See jenkins-k8s-jobs.md for the full CD flow that uses these templates.

Structure¶

ops-manifest/
├── jenkins/Jenkinsfile/
│   ├── Jenkinsfile-K8S          # Standard backend microservices
│   ├── Jenkinsfile-K8S-MIG      # Backend + DB migrations in Jenkins
│   ├── Jenkinsfile-K8S-SST      # Backend + AWS infra (SQS, Lambda, DynamoDB)
│   ├── Jenkinsfile-UI           # Legacy frontends
│   ├── Jenkinsfile-UI-SST       # Modern frontends (S3/CloudFront)
│   ├── Jenkinsfile-UI-JP        # APAC team frontends (pnpm)
│   ├── Jenkinsfile-hotfix3617   # One-off hotfix
│   └── Jenkinsfile-monorepo     # Docker-based monorepos
└── CLAUDE.md                    # Repo documentation

ops-k8s-infra¶

URL: https://bitbucket.org/securitize_dev/ops-k8s-infra
Size: ~3.5 MB
Status: actively maintained

Declarative Kubernetes manifests applied via GitOps (ArgoCD).

Structure (7 environments)¶

ops-k8s-infra/
├── prod/, apac/, dev/, qa/, rc/, sandbox/, backup/
    # Each environment contains:
    # rbac/, system/, custom-addons/, argo-config/, ingress(es)/

Technologies¶

Kubernetes YAML, ArgoCD, Helm.

What it does¶

RBAC (developer, team lead, QA roles) per environment.
System resources: namespaces, metrics-server, ALB service accounts, EBS CSI, load balancers.
Add-ons: Velero (backups to S3).
ArgoCD apps and configurations.
Ingress / routing rules per environment.
~130 YAML files total.

Ingress and DNS configuration¶

Backend services are exposed through one of three service exposure types. Each type has its own YAML file per environment.

Service types and browser URLs¶

Type	Browser URL pattern	Notes
`internal`	`<service>.internal.<env>.securitize.io`	Only accessible through the company VPN.
`public`	`<service>.<env>.securitize.io`	Publicly accessible, not fronted by a frontend.
`gateway` (BFF)	`id.<env>.securitize.io/gw/<service>`	Traffic routed through CloudFlare before reaching the cluster. The YAML `host` differs from the browser URL (the YAML uses `<service>.<env>.securitize.io`, same format as `public`).

Write target — the `current` file per env × type¶

When adding a new rule, write it to the current file for the target env × service_type. The table below is the canonical list of write targets:

Env	`internal`	`public`	`gateway`
`dev`	`dev/ingress/securitize-internal-alb-dev-extend.yaml`	`dev/ingress/alb.yaml`	`dev/ingress/behind-cf-alb.yaml`
`rc`	`rc/ingresses/securitize-internal-alb-rc-extend.yaml`	`rc/ingresses/alb.yaml`	`rc/ingresses/behind-cf-alb.yaml`
`sandbox`	`sandbox/ingresses/ta-internal.yaml`	`sandbox/ingresses/securitize-ingress.yaml`	`sandbox/ingresses/minimal-ingress.yaml`

Why there are multiple files per environment — `current` vs `at capacity`¶

The ALB ingress controller has a hard limit of ~50 rules per YAML file. When a file reaches that limit, a new one is created — the current file, where all new rules go. The previous files become at capacity: they continue to be applied by the cluster and serve their existing rules normally (they are not legacy, deprecated, or unused), but no new rules can be added to them — the ALB will reject the extra rules on apply.

Practical consequence: a service's existing rule could live in any file of its env × service_type set — current or at capacity. Always grep the full set when checking whether a rule exists; only the current file is a valid write target.

Read target — full file set per env × service_type¶

When checking whether a rule already exists (or removing one), grep the full set below for the relevant env × service_type. The single file marked (current) is the write target; the others are (at capacity) — read-only.

internal — multiple files per env:

Env	Files
`dev`	`dev/ingress/securitize-internal-alb-dev-extend.yaml` (current), `dev/ingress/securitize-internal-alb-dev.yaml` (at capacity), `dev/ingress/bc-internal.yaml` (at capacity)
`rc`	`rc/ingresses/securitize-internal-alb-rc-extend.yaml` (current), `rc/ingresses/securitize-internal-alb-rc.yaml` (at capacity), `rc/ingresses/bc-internal.yaml` (at capacity)
`sandbox`	`sandbox/ingresses/ta-internal.yaml` (current), `sandbox/ingresses/internal.yaml` (at capacity), `sandbox/ingresses/bc-internal.yaml` (at capacity)

public and gateway currently have a single (current) file per environment (no at capacity files yet) — see the "Write target" table above. The full read set for these is just that one file.

Never mark a rule as MISSING based on a single-file grep

Always include every file from the set above for the target env × service_type in the same grep command. A false MISSING leads to a duplicated rule on apply, which the ALB will then reject.

Folder and filename inconsistencies across environments

dev/ uses the folder ingress/ (singular); rc/ and sandbox/ use ingresses/ (plural).
File names differ across environments (e.g. ta-internal.yaml in sandbox vs. securitize-internal-alb-{env}-extend.yaml in dev/rc).
public and gateway use the same host format (<service>.<env>.securitize.io); what distinguishes them is which YAML file the rule lives in.

Why gateway browser URL and YAML host don't match — and how to search

The browser URL id.<env>.securitize.io/gw/<service> is handled by a CloudFront Lambda@Edge function that rewrites the request: it strips the /gw/<service> prefix and forwards to <service>.<env>.securitize.io. This rewrite happens before the request reaches the cluster, so the Kubernetes ingress only ever sees the rewritten host — there is no /gw/ path in any YAML.

Practical consequence: when looking up or adding a gateway service in ops-k8s-infra, search for <service-name> (e.g. bc-labs-gw), not for the browser path (/gw/bc-labs-gw). The rule will be a plain host: <service>.<env>.securitize.io entry in the behind-cf-alb.yaml / minimal-ingress.yaml file for the target environment.

Ingress rule format¶

Each rule appended to the target YAML follows this structure:

- host: <host>
  http:
    paths:
    - path: /*
      pathType: ImplementationSpecific
      backend:
        service:
          name: <service>
          port:
            number: <port>

Default port when unspecified: 8080.

Production ingress changes¶

prod ingress is not configured via this repo

ops-k8s-infra covers dev, rc, and sandbox for ingress changes. Production DNS / ingress changes require a DevOps ticket via the Service Desk.

ops-infra¶

URL: https://bitbucket.org/securitize_dev/ops-infra
Size: ~478 MB (largest repo overall)
Status: actively maintained

Terraform IaC managing all of Securitize's real AWS infrastructure.

Structure¶

ops-infra/
├── aws-dev/
│   ├── dev/, qa/, rc/, testing/
├── aws-prod/
│   ├── prod/, apac/, sandbox/, general/
├── aws-navigate/          # Navigate product infrastructure
└── aws-backup/            # EKS backup configuration

Technologies¶

Terraform HCL (~394 .tf files), AWS.

Coverage¶

EKS clusters, VPC, IAM, Security Groups.
RDS (PostgreSQL, including BBVA migration).
Lambda, Glue jobs, Kinesis, DynamoDB, QLDB.
Redis (ElastiCache).
External Secrets, Fireblocks Cosigner, SFS Portal, CA Bank Zerohash.
Terraform state in S3 per environment (separate backends).

Account IDs for the aws-dev/ and aws-prod/ folders are in aws-infrastructure.md.

ops-terraform-modules¶

URL: https://bitbucket.org/securitize_dev/ops-terraform-modules
Size: ~1.3 MB
Last updated: 2024-07-17

Reusable Terraform module library consumed by ops-infra.

Available modules (all under `aws/`)¶

Module	AWS Service
`acm/`	Certificate Manager
`cloudfront/`	CloudFront CDN
`cloudwatch_alarms/`	CloudWatch Alarms
`eks/`	Elastic Kubernetes Service
`event_bridge/`	EventBridge
`iam/`	IAM roles, users, groups
`redis/`	ElastiCache Redis
`rds/`	Relational Database Service
`route53_healthcheck/`	Route53 Health Checks
`s3-bucket/`	S3 Buckets
`security_group/`	VPC Security Groups
`vpc/`	Virtual Private Cloud

Standard module pattern: main.tf + vars.tf + output.tf + providers.tf — 54 .tf files total.

Tags¶

ops #repositories #kubernetes #terraform #jenkins #gitops #argocd #ops-scripts #ops-manifest #ops-infra¶

Ops Repositories¶

Overview¶

Relationship map¶

ops-scripts¶

Structure¶

Technologies¶

What it does¶

Per-service ConfigMaps¶

Global ConfigMaps / Secrets (k8s/global/)¶

Per-service secret references¶

ops-deployments¶

Structure¶

Technologies¶

What it does¶

ops-manifest¶

Structure¶

ops-k8s-infra¶

Structure (7 environments)¶

Technologies¶

What it does¶

Ingress and DNS configuration¶

Service types and browser URLs¶

Write target — the current file per env × type¶

Why there are multiple files per environment — current vs at capacity¶

Read target — full file set per env × service_type¶

Ingress rule format¶

Production ingress changes¶

ops-infra¶

Structure¶

Technologies¶

Coverage¶

ops-terraform-modules¶

Available modules (all under aws/)¶

See also¶

Tags¶

ops #repositories #kubernetes #terraform #jenkins #gitops #argocd #ops-scripts #ops-manifest #ops-infra¶

Global ConfigMaps / Secrets (`k8s/global/`)¶

Write target — the `current` file per env × type¶

Why there are multiple files per environment — `current` vs `at capacity`¶

Available modules (all under `aws/`)¶