Building a cloud-agnostic, zero-downtime deployment platform
Signed releases promoted by digest, rollouts that roll themselves back, and workers that scale on queue depth. Here is how I designed it, what I proved under load, and what I'd still do differently.
The brief
This was a senior DevOps assignment with a big scope. I had to build a platform that can:
- Provision infrastructure on any cloud from an empty account, with no clicking in a console.
- Deploy an API and a background worker, backed by PostgreSQL and Redis, through dev, staging and production with zero downtime.
- Stop a bad release automatically, based on health and metrics, without a human watching.
- Scale to 15× peak load in under five minutes and back down, without dropping a single job.
- Serve shared and dedicated tenants from one template, onboard a new tenant in under 30 minutes, and keep EU data in the EU.
- Come with an architecture document, cost estimate, runbooks, a 90-day roadmap and a change request for a second cloud.
Architecture: three layers, one contract
To keep it cloud-agnostic, I split the platform into three layers. They communicate only through a fixed set of outputs: the cluster endpoint, database and Redis hosts, secret references, region and residency.
- Provider layer (Terraform): network, Kubernetes, PostgreSQL, Redis/Valkey, KMS, secrets, IAM/OIDC and remote state. There is one module per provider. Today that means AWS plus a local kind target. Adding a cloud means writing one new module, not editing the others.
- Platform layer: metrics-server, KEDA, External Secrets, Argo CD, Argo Rollouts and Prometheus.
- Application layer: one Helm chart for both the API and the worker, plus one values file per tenant, rolled out in rings.
Requests flow from the load balancer to stateless API pods (with /healthz, /readyz and /metrics), then through a Redis stream to worker pods, which write to PostgreSQL.
How a release ships
The rule I built everything around is build once, promote the digest. GitHub Actions builds one image and pushes it to GHCR. From then on, its sha256 digest is the release's identity. Trivy fails the build on any fixable CRITICAL or HIGH vulnerability. syft generates a CycloneDX SBOM, and Cosign signs the image keylessly using GitHub's OIDC token, so there is no signing key that could leak.
Each promotion re-verifies the signature and the SBOM attestation against this repository's workflow identity. It then points the environment tag at the same digest, with nothing rebuilt. Staging and production also need a manual approval. The bytes tested in staging are exactly the bytes that run in production.
Rollouts that roll themselves back
Deploys go through a gated rollout script. It runs helm upgrade --wait and then four metric-analysis steps against the live pods. If a probe or a metric check fails, it runs helm rollback on its own. Readiness probes, graceful shutdown and PodDisruptionBudgets keep traffic flowing while pods are replaced. When a worker receives SIGTERM, it finishes its in-flight job before it exits.
Database changes use expand/contract migrations, so the old and new versions can run against the same schema during a rollout.
Scaling and tenancy
The API scales on CPU with an HPA. Workers scale on queue lag with KEDA, because for pull-based, I/O-bound jobs, the backlog is the real demand signal and CPU isn't.
Shared tenants run in namespaces. Dedicated tenants get their own provider stack, built from the same Terraform module. Onboarding a tenant means adding one values file, which goes through one reviewed change. EU residency is enforced in three places: Terraform validation, a policy gate during onboarding, and a Kubernetes node-affinity guard.
What I proved
I ran every runtime test on a local Kubernetes (kind) cluster. Each one is backed by raw logs and terminal recordings in the repository.
The CI pipeline is green on GitHub. One digest was built, scanned, signed and promoted to dev, staging and production, with both approval gates exercised.
What went wrong, and what I learned
I kept every failed attempt in the evidence folder. The failures taught me more than the passing runs did:
- CI failed twice, and it wasn't a CVE. The Trivy setup action's installer exited with code 6, so the scan never actually ran. I replaced the action with the official Trivy container pinned by digest. Real findings now fail with exit code 1 and scanner errors with exit code 2, so the two can no longer be confused.
- My own rollout gate had a bug. When the HPA scaled from 2 to 5 pods mid-rollout, the request delta went negative. The gate failed safe and rolled back, but for the wrong reason. I switched to per-pod deltas.
- Measure from the right place. An early test showed 50 of 120 requests failing through a
kubectl port-forward, which most likely stayed attached to a terminated pod. Since then, all load is generated inside the cluster, through the Service. - Rolling releases have a blast radius. A bad rolling release exposed about 13% of requests for roughly 87 seconds before the gate caught it. That measurement is why the cloud design uses a 10% canary instead.
Honest scope
- All runtime results come from a single-node local cluster on a laptop. The AWS Terraform passes
validateandterraform testbut was not applied, so no cloud resources exist. - The 15× test was a ratio (10 → 150 rps). The full 2,250 rps peak needs cloud capacity and was not run.
- The assignment's own application wasn't supplied, so I built a small FastAPI and Redis reference app that follows the same contract.
- Argo CD, Argo Rollouts and Prometheus alerting are written as configuration but weren't installed locally.
Being explicit about what is tested, what is only implemented and what is only designed was as much a part of the work as the YAML.