CampusSlot

THE PROBLEM

A small booking service is easy to make work once. This project shows what happens when it is deployed, changed, broken and grown, with numbers instead of claims.

HOW IT WORKS

• FastAPI, React and PostgreSQL. The rule that two confirmed bookings never overlap is enforced in request validation, in an application check and by a PostgreSQL exclusion constraint on a time range.
• Delivery: GitHub Actions gates (tests, secret scan, Trivy on the images, the Terraform and the Helm chart), signed image attestations verified before deploy, a deploy job that installs the chart into a throw-away kind cluster and smoke-tests it through the ingress, Argo CD, Prometheus, Loki and Grafana on a local Minikube cluster, and Terraform for a VPC and EKS cluster that was applied once and destroyed.
• Each claim in the README has a measurement behind it in docs/engineering and raw output in docs/evidence, and numbers that did not reproduce were withdrawn in writing.

THE PIPELINE

ENGINEERING EVIDENCE

• A concurrent-write test on real PostgreSQL creates exactly one booking when identical requests arrive together; separate tests show the database rejects overlaps even when the application check is bypassed. test_postgres.py ↗
• At 200,000 bookings the overlap query read 2,469 pages; a range operator with a GiST index reads 30, and the all-rooms day view went from 21 to 113 requests per second. Synthetic data, one laptop. data-scale.md ↗
• Rolling updates lost requests (7 of 2,500) until an 8-second preStop delay was added (0 of 1,800 in three runs). A deliberately broken release rolled back by itself at 62 s with 0 of 4,800 user requests failing across three runs, and a control without --atomic showed what the rollback adds. bad-release-drill.md ↗
• A database outage returned 500s that Prometheus did not count; the backend now answers 503 and the metric counts them, and an alert drill fired after 185 s. resilience.md ↗
• Median pipeline time fell from 238 s to 148 s (three runs after the change) by running independent jobs in parallel and skipping the heavy jobs for documentation-only changes. ci-speed.md ↗
• The Terraform for a 48-resource VPC and EKS cluster failed on first apply, was diagnosed and fixed, and was destroyed about 45 minutes after creation for roughly 6 US cents; an all-region check found nothing left running. terraform-destroy.txt ↗
• 14 architecture decision records on main, each listing the options that were rejected, and 26 merged pull requests. README.md ↗

DECISIONS & INVESTIGATIONS

• A rolling update that was configured as zero-downtime lost requests until a preStop delay was addedCONFIRMED

  No, it was not zero downtime.

• A database outage returned 500s that Prometheus never counted, so availability alerts stayed silentCONFIRMED

  Users received 18 responses with status 500; Prometheus recorded none and the 5-minute error ratio stayed at 0.

• The overlap query read 2,469 pages at 200,000 bookings; a range operator and GiST index brought it to 30ADOPTED

  Before: cost grew in exact proportion to the table, because the overlap question was written as start_time < end AND end_time > start; for a period near today the first condition is true for almost every old row, and...

• A CPU limit chosen without measuring throttled the pod; two workers and an ETag cache gave nothingADOPTED

  At the original 500m limit one pod served 153 requests per second on the room list (p95 178 ms) and the kernel throttled it in 96 to 100 percent of scheduling periods, so it was waiting for CPU quota.

• A broken release rolled back at 62 s with 0 of 4,800 requests failing; a control shows what --atomic addsCONFIRMED

  Kubernetes saw the new pod was not answering within 3.6 to 5.4 s.

• The pipeline's median fell from 238 s to 148 s once jobs that did not depend on each other ran in parallelADOPTED

  The jobs ran one after another: gates (about 34 s), build, scan and push (about 98 s), deploy (about 97 s).

• The first EKS apply failed on a subnet setting; the cluster was destroyed after 45 minutes for about 6 centsCONFIRMED

  The first apply failed: the node group was created in CREATE_FAILED with Ec2SubnetInvalidConfiguration, because the public subnets did not assign public IPs and EKS refuses such a group when there is no NAT gateway....

• The no-double-booking rule is enforced three times, ending in a PostgreSQL exclusion constraintADOPTED

  Some redundancy is accepted on purpose.

• Images are attested at build and the deploy job refuses any image whose attestation does not verifyADOPTED

  A full pipeline run took about a minute longer (143 s without attestations; 202 and 214 s with them; three runs is a small sample).

• Backups are a CronJob dump in the same cluster, accepted after a restore drill that deleted the dataPARTIAL

  Stated plainly: the backup volume is separate from the database volume, so a mistake or a corrupted database volume is covered, but losing the node or the cluster loses both.

• An expiring scanner exception was fixed at the root: PostgreSQL now runs with a read-only root filesystemADOPTED

  The change was checked three ways: the same settings in Docker (a write to /etc is refused and the database accepts writes), the real chart on a throw-away local kind cluster with the real published images (rooms...

NEXT STEPS

Each one comes from a gap listed above. It says what fixing the gap would take; it is not a promise.

• Add Alertmanager so alerts and automatic rollbacks reach a person.
• Copy backups off the cluster, or move to a managed database with point-in-time recovery.
• Add an idempotency key to booking creation so a client retry cannot book twice.
• Enforce the image-attestation check inside the cluster with an admission policy.

STACK

• FastAPI
• React
• PostgreSQL
• Docker
• Kubernetes
• Helm
• Argo CD
• Terraform
• Prometheus
• GitHub Actions