3B Infrastructure home

DevOps

How teams ship software often and safely: from the first commit, through the pipeline, to keeping it running in production. Press play on any sketch to watch it happen.

The DevOps way of working

1 What DevOps is

DevOps is a way of working where the people who build software also take responsibility for running it. The aim is small, frequent, safe changes, supported by automation and quick feedback. It is a culture first and a set of tools second.

  • Shared ownership: no wall between "dev wrote it" and "ops must run it".
  • Automation: anything repeated, like tests and deploys, is done by machines.
  • Small batches: small changes are easier to review, test and roll back.
  • Feedback: production data flows back and shapes what gets built next.

Cycle diagram

PlanCodeBuildTestReleaseDeployOperateMonitor

1/6 An idea becomes a small, clear piece of work.

Rule of thumb: If a release feels scary, release more often and make each one smaller.

2 Measuring it: DORA metrics

Years of research from the DORA team show that software delivery can be measured, and that speed and stability go together. High performers are both faster and safer. Four metrics are the classic starting point.

  • Deployment frequency: how often you ship to production.
  • Lead time for changes: how long a commit takes to reach production.
  • Change failure rate: how many deployments cause a problem.
  • Time to restore: how quickly you recover from a failure.

Metrics map

SpeedStabilityDeploysonce a monthLead time3 weeksFailed deploys30%Time to restore1 day

1/4 A team ships big monthly releases. They feel careful, but they are slow and fail often.

Rule of thumb: Use these to improve the system, never to rank individual people.

Building the pipeline

3 Branches and pull requests

Every change starts in version control. Long-lived branches drift apart and create painful merges, so most high-performing teams keep branches short and merge into the main line often.

  • Trunk-based development: small branches that merge into main within a day or so.
  • Pull request: a place to review a change, with CI results attached.
  • Feature flags: merge unfinished work switched off, so main is always releasable.
  • Branch protection: require a review and passing checks before anything reaches main.

Git flow diagram

mainmainShortbranchPull request+ CI + reviewmerge

1/5 Create a short branch from main for one small change.

Rule of thumb: If a branch lives for more than a few days, split the work.

4 Continuous Integration

Every push is automatically built and tested, so problems surface within minutes while the change is still fresh. This site does exactly that with GitHub Actions.

  • Trigger: every push and pull request starts a run.
  • Fail fast: run the quick checks first, like linting and unit tests.
  • Same every time: a clean machine each run, so "works on my laptop" stops mattering.
  • Fix it first: a broken main branch is the team's top priority.
on: [push, pull_request]
jobs:
  quality:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - run: npm ci
      - run: npm run lint
      - run: npm test
      - run: npm run build

Pipeline diagram

PushLintTestBuild

1/5 A developer pushes a commit, and a clean machine starts up.

Rule of thumb: Keep CI under ten minutes. Past that, people stop waiting for it.

5 Continuous Delivery and Deployment

The two terms sound alike but differ by one human click. In Continuous Delivery, every passing build is ready to release and a person decides when. In Continuous Deployment, every passing build goes live on its own.

  • Artifact: build once, then promote that exact build through each environment.
  • Staging: a copy of production for a last check.
  • Smoke test: a quick check after deploying that the key paths still work.
  • Rollback: the old version must be one command away.

Pipeline diagram

Buildv2.4StagingApprove?Production

1/5 CI produces one build. This same build moves through every environment.

Rule of thumb: Never rebuild between environments, or you are not testing what you ship.

Packaging and running

6 Containers and Docker

A container packages your app with everything it needs, so it runs the same on a laptop, in CI and in production. It shares the host kernel, so it starts in moments and is far lighter than a virtual machine.

  • Dockerfile: the recipe for building an image.
  • Image: a read-only package of your app. Tag each one with a version.
  • Registry: where images are stored and shared, like Docker Hub or Amazon ECR.
  • Container: a running instance of an image. Treat it as disposable.
FROM node:24-slim
WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY . .
USER node
CMD ["node", "server.js"]

Pipeline diagram

DockerfileImagev1.2Registrybuildpush

1/4 The Dockerfile describes exactly how to build your app.

Rule of thumb: Keep data outside the container, in a volume or a database. Containers come and go.

7 Kubernetes

Kubernetes runs containers across a group of machines. You declare what you want, such as three copies of an app, and it keeps working to make reality match. It is powerful, and it is also a lot to operate.

  • Pod: the smallest unit, usually one container.
  • Deployment: declares how many Pods you want and handles rolling updates.
  • Service: a stable address that shares traffic across changing Pods.
  • Node: a machine that runs Pods.
  • Control plane: the brain that watches the desired state and fixes differences.

Deployment diagram

Node 1Node 2ControlplaneServicePodPodPod

1/5 You declared "I want 3 Pods". The Service shares traffic across them.

Rule of thumb: Most small teams do not need Kubernetes. Reach for it when you outgrow simpler platforms.

8 Infrastructure as Code

Servers, networks and databases are described in text files and created by a tool, not clicked together in a console. Your infrastructure becomes reviewable, repeatable and recoverable.

  • Terraform: declarative and works across many clouds.
  • AWS CloudFormation: the AWS native option.
  • Plan: preview exactly what will change before anything happens.
  • State: the tool's record of what exists. Store it remotely and lock it.
  • Drift: when someone changes things by hand, reality no longer matches the code.
resource "aws_instance" "web" {
  ami           = "ami-0123456789abcdef0"
  instance_type = "t3.small"
}

Process diagram

Code4 serversPlan +reviewApplyCloud2 servers

1/5 You change "2 servers" to "4 servers" in code and open a pull request.

Rule of thumb: Treat a manual console change as a bug, and put it back into code.

Real environments: Dev, Test, Production

9 Dev, Test, Staging, Production

Real companies never build on the live system. They keep several separate copies of it, called environments, and a change has to prove itself in each one before customers see it. The further along it gets, the more real the environment is and the more a mistake costs.

  • Local: your own laptop. Break anything you like.
  • Dev: a shared environment where the team's merged work first runs together.
  • Test or QA: where testers and automated suites try to break the feature.
  • Staging or UAT: a close copy of production, where the business signs off. UAT means user acceptance testing.
  • Production: the real system, with real customers and real money.
  • Parity: keep them as alike as you can, or passing in Test proves little.

Environment pipeline

DevTestStagingProduction

1/6 A developer merges a new checkout feature, and it lands in Dev.

EnvironmentUsed byDataIf it breaks
DevDevelopersFake seed dataAnnoying
TestQA, automated testsFake test dataDelays testing
StagingQA, business ownersMasked or realisticDelays the release
ProductionReal customersReal dataCosts money and trust

Rule of thumb: A bug costs minutes in Dev, hours in Test, and customers in Production. Catch it as early as you can.

10 A change, from ticket to live

Here is the path a real feature takes in a company. Each stage has a person or a check that can stop it. Teams with good automation can run this whole path in under an hour.

  • Pull request: a teammate reviews it and CI tests it.
  • Dev deploy: merging deploys to Dev automatically.
  • QA sign-off: testers check the ticket against its acceptance criteria.
  • UAT sign-off: the product owner confirms it does what the business asked for.
  • Release: the same tested build goes to Production, usually gradually.
  • Watching: someone stays on hand and checks dashboards after the release.

Process diagram

PullrequestDeployedto DevQA inTestSign-off inStagingRelease toProductionWatchdashboards

1/6 A pull request passes CI and gets a teammate's approval.

Rule of thumb: Avoid releasing late on a Friday. Release when the people who can fix it are awake and around.

11 Same code, different config

Each environment runs the same build. What differs is configuration: database addresses, API keys, feature flags. Keeping config outside the code is what makes this possible.

  • Environment variables: the app reads its settings when it starts, so one image fits everywhere.
  • Secrets: keep passwords in a secrets manager or the deploy platform, never in Git.
  • Separate accounts: give Production its own cloud account and credentials.
  • Different keys: payment providers offer test keys, so Test never charges real cards.
# Test
DATABASE_URL=postgres://test-db.internal/shop
PAYMENT_KEY=sk_test_xxx
LOG_LEVEL=debug

# Production: values come from the secrets manager
DATABASE_URL=postgres://prod-db.internal/shop
PAYMENT_KEY=(injected at deploy time)
LOG_LEVEL=info

Skip vs set up

One imagev2.4Test configProd configTest DBProd DB

1/5 SET UP the same image starts everywhere, and only its config changes.

If you skip this: one wrong address in a test config can send fake orders to real customers or delete production data.

Rule of thumb: Make it impossible to reach Production from Dev. Different accounts and credentials beat being careful.

12 Test data and database changes

Test environments need realistic data, but copying real customer data into them is a privacy risk, and often a legal one. Database changes also need care, because you cannot roll data back the way you roll back code.

  • Seed data: scripts that create fake but believable records.
  • Masking: if you need production-like data, scramble names, emails and card details first.
  • Migrations: versioned scripts that run through every environment in the same order.
  • Expand, then contract: add the new column first, move the code over, and remove the old column later.

Skip vs set up

Prod dataReal names

1/5 SKIPPED someone copies the production database into Test to reproduce a bug.

If you skip this: real customer data ends up on laptops and in less protected systems, where a leak is far more likely.

Rule of thumb: Run every migration against a production-sized copy in Staging. A change that takes a second on test data can lock a real table for an hour.

13 Fixing Production in a hurry

When Production breaks you do not repeat the whole path, but you do not skip the checks either. Roll back first, then send a small hotfix through a fast version of the same pipeline.

  • Roll back first: put the last good version back while you investigate.
  • Hotfix: a small branch from the released version, containing only the fix.
  • Same pipeline: tests still run. Editing files directly on a server is how outages get worse.
  • Merge back: the fix must return to main, or the next release brings the bug back.
  • Least access: few people should be able to touch Production, and every action should be logged.

Git flow diagram

Productionv2.4main

1/6 Version 2.4 is live, and checkout starts failing.

Rule of thumb: Plan your rollback before you release, because you will not think clearly at 2am.

Releasing safely

14 Deployment strategies

How you switch from the old version to the new one decides how risky a release is. The goal is to limit how many users see a bad version, and to be able to go back quickly.

  • Recreate: stop the old, start the new. Simple, but causes downtime.
  • Rolling: replace servers a few at a time. No downtime, and two versions run together.
  • Blue/green: run a full new copy, then switch all traffic at once. Rollback is instant, but it costs double while it runs.
  • Canary: send a small share of traffic to the new version, watch it, then increase.

Traffic diagram

UsersRouter100 / 0Version 1Version 2

1/5 All traffic is on version 1. Version 2 is deployed but receives nothing.

Rule of thumb: Make database changes backwards compatible, because old and new code run side by side during a deploy.

15 GitOps

Git becomes the single source of truth for what should run. Instead of a pipeline pushing changes into a cluster, an agent inside the cluster keeps pulling from Git and corrects any difference it finds.

  • Declarative: the desired state is written down in a repository.
  • Pull based: the cluster fetches changes, so CI needs no production credentials.
  • Self-healing: manual changes get reverted to what Git says.
  • Rollback: revert the commit, and the system follows.

Control loop diagram

Git reporeplicas: 3Agent(Argo CD)Clusterrunning: 3watchessyncs

1/5 Git says 3 copies, the cluster runs 3. Everything matches.

Rule of thumb: If it is not in Git, it should not be running.

Operating in production

16 Observability

You cannot fix what you cannot see. Observability means being able to tell what is broken and why, from the outside, using the data your system emits.

  • Logs: timestamped records of single events. Structure them so they can be searched.
  • Metrics: numbers over time, like request rate, error rate and latency. Cheap to keep and good for alerts.
  • Traces: the path of one request across services, showing where the time went.
  • OpenTelemetry: an open standard for collecting all three, so you are not tied to one vendor.

Data flow diagram

YourservicesLogsMetricsTracesDashboards+ alerts

1/4 Everything your services do leaves three kinds of evidence.

Rule of thumb: Alert on what users feel, like errors and slow requests, not on every CPU spike.

17 SLOs and error budgets

Site Reliability Engineering, from Google, treats reliability as an engineering problem with numbers. Instead of chasing 100% uptime, you agree how reliable a service must be, and use what is left over to ship faster.

  • SLI: a measurement, like the share of requests that succeed.
  • SLO: the target for that measurement, like 99.9% over 30 days.
  • SLA: a promise to customers, usually with money attached.
  • Error budget: what the target allows you to fail. 99.9% over 30 days is about 43 minutes.

State diagram

Error budget43 min leftShip featuresPause releasesbudget leftbudget gone

1/5 The month starts with 43 minutes of allowed failure.

Rule of thumb: When the budget is healthy, ship. When it is spent, stop features and fix reliability.

18 Incident response

Things will break. What separates strong teams is how calmly they respond, and whether they learn from it afterwards.

  • Detect: alerts should find the problem before customers do.
  • Roles: one incident lead makes decisions while others investigate.
  • Mitigate first: stop the damage, usually by rolling back. Find the root cause later.
  • Blameless review: ask how the system allowed it, not who did it.

Process diagram

1. Detect2. Triage3. Mitigate4. Reviewbetter alerts

1/5 02:14. An alert fires because checkout errors jumped.

Rule of thumb: Roll back first and investigate after. Users do not care why it is broken.

19 DevSecOps

Security stops being a final gate and becomes part of every step. Finding a problem in a pull request costs far less than finding it in production.

  • Secret scanning: block passwords and keys from ever entering Git.
  • Dependency scanning: alert on libraries with known vulnerabilities.
  • Code scanning: static analysis that catches risky patterns.
  • Image scanning: check containers before they ship.
  • Least privilege: give every service and person only the access they need.

Pipeline diagram

Commitblocked: secretBuild+ dependency scanImagescanProduction

1/5 A developer accidentally commits an API key. The scan blocks it before it spreads.

Rule of thumb: Treat a leaked secret as burned. Removing it from Git is not enough, so rotate it.