Immutable Infrastructure

When you run servers for a long time, they slowly fill up with hand-made changes: a package installed during a 2 a.m. incident, a config file someone edited and forgot, an old log nobody cleaned up. Over months, no two servers look alike. Immutable infrastructure is a way of working that stops this from ever happening. The rule is simple: once a server is built and running, you never change it. If you need a new version, you build a brand-new server image and replace the old one. This page explains the idea, how to build “golden images” with Packer, and why rebuild-and-replace beats editing servers in place.

Mutable vs immutable infrastructure

The old, traditional way is mutable infrastructure (mutable means “changeable”). You provision a server once, then keep updating it forever — you SSH in, run apt upgrade, edit configs, deploy new code on top of the old code. The same physical (or virtual) server lives for years and keeps mutating.

Immutable infrastructure flips this. A running server is treated as read-only. To deploy a change you build a fresh image with the change baked in, launch new servers from that image, send traffic to them, and then destroy the old ones. Servers become disposable, like shipping containers rather than pets you nurse back to health.

Aspect	Mutable (traditional)	Immutable (rebuild-and-replace)
How you deploy	SSH in and edit the live server	Build a new image, launch new servers
Config drift	Common — servers slowly diverge	Impossible — every server is identical
Rollback	Undo changes by hand (risky)	Re-launch the previous image
Debugging	”Why is this one server weird?”	All servers are byte-for-byte the same
Patching	`apt upgrade` on each host	Rebuild image with patches, replace fleet

The core problem immutable infra solves: config drift. Config drift is when servers that should be identical slowly become different because of manual, undocumented changes. Drift causes the classic “it works on server A but not server B” bug that nobody can explain. Immutable infra makes drift impossible because nobody ever logs in to change a running server.

When to use this (and when not to)

Use immutable infrastructure when you run more than one server, deploy often, or need fast and safe rollbacks. It pairs perfectly with autoscaling and cloud environments where launching a new machine takes a minute.

It is less helpful when you have a single hand-managed box, stateful software that is hard to move (a big database with local disks), or a tiny hobby project where the rebuild tooling is more effort than it saves. Note that state (databases, uploaded files) must live outside the immutable servers — on managed databases or object storage — or you lose it every time you replace a server.

Golden images and Packer

A golden image is a pre-built machine image that already contains everything a server needs: the operating system, your runtime, dependencies, config, and often your application code. You build it once, then launch as many identical servers from it as you want. “Golden” just means it is the blessed, tested, ready-to-ship version.

Packer (by HashiCorp) is the standard tool for building golden images. It takes a config file, spins up a temporary build server, runs your install steps on it, then saves the result as a reusable image — for example an AWS AMI (Amazon Machine Image, a snapshot you launch EC2 servers from) or a Docker image.

Install Packer on Ubuntu 22.04 / 24.04

wget -O - https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update
sudo apt install -y packer
packer version

Output:

Packer v1.11.2

A Packer template that bakes a golden image

This builds an Ubuntu 22.04 AMI with Nginx (a fast web server and reverse proxy) pre-installed and enabled. Packer files use HCL (HashiCorp Configuration Language). Save it as web.pkr.hcl.

packer {
  required_plugins {
    amazon = {
      version = ">= 1.3.0"
      source  = "github.com/hashicorp/amazon"
    }
  }
}

source "amazon-ebs" "web" {
  region        = "us-east-1"
  instance_type = "t3.micro"
  ssh_username  = "ubuntu"
  ami_name      = "web-golden-{{timestamp}}"

  source_ami_filter {
    filters = {
      name                = "ubuntu/images/*ubuntu-jammy-22.04-amd64-server-*"
      virtualization-type = "hvm"
      root-device-type    = "ebs"
    }
    owners      = ["099720109477"] # Canonical
    most_recent = true
  }
}

build {
  sources = ["source.amazon-ebs.web"]

  provisioner "shell" {
    inline = [
      "sudo apt-get update",
      "sudo DEBIAN_FRONTEND=noninteractive apt-get install -y nginx",
      "sudo systemctl enable nginx"
    ]
  }
}

Build the image:

packer init web.pkr.hcl
packer build web.pkr.hcl

Output:

amazon-ebs.web: output will be in this color.

==> amazon-ebs.web: Creating temporary keypair...
==> amazon-ebs.web: Launching a source AWS instance...
==> amazon-ebs.web: Provisioning with shell script...
    amazon-ebs.web: Setting up nginx (1.18.0-6ubuntu14.4) ...
==> amazon-ebs.web: Stopping the source instance...
==> amazon-ebs.web: Creating AMI web-golden-1718409600...
Build 'amazon-ebs.web' finished after 4 minutes 12 seconds.

==> Builds finished. The artifacts of successful builds are:
--> amazon-ebs.web: AMIs were created:
us-east-1: ami-0abc123def4567890

The ami-0abc123def4567890 ID is your golden image. Every server you launch from it is identical.

Why rebuild-and-replace wins

No config drift. Servers are never edited, so they cannot diverge. What you test is exactly what runs in production.
Trivial rollback. A bad release? Launch servers from the previous image ID and shift traffic back. There is nothing to “undo” by hand.
Repeatable and auditable. The image is built from a file in Git. You can see exactly what is on a server by reading the Packer template, not by SSHing in to look.
Faster recovery. If a server misbehaves, you do not debug it — you terminate it and let a new one boot from the known-good image.

Containers are immutable by nature

A Docker container is immutable infrastructure in miniature. A container image is a golden image, and you never edit a running container — you build a new image and roll it out. The same Dockerfile discipline (bake everything in, treat state as external) is exactly the immutable pattern, just at the process level instead of the whole-machine level. If you already use containers, you are already doing immutable infra.

Pairing with blue-green deploys

Blue-green deployment is the natural way to swap immutable servers safely. You run two identical environments: blue (the current live version) and green (the new version built from your new image). You launch green from the new golden image, test it, then flip the load balancer (a server that spreads traffic across machines) to send users to green. If something breaks, you flip back to blue instantly — both fleets are still standing. Once green is proven healthy, you destroy blue.

Best Practices

Keep your Packer templates in Git and tag every image with the commit or version it was built from.
Bake dependencies and config into the image; never install or edit things on a running server.
Store all state (databases, user uploads, sessions) outside the servers, in managed databases or object storage like S3.
Rebuild images regularly — at least to pick up OS security patches — rather than running apt upgrade on live hosts.
Never enable SSH-based manual edits as a “quick fix”; if a fix is needed, change the template and rebuild.
Combine immutable images with blue-green or rolling deploys so you can roll back by re-pointing traffic, not by repairing servers.