Rolling Back a Bad Deploy

Every engineer ships a broken release eventually. The skill that separates a calm incident from a multi-hour outage is the rollback (the act of putting the previous, known-good version of your app back into production quickly). When something breaks, your first instinct should not be to debug live in production while users suffer. It should be to get back to the last version that worked, and then investigate calmly. This page shows you how to design your deployments so rolling back takes seconds, not hours.

Roll back vs fix forward

When a deploy goes wrong you have two choices. Fix forward means you write a new patch, build it, and deploy it on top of the broken release. Roll back means you restore the previous release that you already know works.

Approach	When to use it	Risk
Roll back	Production is broken or degraded right now; users are affected	Low — you return to a state that already worked
Fix forward	The bug is tiny and obvious, OR a rollback is impossible (e.g. an irreversible database change already ran)	High — you are shipping untested code under pressure

Rule of thumb: during an active incident, roll back first and debug later. A fix you wrote in a panic in two minutes is almost never as safe as a version that ran fine for the last week.

The whole reason fast rollback is possible is immutability. An immutable release is one you never edit in place. You build version v42, you deploy it, and you never touch those files again. If v43 is bad, v42 is still sitting there, untouched, ready to serve traffic.

Keep releases versioned and immutable

The classic pattern on a Linux server (we target Ubuntu 22.04 / 24.04 LTS here) is the releases directory with a current symlink. A symlink (a special file that points to another path, like a shortcut) called current always points at the live release. Rolling back is just re-pointing that symlink.

Set up the directory layout once:

sudo mkdir -p /var/www/myapp/releases
sudo mkdir -p /var/www/myapp/shared

Each deploy lands in its own timestamped (or git-SHA-named) folder under releases/, and current points at it:

ls -l /var/www/myapp

Output:

lrwxrwxrwx 1 deploy deploy   41 Jun 15 10:02 current -> /var/www/myapp/releases/2026-06-15-1002-a3f9c1
drwxr-xr-x 5 deploy deploy 4096 Jun 15 10:02 releases
drwxr-xr-x 2 deploy deploy 4096 Jun 01 09:00 shared

Your service (systemd unit, Nginx root, etc.) always references /var/www/myapp/current, never a specific release folder. That indirection is what makes rollback instant.

Rolling back the symlink

Suppose the newest release is broken. List the releases, pick the previous good one, and re-point the symlink atomically with ln -sfn:

ls -1t /var/www/myapp/releases

Output:

2026-06-15-1002-a3f9c1
2026-06-15-0930-7b21de
2026-06-14-1745-c004aa

# Point 'current' back at the previous good release
sudo ln -sfn /var/www/myapp/releases/2026-06-15-0930-7b21de /var/www/myapp/current
sudo systemctl restart myapp

The flags matter: -s makes a symlink, -f forces overwrite of the existing one, and -n treats the existing current symlink as a file rather than following into it. Together they swap the link in a single step so there is no moment where current is missing.

Keep at least 3-5 old releases on disk. A rollback is only fast if the previous version is still there. Prune old releases (keep the most recent few), never the one you might need to fall back to.

Re-deploying the last good tag

If you deploy by pulling from git rather than uploading folders, your immutable unit is a git tag (a permanent, human-readable name pinned to one commit, like v1.8.3). Rolling back means checking out the previous tag and restarting.

cd /var/www/myapp/current
sudo -u deploy git fetch --tags
sudo -u deploy git checkout v1.8.3
sudo systemctl restart myapp

Output:

Note: switching to 'v1.8.3'.
HEAD is now at 9c1d2ef Release 1.8.3

This works because tags are immutable by convention — you never move a tag once it is released. Always tag every release so “the last good version” has a name you can check out, instead of hunting through commit hashes during an outage.

Blue-green switch-back

A blue-green deployment runs two identical environments: “blue” (the old version) and “green” (the new one). A reverse proxy (a server such as Nginx that sits in front of your app and forwards requests to it) sends traffic to whichever one is live. To roll back, you simply point traffic back at the old environment — which is still running, untouched.

Suppose Nginx routes to whichever upstream the file /etc/nginx/sites-available/myapp includes. You keep one tiny file that decides the target:

# /etc/nginx/conf.d/active_upstream.conf
upstream myapp {
    server 127.0.0.1:3001;   # 3001 = blue (old), 3002 = green (new)
}

To switch back to blue, edit the port, test the config, and reload (a reload re-reads config without dropping live connections):

sudo sed -i 's/3002/3001/' /etc/nginx/conf.d/active_upstream.conf
sudo nginx -t
sudo systemctl reload nginx

Output:

nginx: configuration file /etc/nginx/nginx.conf test is successful

Because the blue environment never stopped running, this rollback is near-instant and carries no risk of a failed restart. When to use blue-green: for high-traffic or revenue-critical apps where you cannot tolerate even a few seconds of downtime. When not to: for small apps where running two full copies wastes resources — the symlink method is plenty.

The database caveat

Code rolls back cleanly. Database migrations (changes to your database structure or data) often do not. This is the single biggest rollback trap.

If v43 ran a migration that dropped a column, rolling the code back to v42 will not bring that column’s data back — it is gone. The safe practice is expand-and-contract (also called backward-compatible migrations):

1. Expand: add the new column/table, but keep the old one. Deploy code that writes to both.
2. Migrate data in the background.
3. Contract: only after the new version has been stable for a while, remove the old column in a separate, later release.

During the window between steps, an old release still works because you never deleted anything it depends on. That is what makes the code rollback safe.

Never combine a destructive migration with a new feature in the same release. If you must drop a column, do it in its own deploy, days after the feature that stopped using it has proven stable. That way a rollback never lands on missing data.

Best Practices

• Make every release immutable and versioned — a timestamped folder or a git tag — and never edit a deployed release in place.
• Point production at a current symlink (or a single proxy config) so rollback is one atomic swap plus a restart.
• Keep the last 3-5 releases on disk; prune old ones, never the fallback you might need.
• Roll back first, debug second — restore the working version before investigating the cause.
• Use backward-compatible (expand-and-contract) migrations so a code rollback never lands on a broken schema.
• Run nginx -t (or your app’s config check) before every reload so a typo never compounds an incident.
• Practice a rollback in staging so the exact commands are muscle memory before you ever need them under pressure.

Related Topics

• Deployment overview
• Zero-downtime deploys
• Deployment strategies
• Deploy with Git
• Nginx in front of your app