Logging Fundamentals

Logs are the diary of your servers and applications: a running record of what happened, when it happened, and often why. When a website goes down at 3 AM, you are not in the room watching it break, so logs are usually the only evidence you have. Learning to read, write, and manage logs well is one of the highest-value skills in DevOps (the practice of building, deploying, and running software reliably). This page lays the foundation for the rest of the Logging section.

What a log actually is

A log is a stream of text messages produced by a program as it runs. Each line is a log entry, and a good entry records a single event: a request arrived, a database query failed, a user signed in. Most entries follow a loose pattern of timestamp + severity + message, for example:

2026-06-15T14:03:22Z INFO  user 482 logged in from 203.0.113.7
2026-06-15T14:03:25Z ERROR payment gateway timeout after 30s (order 9912)

On Ubuntu, most logs land in one of two places: plain text files under /var/log/, or the systemd journal (a structured, binary log store managed by systemd, the program that starts and supervises services on modern Ubuntu). You read the journal with journalctl.

# Show the last 20 lines of the system journal, newest at the bottom
journalctl -n 20

Output:

Jun 15 14:01:10 web01 systemd[1]: Started nginx.service - A high performance web server.
Jun 15 14:02:44 web01 sshd[2210]: Accepted publickey for deploy from 198.51.100.4
Jun 15 14:03:25 web01 myapp[3187]: ERROR payment gateway timeout after 30s

When to use logs (and when not to)

Use logs to investigate something that already happened (an error, a crash, a slow request), to audit who did what, and to spot patterns over time. Do not rely on logs as your primary alerting system or as a real-time metrics dashboard — for “how many requests per second right now” you want metrics tools (covered elsewhere in DevOps), and for “tell me the instant something breaks” you want alerts. Logs are for context and forensics; metrics are for trends; alerts are for urgency.

Log levels

A log level (also called severity) tells you how important a message is. Almost every logging system uses the same core ladder. Picking the right level is what lets you later filter out noise and zoom in on real problems.

Level	Meaning	Example	Show in production?
DEBUG	Fine-grained detail for developers	”SQL query built: SELECT * FROM users WHERE id=482”	No (too noisy)
INFO	Normal, expected events worth recording	”Server started on port 8080”	Yes
WARN	Something odd but recoverable	”Retrying connection (attempt 2 of 3)“	Yes
ERROR	An operation failed	”Could not write to database”	Yes
FATAL / CRITICAL	The app cannot continue	”Out of memory, shutting down”	Yes

The levels are ordered. When you set a level, you get that level and everything more severe. So setting the level to WARN shows WARN, ERROR, and FATAL, but hides DEBUG and INFO. This is the single most useful logging control you have:

In development, run at DEBUG to see everything.
In production, run at INFO (or WARN for very chatty apps) to keep logs readable and storage cheap.

Resist the urge to log everything at INFO. If every line is “important”, nothing is. Reserve ERROR for things a human should actually look at, or your real errors will drown in noise and your on-call engineer will start ignoring the logs.

What to log — and what to never log

The goal of a log message is to let future you, at 3 AM, understand what happened without re-reading the source code. A useful message names the event, includes the identifiers needed to trace it (a user ID, an order ID, a request ID), and the outcome.

Good things to log:

Application startup and shutdown, with version and config summary.
Incoming requests and their result (status code, duration).
Errors with enough context to reproduce them — what failed and which record.
Important state changes: a user was created, a payment was captured.

Things to never log:

Passwords, API keys, tokens, or session cookies.
Full credit card numbers, government IDs, or other sensitive personal data.
Entire request bodies that might contain any of the above.

Logs are often shipped to third-party systems and read by many people. A leaked password in a log file is a real breach. Treat your log output as if it could become public, and redact secrets before they are ever written. Replace them with a marker like **** or a one-way hash.

A weak message versus a strong one:

ERROR something went wrong
ERROR failed to charge order 9912 for user 482: gateway timeout after 30s

The first tells you nothing. The second tells you the operation, the records involved, and the cause — you can act on it immediately.

App logs vs system logs

Two broad families of logs live on a server, and beginners often confuse them.

	App logs	System logs
Produced by	Your application code	The OS and its services
Examples	Web requests, business errors	SSH logins, kernel messages, service crashes
Where on Ubuntu	App’s own file (e.g. `/var/log/myapp/app.log`) or the journal	`/var/log/syslog`, `/var/log/auth.log`, the systemd journal
You read with	`tail`, `less`, `journalctl -u myapp`	`journalctl`, `cat /var/log/auth.log`

Some quick, real commands to inspect each on Ubuntu 22.04 / 24.04:

# Follow your app's log live (Ctrl+C to stop)
sudo journalctl -u myapp -f

# See system authentication events (logins, sudo use)
sudo tail -n 50 /var/log/auth.log

# Show only kernel messages since the last boot
journalctl -k -b