The ELK / Elastic Stack

When you run many servers, reading logs by hand with tail and grep stops working. The ELK stack solves this: it pulls logs from every machine into one place where you can search, filter, and chart them in seconds. “ELK” stands for the three original tools — Elasticsearch, Logstash, and Kibana — and together they are now branded the “Elastic Stack”. This page explains what each part does, how the pipeline fits together, and when a lighter alternative makes more sense.

What each piece does

The stack is built from a few independent programs. Each does one job and hands off to the next.

Component	Role	Plain-English job
Elasticsearch	Store + search engine	A database built for text. It indexes every log line so a full-text search returns in milliseconds.
Logstash	Processing pipeline	Receives raw logs, parses them (splits a line into fields like `ip`, `status`, `time`), and writes them to Elasticsearch.
Beats	Lightweight shippers	Tiny agents that run on each server and forward log files. Filebeat ships log files; Metricbeat ships CPU/memory metrics.
Kibana	Web UI	A browser dashboard to search logs, build charts, and set alerts.

A few terms first. Indexing means building a lookup table so search is fast, the same way a book index lets you jump to a page. A field is a named piece of data inside a log line (for example, status=404). An agent is a small background program that runs on a server and does one task.

The pipeline

The modern flow puts Beats at the edge and treats Logstash as optional:

[ Your app / Nginx / syslog ]
            |
            v
   Filebeat (on each server)   <- reads log files, tails them
            |
            v
   Logstash (optional)         <- parses & enriches (grok, geoip)
            |
            v
   Elasticsearch               <- stores + indexes
            |
            v
   Kibana                      <- search & dashboards in the browser

In small setups Filebeat can ship straight to Elasticsearch and skip Logstash entirely. Add Logstash only when you need heavy parsing — for example turning a raw Nginx line into structured fields, or adding the country an IP came from.

When to use this

Use ELK when you have several servers, want one searchable home for all logs, and need dashboards or alerts.
Use just Filebeat + Elasticsearch + Kibana (“EFK-lite”) when your logs are already structured (JSON) and need no parsing.
Do NOT reach for ELK on a single small VPS — it is heavy (see below). A simple journalctl plus log rotation is enough there.

Installing the stack on Ubuntu

These steps target Ubuntu 22.04 / 24.04 LTS. Run them on a server with at least 4 GB of RAM — Elasticsearch alone wants 2 GB or more.

First add Elastic’s package repository (a repository is a remote source apt downloads software from):

sudo apt update
sudo apt install -y apt-transport-https gnupg curl
curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | \
  sudo gpg --dearmor -o /usr/share/keyrings/elastic.gpg
echo "deb [signed-by=/usr/share/keyrings/elastic.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | \
  sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt update

Install and start Elasticsearch:

sudo apt install -y elasticsearch
sudo systemctl enable --now elasticsearch
sudo systemctl status elasticsearch --no-pager

Output:

● elasticsearch.service - Elasticsearch
     Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled)
     Active: active (running) since Mon 2026-06-15 10:12:03 UTC; 21s ago
   Main PID: 4412 (java)

Elasticsearch 8.x turns on security by default and prints a password the first time. Grab the built-in elastic user password like this:

sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic

Now install Kibana and start it:

sudo apt install -y kibana
sudo systemctl enable --now kibana

Kibana listens on port 5601. By default it binds only to localhost, which is correct — you should reach it through an SSH tunnel or a reverse proxy (a server that sits in front of an app and forwards requests to it), never expose it raw to the internet.

Security gotcha: Never open Elasticsearch’s port 9200 or Kibana’s 5601 to the public internet with ufw. An exposed Elasticsearch is one of the most common sources of mass data leaks. Keep them on localhost and front Kibana with Nginx + basic auth, or use an SSH tunnel: ssh -L 5601:localhost:5601 user@server.

Shipping logs with Filebeat

On every server whose logs you want, install Filebeat (the repo is already added):

sudo apt install -y filebeat

Edit /etc/filebeat/filebeat.yml to point at your Nginx logs and your Elasticsearch host:

filebeat.inputs:
  - type: filestream
    id: nginx-access
    enabled: true
    paths:
      - /var/log/nginx/access.log

output.elasticsearch:
  hosts: ["https://10.0.0.5:9200"]
  username: "elastic"
  password: "your-password-here"
  ssl.verification_mode: "certificate"

Filebeat ships handy pre-built parsers called modules. Enable the Nginx one so logs arrive already split into fields:

sudo filebeat modules enable nginx
sudo filebeat setup --dashboards
sudo systemctl enable --now filebeat

Output:

Enabled nginx
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards

Open Kibana at http://localhost:5601, go to Discover, and you will see live log lines you can filter by field — for example nginx.access.response_code: 500.

Resource cost and the Loki alternative

The catch with ELK is weight. Elasticsearch keeps indexes in memory and on fast disk, so a real deployment needs several gigabytes of RAM and SSD storage. For high log volumes you often run a cluster of three or more nodes (a node is one Elasticsearch process).

If that is too much, Grafana Loki is the popular lightweight alternative. Loki indexes only labels (like job and host), not the full log text, so it uses far less storage and RAM.

Need	ELK / Elastic Stack	Grafana Loki
Full-text search on log bodies	Excellent	Limited (label-based)
Resource footprint	Heavy (GBs of RAM)	Light
Storage cost	Higher	Much lower (cheap object storage)
Best when	Deep search, complex dashboards	Cost-sensitive, label-driven queries

Choose ELK when rich search and analytics justify the hardware. Choose Loki when you mostly query by labels and want to keep costs down.

Best Practices

Send Beats agents straight to Elasticsearch and only add Logstash when you genuinely need parsing or enrichment.
Keep Elasticsearch and Kibana bound to localhost or a private network; never expose ports 9200 or 5601 publicly.
Use Index Lifecycle Management (ILM) to auto-delete old indexes so disks do not fill — log data grows fast.
Set the JVM heap (Elasticsearch’s memory) to about half your RAM, capped at 32 GB, in /etc/elasticsearch/jvm.options.
Prefer Filebeat modules over hand-written parsers for common sources like Nginx, system, and MySQL logs.
Monitor the stack itself with Metricbeat — a logging system that silently dies helps no one.