Shipping Logs (Filebeat, Fluentd, Loki)

Once you have logs on more than one server, you cannot keep typing tail -f /var/log/... on each machine. You need a small program that reads your log files and forwards them to a central place. That program is called a log shipper (an agent that tails, or continuously reads, log files and sends new lines to a storage system over the network). This page compares the main shippers and shows you a real working setup so you can pick the right one.

What a log shipper actually does

A log shipper runs as a background service (a systemd unit, the Ubuntu service manager) on each server. It does three jobs:

1. Tail — watches files like /var/log/syslog or /var/log/nginx/access.log and notices new lines as they are written.
2. Parse / enrich — optionally turns each line into structured fields (for example pulling out the HTTP status code) and adds labels like the hostname or environment.
3. Forward — sends those lines to a central store (Elasticsearch, Loki, or a queue like Kafka) and remembers its position so it never loses data or sends duplicates after a restart.

The “remember its position” part matters. A good shipper keeps a small cursor file (Filebeat calls it the registry). If the central store is down, the shipper waits and retries instead of dropping logs. Never write your own shipper with a cron job and curl — you will lose logs on every reboot.

The main shippers compared

Shipper	Pairs with	Language / footprint	Best when
Filebeat	Elasticsearch / Logstash (the ELK stack)	Go, ~50 MB RAM, very light	You already run Elasticsearch and want the official, batteries-included agent
Fluentd	Anything (huge plugin ecosystem)	Ruby + C, ~200 MB RAM	You need to route to many destinations or do heavy parsing
Fluent Bit	Anything (Fluentd’s little sibling)	C, ~5 MB RAM, extremely light	Kubernetes nodes, edge devices, or any RAM-constrained host
Promtail / Grafana Alloy	Loki	Go, very light	You want a cheap, Prometheus-style stack queried in Grafana

A note on names: Promtail is the classic Loki shipper, but Grafana now ships Alloy as its successor. Alloy does the same tailing job plus metrics and traces. Promtail is still fully supported in 2026 and is simpler to learn, so we use it below.

When to use which

• Reach for Loki + Promtail first if you are starting fresh. Loki does not index the full text of every log line — it only indexes a few labels (like job and host) and stores the rest compressed. That makes it far cheaper on disk and RAM than Elasticsearch, and you query it in Grafana with a syntax that feels like Prometheus.
• Use Filebeat when ELK is already in the building. It is the standard, well-documented agent for that world.
• Use Fluent Bit on Kubernetes or tiny VMs where every megabyte of RAM counts.

Setting up Loki and Promtail

Here we run Loki (the store) on one central server and Promtail (the shipper) on every server whose logs you want.

Install Loki on the central server

Download the single Loki binary and a minimal config.

sudo apt update
sudo mkdir -p /etc/loki /var/lib/loki
cd /tmp
curl -fSL -o loki.zip "https://github.com/grafana/loki/releases/download/v3.4.2/loki-linux-amd64.zip"
unzip loki.zip
sudo mv loki-linux-amd64 /usr/local/bin/loki
sudo chmod +x /usr/local/bin/loki
loki --version

Output:

loki, version 3.4.2 (branch: HEAD, revision: a1b2c3d4)
  go version:       go1.23.5
  platform:         linux/amd64

Create a small config at /etc/loki/config.yaml:

auth_enabled: false

server:
  http_listen_port: 3100

common:
  instance_addr: 127.0.0.1
  path_prefix: /var/lib/loki
  storage:
    filesystem:
      chunks_directory: /var/lib/loki/chunks
      rules_directory: /var/lib/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: "2024-01-01"
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

Run Loki as a systemd service. Create /etc/systemd/system/loki.service:

[Unit]
Description=Loki log aggregation
After=network.target

[Service]
ExecStart=/usr/local/bin/loki -config.file=/etc/loki/config.yaml
Restart=on-failure
User=root

[Install]
WantedBy=multi-user.target

Start it and confirm it is listening:

sudo systemctl daemon-reload
sudo systemctl enable --now loki
curl -s http://localhost:3100/ready

Output:

ready

Install Promtail on each app server

Promtail tails the local logs and pushes them to Loki over the network.

cd /tmp
curl -fSL -o promtail.zip "https://github.com/grafana/loki/releases/download/v3.4.2/promtail-linux-amd64.zip"
unzip promtail.zip
sudo mv promtail-linux-amd64 /usr/local/bin/promtail
sudo chmod +x /usr/local/bin/promtail
sudo mkdir -p /etc/promtail /var/lib/promtail

Create /etc/promtail/config.yaml. Replace 10.0.0.5 with your Loki server’s IP.

server:
  http_listen_port: 9080

positions:
  filename: /var/lib/promtail/positions.yaml

clients:
  - url: http://10.0.0.5:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets: [localhost]
        labels:
          job: varlogs
          host: web-01
          __path__: /var/log/{syslog,auth.log}

  - job_name: nginx
    static_configs:
      - targets: [localhost]
        labels:
          job: nginx
          host: web-01
          __path__: /var/log/nginx/*.log

The positions.filename is the cursor file mentioned earlier — it lets Promtail resume exactly where it stopped. The __path__ field is a glob (a wildcard file pattern) telling Promtail which files to tail.

Run it as a service with /etc/systemd/system/promtail.service:

[Unit]
Description=Promtail log shipper
After=network.target

[Service]
ExecStart=/usr/local/bin/promtail -config.file=/etc/promtail/config.yaml
Restart=on-failure
User=root

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now promtail
sudo systemctl status promtail --no-pager

Output:

● promtail.service - Promtail log shipper
     Loaded: loaded (/etc/systemd/system/promtail.service; enabled)
     Active: active (running) since Mon 2026-06-15 09:14:02 UTC; 4s ago
   Main PID: 4821 (promtail)

Open the firewall

Loki listens on port 3100. Allow only your app servers’ subnet, not the whole internet:

sudo ufw allow from 10.0.0.0/24 to any port 3100 proto tcp

Never expose port 3100 publicly with auth_enabled: false. Anyone who can reach it can read or inject logs. Keep Loki behind a private network or a reverse proxy with authentication.

A quick Filebeat alternative

If you are committed to Elasticsearch, install Filebeat from Elastic’s apt repository instead. The config in /etc/filebeat/filebeat.yml is conceptually the same — inputs, then an output:

filebeat.inputs:
  - type: filestream
    id: nginx-logs
    paths:
      - /var/log/nginx/*.log

output.elasticsearch:
  hosts: ["https://es-01.internal:9200"]
  username: "filebeat_writer"
  password: "${ES_PWD}"

sudo filebeat modules enable nginx
sudo systemctl enable --now filebeat

Best Practices

• Keep labels few and low-cardinality. In Loki, every unique combination of label values creates a separate stream. Use job, host, and env — never put a request ID or user ID in a label.
• Ship structured (JSON) logs when you can, so the central store can filter on real fields instead of regex over raw text.
• Always run the shipper as a systemd service with Restart=on-failure so it survives reboots and crashes.
• Confirm the registry/positions file persists on disk (not in /tmp) so no logs are re-sent or lost after a restart.
• Lock down the ingest port with ufw to your private subnet and enable authentication before any public exposure.
• Set retention on the central store (Loki’s limits_config or Elasticsearch ILM) so old logs are deleted automatically and disks never fill up.

Related Topics

• Centralized logging
• The ELK stack
• Structured logging
• Logging basics
• Log rotation