ELK Stack (Podman + Ansible)

Fully automated deployment of an ELK Stack (Elasticsearch, Logstash, Kibana) with Metricbeat host monitoring, running as rootless Podman containers and managed by Ansible playbooks.

Quick start

ansible-galaxy collection install -r requirements.yml
cp .env.example .env          # then edit passwords
ansible-playbook init.yml     # deploy the full stack
ansible-playbook test.yml     # verify everything works

Kibana: http://localhost:5601 — log in as elastic with your ELASTIC_PASSWORD.

Architecture

┌──────────────────────────────────────────────────────┐
│  Podman network: elk-net                             │
│                                                      │
│  ┌──────────────────┐                                │
│  │ Elasticsearch 8.x │◄──────────────────┐           │
│  │ :9200             │◄────────────┐     │           │
│  └──────▲───────────┘             │     │           │
│         │                          │     │           │
│  ┌──────┴───────────┐   ┌────────┴─────┴─────────┐ │
│  │ Kibana            │   │ Logstash                │ │
│  │ :5601             │   │ :5044 (Beats)           │ │
│  │                   │   │ :5000 (TCP/JSON)        │ │
│  └──────────────────┘   └────────────────────────┘ │
│                                                      │
│  ┌──────────────────────────────────────────────┐   │
│  │ Metricbeat                                    │   │
│  │ system metrics → Elasticsearch                │   │
│  │ mounts: /proc, /sys, / (read-only)            │   │
│  └──────────────────────────────────────────────┘   │
└──────────────────────────────────────────────────────┘

Containers

• elk-elasticsearch — Elasticsearch 8.x single-node cluster with basic security (username/password auth, no TLS)
• elk-kibana — Kibana web UI for search, visualization, and dashboards
• elk-logstash — Logstash data processing pipeline (Beats input on 5044, TCP/JSON input on 5000)
• elk-metricbeat — Metricbeat agent for local machine monitoring (CPU, memory, disk, network, processes)

Post-deployment configuration

The playbook automatically:

• Sets the kibana_system internal user password via the Elasticsearch Security API
• Creates a logstash-* data view in Kibana for exploring ingested data
• Imports Metricbeat system dashboards into Kibana

Prerequisites

• Podman installed with a running machine (podman machine start on macOS)
• Ansible >= 2.14
• containers.podman Ansible collection

ansible-galaxy collection install -r requirements.yml

Deployment

1. Configure credentials

cp .env.example .env

Edit .env and set your passwords:

ELASTIC_PASSWORD=<your-elasticsearch-password>
KIBANA_SYSTEM_PASSWORD=<your-kibana-system-password>

The .env file is gitignored and never committed. Only .env.example (with placeholder values) is tracked.

2. Run the playbook

ansible-playbook init.yml

This will:

1. Create the elk-net Podman network and a persistent Elasticsearch data volume
2. Start Elasticsearch and wait for cluster health
3. Set the kibana_system user password via the Security API
4. Start Kibana and Logstash, waiting for each to become ready
5. Start Metricbeat and import system dashboards into Kibana
6. Create a logstash-* data view in Kibana

Subsequent runs

The playbook is idempotent — re-running it applies only what has changed.

To force-recreate the Elasticsearch container (e.g. after changing JVM settings):

ansible-playbook init.yml -e es_recreate=true

Start / stop

ansible-playbook stop.yml     # stop all containers (reverse dependency order)
ansible-playbook start.yml    # start all containers (dependency order + readiness checks)

Verify

ansible-playbook test.yml

This runs health checks on all four containers, verifies the Elasticsearch cluster health, Kibana API, Logstash monitoring endpoint, Metricbeat data shipping, and performs a test document index/delete cycle.

Access

Service	URL / Port
Kibana	http://localhost:5601
Elasticsearch	http://localhost:9200
Logstash Beats	localhost:5044
Logstash TCP	localhost:5000

Credentials: elastic / <ELASTIC_PASSWORD from .env>

Note: Log into Kibana with the elastic superuser, not kibana_system. The kibana_system account is an internal user used by Kibana to communicate with Elasticsearch and cannot be used for UI login.

Monitoring dashboards

Metricbeat ships system metrics (CPU, memory, disk, network, processes) every 10 seconds and pre-built dashboards are imported automatically during init.yml.

To view them:

1. Open Kibana at http://localhost:5601
2. Log in with elastic / your ELASTIC_PASSWORD
3. Go to Analytics → Dashboards
4. Search for "system" — key dashboards include:
   • [Metricbeat System] Host overview ECS — CPU, memory, disk, network at a glance
   • [Metricbeat System] Overview ECS — fleet-wide summary

Sending data to Logstash

Via TCP (JSON)

echo '{"message": "hello from ELK sandbox"}' | nc localhost 5000

Via Filebeat

Point Filebeat's output to localhost:5044:

output.logstash:
  hosts: ["localhost:5044"]

Directly to Elasticsearch

curl -u elastic:$ELASTIC_PASSWORD -X POST http://localhost:9200/my-index/_doc \
  -H 'Content-Type: application/json' \
  -d '{"@timestamp": "2024-01-01T00:00:00Z", "message": "direct ingest"}'

Exploring data in Kibana

Once data is flowing, go to Analytics → Discover in Kibana. Select a data view (logstash-* or metricbeat-*) and use the time picker and KQL search bar to explore your data.

Configuration

All tunables live in group_vars/all.yml:

Images:

• elasticsearch_version — Elasticsearch/Logstash/Kibana version (default: 8.17.0)
• Images are pulled from docker.elastic.co

JVM:

• elasticsearch_heap_size — Elasticsearch JVM heap (default: 512m)
• logstash_heap_size — Logstash JVM heap (default: 256m)

Networking:

• elasticsearch_port — host port for Elasticsearch API (default: 9200)
• kibana_port — host port for Kibana web UI (default: 5601)
• logstash_beats_port — host port for Beats input (default: 5044)
• logstash_tcp_port — host port for TCP/JSON input (default: 5000)
• logstash_monitoring_port — host port for Logstash monitoring API (default: 9600)

Other:

• elasticsearch_cluster_name — cluster name (default: elk-sandbox)
• elk_timezone — timezone (default: Europe/Berlin)

Metricbeat

The Metricbeat configuration is in configs/metricbeat/metricbeat.yml and is bind-mounted into the container. It collects system metrics every 10 seconds: cpu, load, memory, network, process, process_summary, socket_summary, filesystem, and diskio.

The container mounts /proc, /sys, and / as read-only from the host to access real system metrics from inside the container.

Logstash pipeline

The Logstash pipeline configuration is in configs/logstash/pipeline.conf and is bind-mounted into the container. Edit it and restart Logstash to apply changes:

ansible-playbook stop.yml && ansible-playbook start.yml

Project structure

.
├── .env                   # Secrets (gitignored)
├── .env.example           # Secrets template (committed)
├── .gitignore
├── ansible.cfg            # Ansible settings
├── requirements.yml       # Ansible Galaxy dependencies
├── inventory.yml          # Localhost with local connection
├── group_vars/
│   └── all.yml            # Non-sensitive configuration
├── tasks/
│   └── load_env.yml       # Shared .env file parser
├── configs/
│   ├── logstash/
│   │   └── pipeline.conf  # Logstash pipeline configuration
│   └── metricbeat/
│       └── metricbeat.yml # Metricbeat system module configuration
├── init.yml               # Full deploy (containers + API setup)
├── start.yml              # Start all containers
├── stop.yml               # Stop all containers
├── test.yml               # Verify the stack
└── README.md

Teardown

Stop containers:

ansible-playbook stop.yml

To fully remove containers, the network, and the data volume:

podman rm elk-metricbeat elk-logstash elk-kibana elk-elasticsearch
podman network rm elk-net
podman volume rm elk-elasticsearch-data

Troubleshooting

Kibana login fails with changeme123 or similar password: Log in as elastic (not kibana_system). Verify the password works against Elasticsearch directly: curl -u elastic:<password> http://localhost:9200

Elasticsearch cluster status is yellow: This is normal for a single-node cluster — yellow means all primary shards are assigned but replicas have nowhere to go. It does not indicate a problem.

Metricbeat not shipping data: Check the container logs: podman logs elk-metricbeat. Common issues include permission errors on /proc or /sys mounts, or Elasticsearch connection failures.

Container fails to start after config change: Remove the old container first, then re-run init:

podman rm -f <container-name>
ansible-playbook init.yml

Port conflict: If a port is already in use, change it in group_vars/all.yml and re-run ansible-playbook init.yml.