Logging Tutorial: ELK Setup with GELF and Filebeat

This project demonstrates two approaches to setting up centralized logging using the ELK (Elasticsearch, Logstash, Kibana) stack. The two setups are organized into separate folders:

1. test1-gelf/: Uses the GELF (Graylog Extended Log Format) logging driver for Docker containers.
2. test2-filebeat/: Uses Filebeat to ship logs from applications to Logstash.

---

Folder Structure

test1-gelf/

This folder contains the setup for logging using the GELF driver. The key components are:

• docker-compose.yml: Defines services for Elasticsearch, Kibana, Logstash, and an Express application (app1).
• logstash/logstash.conf: Configures Logstash to receive logs via the GELF input plugin and forward them to Elasticsearch.
• app1/: A simple Express application that logs messages to the console. Docker captures these logs and sends them to Logstash using the GELF driver.

test2-filebeat/

This folder contains the setup for logging using Filebeat. The key components are:

• docker-compose.yml: Defines services for Elasticsearch, Kibana, Logstash, Filebeat, and two Express applications (app1 and app2).
• logstash/logstash.conf: Configures Logstash to receive logs from Filebeat and parse them using Grok patterns.
• filebeat/: Contains the Filebeat configuration (filebeat.yml) and Dockerfile for building the Filebeat container.
• app1/ and app2/: Simple Express applications that log messages to both the console and a file (app.log). Filebeat reads these log files and forwards them to Logstash.

---

How to Run

Prerequisites

• Docker and Docker Compose installed on your system.

Running test1-gelf/

1. Navigate to the test1-gelf/ directory:

 cd test1-gelf

2. Start the services:

docker compose up

3. Access Kibana at http://localhost:5601 to view logs.

Running test2-filebeat/

1. Navigate to the test2-filebeat/ directory:

 cd test2-filebeat

2. Start the services:

docker compose up

Access Kibana at http://localhost:5601 to view logs.

Key Differences

Feature	test1-gelf/	test2-filebeat/
Log Shipping	Docker GELF driver	Filebeat
Log Sources	Console logs from Docker containers	Log files generated by applications
Flexibility	Limited to Docker GELF driver options	Highly customizable with Filebeat and Logstash

Notes

1. The test1-gelf/ setup is simpler but less flexible, as it relies on Docker's GELF driver.
2. The test2-filebeat/ setup provides more control over log collection and parsing, making it suitable for complex logging requirements.