From 63a33b97e90c5fc89b31516b717a602a90855be6 Mon Sep 17 00:00:00 2001 From: Febbweiss Date: Fri, 17 Mar 2017 15:59:24 +0100 Subject: [PATCH] Feature: add README --- README.md | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..0e366c3 --- /dev/null +++ b/README.md @@ -0,0 +1,34 @@ +This a end-to-end log centralizer powered by the ELK stask. + +Embedded containers +- + 1. [Filebeat](https://www.elastic.co/products/beats/filebeat) - An agent to poll logs + 2. [Logstash-Forwarder](https://github.com/elastic/logstash-forwarder) - An other agent to poll logs + 3. [Logstash](https://www.elastic.co/products/logstash) - The collector / analyzer / parser solution + 4. [Kafka](http://kafka.apache.org) - The queueing solution for logs + 5. [ZooKeeper](https://zookeeper.apache.org/) - The cluster on which Kafka is running + 6. [ElasticSearch](https://www.elastic.co/products/elasticsearch) - The indexing engine + 7. [Kibana](https://www.elastic.co/products/kibana) - The visualization / dashboard tool for ElasticSearch + 8. [Kafka Manager](https://github.com/yahoo/kafka-manager) - The Kafka cluster web manager + +How it works +- +There are 2 agent types : + + - Filebeat + - Logstash-Forward + +These agents push logs to a Logstasth shipper filling a Kafka queue (one type of log for one topic). +A Logstash indexer polls the Kafka topics indexing logs into a ElasticSearch. + +A short schema : +``` +Agent -> Logstach shipper -> Kafka <- Logstash indexer -> ElasticSearch +``` + +Tools access +- +Kibana is available at http://localhost:5601. +Kafka Manager is available at http://localhost:9000 + +