From ab310288ef18778af0bc97a4abb34a4f20350617 Mon Sep 17 00:00:00 2001 From: Febbweiss Date: Wed, 24 May 2017 15:05:33 +0200 Subject: [PATCH] Feature: update README.md --- README.md | 27 +++++++++++++++------------ 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index 813cc05..469b56e 100644 --- a/README.md +++ b/README.md @@ -2,25 +2,28 @@ This a end-to-end log centralizer powered by the ELK stask. Embedded containers - - 1. [Filebeat](https://www.elastic.co/products/beats/filebeat) - An agent to poll logs - 2. [Logstash-Forwarder](https://github.com/elastic/logstash-forwarder) - An other agent to poll logs - 3. [Logstash](https://www.elastic.co/products/logstash) - The collector / analyzer / parser solution - 4. [Kafka](http://kafka.apache.org) - The queueing solution for logs - 5. [ZooKeeper](https://zookeeper.apache.org/) - The cluster on which Kafka is running - 6. [ElasticSearch](https://www.elastic.co/products/elasticsearch) - The indexing engine - 7. [Kibana](https://www.elastic.co/products/kibana) - The visualization / dashboard tool for ElasticSearch - 8. [Kafka Manager](https://github.com/yahoo/kafka-manager) - The Kafka cluster web manager - 9. [Apache log generator](https://github.com/Febbweiss/docker-apache-log-generator) - A container generating fake apache logs - 10. [Random log generator](https://hub.docker.com/r/davidmccormick/random_log_generator) - A container genrating text logs (Star Wars quotes) + * [Filebeat](https://www.elastic.co/products/beats/filebeat) - An agent to poll logs + * [Logstash-Forwarder](https://github.com/elastic/logstash-forwarder) - An other agent to poll logs + * [rsyslog](http://www.rsyslog.com/) - A standard Linux log manager + * [Logstash](https://www.elastic.co/products/logstash) - The collector / analyzer / parser solution + * [Kafka](http://kafka.apache.org) - The queueing solution for logs + * [ZooKeeper](https://zookeeper.apache.org/) - The cluster on which Kafka is running + * [ElasticSearch](https://www.elastic.co/products/elasticsearch) - The indexing engine + * [Kibana](https://www.elastic.co/products/kibana) - The visualization / dashboard tool for ElasticSearch + * [Kafka Manager](https://github.com/yahoo/kafka-manager) - The Kafka cluster web manager + * [Apache log generator](https://github.com/Febbweiss/docker-apache-log-generator) - A container generating fake apache logs + * [Random log generator](https://hub.docker.com/r/davidmccormick/random_log_generator) - A container generating text logs (Star Wars quotes) + * [Java log generator](https://github.com/Febbweiss/docker-java-log-generator) - A container generating Java logs (with exception stack trace) How it works - -There are 2 agent types : +There are 3 agent types : - Filebeat - Logstash-Forward + - rsyslog -These agents push logs (from the apache and random generators) to a Logstasth shipper filling a Kafka queue (one type of log for one topic). +These agents push logs (from the generators) to a Logstasth shipper filling a Kafka queue (one type of log for one topic). A Logstash indexer polls the Kafka topics indexing logs into a ElasticSearch. A short schema :