ELK - Installation ELK (Elasticsearch Logstash Kibana)
Alasta 8 Septembre 2014 bigdata Apache bash BigData CentOS cli Linux monitoring Open Source
Description : Voici comment installer la suite de logiciel open source Elasticsearch Logstash Kibana qui permet de faire de magnifique dashboard et des recherches dans le "Big-Data". C'est une excellente alternative au produit commercial Splunk ou Loggly (qui est dans le cloud).
Présentation :
Cette suite (il y a aussi d'autres logiciels qui peuvent s'y greffer comme par exemple redis, flume) se décompose comme cela :
- Elasticsearch : l'indexer (Base de données en NoSQL)
- Logstash : Parser qui va permettre d'alimenter Elasticsearch
- Kibana : la WebUI qui permet de générer des dashboards, recherches de donnée sur Elasticsearch
Contexte :
Nous allons installer cette solution à partir d'une CentOS 6.5 avec SELinux en permissif.
Pour l'installation des packages il y a 2 possibilités les RPMs ou les repositories.
Les fondations :
Le pré-requis minimum sont un serveur Web et Java.
Apache
yum install httpd chkconfig --level 2345 httpd on service httpd start
Java
yum install java-1.7.0-openjdk
Elasticsearch :
RPM
cd /usr/src wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.3.2.noarch.rpm yum localinstall elasticsearch-1.3.2.noarch.rpm #préférer yum localinstall à rpm -ivh car il n'y a pas de corruption de la DB yum chkconfig --level 2345 elasticsearch on service elasticsearch start
Repository
rpm --import http://packages.elasticsearch.org/GPG-KEY-elasticsearch
[elasticsearch-1.3] name=Elasticsearch repository for 1.3.x packages baseurl=http://packages.elasticsearch.org/elasticsearch/1.3/centos gpgcheck=1 gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch enabled=1
yum install elasticsearch chkconfig --level 2345 elasticsearch on service elasticsearch start
Test de bon fonctionnement
curl http://127.0.0.1:9200
{
"status" : 200,
"name" : "John Falsworth",
"version" : {
"number" : "1.3.2",
"build_hash" : "dee175dbe2f254f3f26992f5d7591939aaefd12f",
"build_timestamp" : "2014-08-13T14:29:30Z",
"build_snapshot" : false,
"lucene_version" : "4.9"
},
"tagline" : "You Know, for Search"
}
Logstash :
RPM
wget https://download.elasticsearch.org/logstash/logstash/packages/centos/logstash-1.4.2-1_2c0f5a1.noarch.rpm wget https://download.elasticsearch.org/logstash/logstash/packages/centos/logstash-contrib-1.4.2-1_efd53ef.noarch.rpm yum localinstall logstash*
Repository
[logstash-1.4] name=logstash repository for 1.4.x packages baseurl=http://packages.elasticsearch.org/logstash/1.4/centos gpgcheck=1 gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch enabled=1
yum -y install logstash*
Configuration de base :
Notre fichier de configuration de base permettra de rediriger les messages syslog dans Elasticsearch.
input {
#Ouverture en entrée d'un port d'écoute utilisant le protocol syslog
tcp {
port => 5544
type => syslog
}
udp {
port => 5544
type => syslog
}
}
filter {
# Traitement type syslog, le type étant marqué sur les données entrant par nos ports de type syslog
if [type] == "syslog" {
grok {
# Si on ne veut pas garder le message non traité
overwrite => "message"
match => {
# rsyslong envoi des messages de type : <Numero>Ligne Syslog avec le message
"message" => "^(?:<%{NONNEGINT:syslog_pri}>)?%{SYSLOGBASE2} %{GREEDYDATA:message}"
"message" => "%{SYSLOG5424PRI}%{SYSLOGBASE2} %{DATA:message}"
}
# on ajoute des tags perso, pratique pour filtrer dans l'interface kibana
add_tag => [ "syslog", "grokked" ]
}
}
}
output {
# on stock dans elasticsearch
elasticsearch {
host => "localhost"
}
}
Finalisation de Logstash
chkconfig --level 2345 logstash on service logstash start
Rsyslog
Voici un fichier de configuration pour accépter les messages syslog du réseau et forwarder à logstash ces messages
# rsyslog v5 configuration file # For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html # If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html #### MODULES #### $ModLoad imuxsock # provides support for local system logging (e.g. via logger command) $ModLoad imklog # provides kernel logging support (previously done by rklogd) #$ModLoad immark # provides --MARK-- message capability # Provides UDP syslog reception #on accepte les messages en UDP $ModLoad imudp $UDPServerRun 514 # Provides TCP syslog reception #on accepte les messages en TCP $ModLoad imtcp $InputTCPServerRun 514 #### GLOBAL DIRECTIVES #### # Use default timestamp format $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # File syncing capability is disabled by default. This feature is usually not required, # not useful and an extreme performance hit #$ActionFileEnableSync on # Include all config files in /etc/rsyslog.d/ $IncludeConfig /etc/rsyslog.d/*.conf #### RULES #### # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none /var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg * # Save news errors of level crit and higher in a special file. uucp,news.crit /var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log # ### begin forwarding rule ### # The statement between the begin ... end define a SINGLE forwarding # rule. They belong together, do NOT split them. If you create multiple # forwarding rules, duplicate the whole block! # Remote Logging (we use TCP for reliable delivery) # # An on-disk queue is created for this action. If the remote host is # down, messages are spooled to disk and sent when it is up again. #$WorkDirectory /var/lib/rsyslog # where to place spool files #$ActionQueueFileName fwdRule1 # unique name prefix for spool files #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown #$ActionQueueType LinkedList # run asynchronously #$ActionResumeRetryCount -1 # infinite retries if host is down # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional #*.* @@remote-host:514 #Forward a logstash *.* @@127.0.0.1:5544 # ### end of the forwarding rule ###
service rsyslog restart
Kibana
cd /var/www/html wget https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.tar.gz tar xzvf kibana-3.1.0.tar.gz chown -R apache:apache /var/www/html/kibana-3.1.0
Il faut savoir qu'Elasticsearch utilise le port 9200, donc deux solutions s'offre à nous, utiliser le port standard 9200 mais il faut que le flux soit ouvert à travers les différents du réseau ou utiliser les directives de mod_proxy qui va permettre de proxifier via le port 9200 via le port 80 et/ou 443 ce qui permet via HTTPS de sécuriser le flux.
Configuration Apache avec mod_proxy et config kibana
<LocationMatch "^/+$">
Options -Indexes
ErrorDocument 403 /error/noindex.html
</LocationMatch>
ProxyPass /my_elasticsearch http://127.0.0.1:9200/
ProxyPassReverse /my_elasticsearch http://127.0.0.1:9200/
service httpd graceful
/** @scratch /configuration/config.js/1
*
* == Configuration
* config.js is where you will find the core Kibana configuration. This file contains parameter that
* must be set before kibana is run for the first time.
*/
define(['settings'],
function (Settings) {
/** @scratch /configuration/config.js/2
*
* === Parameters
*/
return new Settings({
/** @scratch /configuration/config.js/5
*
* ==== elasticsearch
*
* The URL to your elasticsearch server. You almost certainly don't
* want +http://localhost:9200+ here. Even if Kibana and Elasticsearch are on
* the same host. By default this will attempt to reach ES at the same host you have
* kibana installed on. You probably want to set it to the FQDN of your
* elasticsearch host
*
* Note: this can also be an object if you want to pass options to the http client. For example:
*
* +elasticsearch: {server: "http://localhost:9200", withCredentials: true}+
*
*/
/**elasticsearch: "http://"+window.location.hostname+":9200",**/
elasticsearch: "//"+window.location.hostname+"/my_elasticsearch",
/** @scratch /configuration/config.js/5
*
* ==== default_route
*
* This is the default landing page when you don't specify a dashboard to load. You can specify
* files, scripts or saved dashboards here. For example, if you had saved a dashboard called
* `WebLogs' to elasticsearch you might use:
*
* default_route: '/dashboard/elasticsearch/WebLogs',
*/
default_route : '/dashboard/file/default.json',
/** @scratch /configuration/config.js/5
*
* ==== kibana-int
*
* The default ES index to use for storing Kibana specific object
* such as stored dashboards
*/
kibana_index: "kibana-int",
/** @scratch /configuration/config.js/5
*
* ==== panel_name
*
* An array of panel modules available. Panels will only be loaded when they are defined in the
* dashboard, but this list is used in the "add panel" interface.
*/
panel_names: [
'histogram',
'map',
'goal',
'table',
'filtering',
'timepicker',
'text',
'hits',
'column',
'trends',
'bettermap',
'query',
'terms',
'stats',
'sparklines'
]
});
});
Test de kibana :
Dans un shell du serveur taper la commande suivante :
logger "Test syslog pour kibana"
Cette commande va envoyer un texte au syslog local du serveur et l'on pourra vérifier dans kibana si cela s'affiche bien (via *Test* dans le champs Query en haut du dashboard).
Utilisez son navigateur et aller à l'adresse http://_Adresse_IP_du_serveur_/kibana-3.1.0/ et cliquer sur le lien "Sample Dashboard"
Il devrait apparaître dans la partie basse les messages syslog.
Voici le résultat :
Un prochain article traitera d'autres sujets sur ELK.
Bonus :
Les packages sur ma ressource ICI en cas de besoins ou sur GitHub.
Logs Elasticsearch : /var/log/elasticsearch/elasticsearch.log
Logs Logstash : /var/log/logstash/logstash.log