ELK
Instalar usando Docker
https://hub.docker.com/r/sebp/elk/
https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html
Elasticsearch
Cluster
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html
- elasticsearch.yml
node:
name: ${HOSTNAME}
data: ${NODE_DATA:true}
ingest: ${NODE_INGEST:true}
master: ${NODE_MASTER:true}
cluster:
name: elasticsearch
remote.connect: false
initial_master_nodes: ["elk-docker01"]
network:
host: 0.0.0.0
discovery:
zen:
ping.unicast.hosts: ["elk-docker01","elk-docker02","elk-docker03"]
minimum_master_nodes: 1
processors: ${PROCESSORS:1}
path:
repo: /var/backups
Crear indice
PUT filebeat-2019.07
{
"settings" : {
"index" : {
"number_of_shards" : 3,
"number_of_replicas" : 0
}
}
}
Eliminar documentos de un indice
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html
POST /filebeat-*/_delete_by_query
{
"query": {
"match": {
"message": "text"
}
}
}
Consultar estadisticas
https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster.html
# If no filters are given, the default is to select all nodes GET /_nodes # Explicitly select all nodes GET /_nodes/_all # Select just the local node GET /_nodes/_local # Select the elected master node GET /_nodes/_master # Select nodes by name, which can include wildcards GET /_nodes/node_name_goes_here GET /_nodes/node_name_goes_* # Select nodes by address, which can include wildcards GET /_nodes/10.0.0.3,10.0.0.4 GET /_nodes/10.0.0.* # Select nodes by role GET /_nodes/_all,master:false GET /_nodes/data:true,ingest:true GET /_nodes/coordinating_only:true GET /_nodes/master:true,voting_only:false # Select nodes by custom attribute (e.g. with something like `node.attr.rack: 2` in the configuration file) GET /_nodes/rack:2 GET /_nodes/ra*:2 GET /_nodes/ra*:2*
Hacer merge a index
https://stackoverflow.com/questions/46153683/elasticsearch-how-to-merge-indexes-into-one-index
- _forcemerge
curl -XPOST 'localhost:9200/logstash-2017.07*/_forcemerge?max_num_segments=1'
- _reindex
curl -XPUT 'localhost:9200/new-logstash-2017.07.01?pretty' -H 'Content-Type: application/json' -d'
{
"settings" : {
"index" : {
"number_of_shards" : 1
}
}
}
'
- Merging multiple indices and limiting number of shards
curl -XPOST 'localhost:9200/_reindex?pretty' -H 'Content-Type: application/json' -d'
{
"conflicts": "proceed",
"source": {
"index": "logstash-2017.07*"
},
"dest": {
"index": "logstash-2017.07",
"op_type": "create"
}
}
'
Performance
https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster
https://www.datadoghq.com/blog/elasticsearch-performance-scaling-problems/
https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration-memory.html
https://qbox.io/blog/optimizing-elasticsearch-how-many-shards-per-index
threadpool
https://www.elastic.co/guide/en/elasticsearch/reference/5.2/modules-threadpool.html
Generar templates de index
https://discuss.elastic.co/t/how-change-default-number-of-shards/117985
curl -XPUT "$(hostname -I):9200/myindex/_settings?pretty" -H 'Content-Type: application/json' -d'
{
"index" : {
"number_of_shards" : 2,
"number_of_replicas" : 0
}
}
'
Logstash
Definir zona horaria para logs de IIS
https://discuss.elastic.co/t/not-able-to-convert-iis-log-time-to-australian-time-zone/128875/3
https://discuss.elastic.co/t/gmt-timezone-to-cst-in-logstash/75424/6
date {
match => ["log_timestamp", "yyyy-MM-dd HH:mm:ss"]
timezone => "Etc/GMT"
}
Configurar indice para suricata
https://gist.github.com/elnappo/729cd7929dd5dae5be7eca6cd6014f5e
Plugins
https://blog.ip2location.com/knowledge-base/how-to-use-ip2proxy-filter-plugin-with-elastic-stack/
https://github.com/WedjaaOpen/logstash-filter-spamhaus
Kibana
Opciones de Kibana
https://www.elastic.co/guide/en/kibana/4.5/kibana-server-properties.html
Importar dashboard
curl -X POST -H "Content-Type: application/json" -H "kbn-xsrf: true" -d @export.json http://localhost:5601/api/kibana/dashboards/import
Filebeat
Configurar Filebeat con Docker
https://www.elastic.co/guide/en/beats/filebeat/current/add-docker-metadata.html
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata:
host: "unix:///var/run/docker.sock"
Opciones de Filebeat
https://www.elastic.co/guide/en/beats/filebeat/current/how-filebeat-works.html
https://www.elastic.co/guide/en/beats/filebeat/6.2/configuration-filebeat-options.html
Configurar filebeat en Windows
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation.html
https://logit.io/sources/configure/iis
https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-module-iis.html
APM
Instalar APM Server
https://www.elastic.co/guide/en/apm/server/current/running-on-docker.html
https://www.elastic.co/guide/en/apm/server/current/apm-server-configuration.html
https://logz.io/blog/application-performance-monitoring/
Instalar APM Agent en Python
https://www.elastic.co/guide/en/apm/agent/python/1.x/configuration.html
https://www.elastic.co/guide/en/apm/agent/python/current/django-support.html
Instalar APM Agent en Node.js
https://www.elastic.co/guide/en/apm/agent/nodejs/current/express.html#express-initialization
https://blog.jmtalarn.com/express-app-and-a-ghost-blog-together/
https://www.elastic.co/guide/en/apm/agent/nodejs/master/configuration.html#active