ELK
Instalar usando Docker
https://hub.docker.com/r/sebp/elk/
https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html
Elasticsearch
Cluster
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html
- elasticsearch.yml
node: name: ${HOSTNAME} data: ${NODE_DATA:true} ingest: ${NODE_INGEST:true} master: ${NODE_MASTER:true} cluster: name: elasticsearch remote.connect: false initial_master_nodes: ["elk-docker01"] network: host: 0.0.0.0 discovery: zen: ping.unicast.hosts: ["elk-docker01","elk-docker02","elk-docker03"] minimum_master_nodes: 1 processors: ${PROCESSORS:1} path: repo: /var/backups
Crear indice
PUT filebeat-2019.07 { "settings" : { "index" : { "number_of_shards" : 3, "number_of_replicas" : 0 } } }
Eliminar documentos de un indice
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html
POST /filebeat-*/_delete_by_query { "query": { "match": { "message": "text" } } }
Consultar estadisticas
https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster.html
# If no filters are given, the default is to select all nodes GET /_nodes # Explicitly select all nodes GET /_nodes/_all # Select just the local node GET /_nodes/_local # Select the elected master node GET /_nodes/_master # Select nodes by name, which can include wildcards GET /_nodes/node_name_goes_here GET /_nodes/node_name_goes_* # Select nodes by address, which can include wildcards GET /_nodes/10.0.0.3,10.0.0.4 GET /_nodes/10.0.0.* # Select nodes by role GET /_nodes/_all,master:false GET /_nodes/data:true,ingest:true GET /_nodes/coordinating_only:true GET /_nodes/master:true,voting_only:false # Select nodes by custom attribute (e.g. with something like `node.attr.rack: 2` in the configuration file) GET /_nodes/rack:2 GET /_nodes/ra*:2 GET /_nodes/ra*:2*
Hacer merge a index
https://stackoverflow.com/questions/46153683/elasticsearch-how-to-merge-indexes-into-one-index
- _forcemerge
curl -XPOST 'localhost:9200/logstash-2017.07*/_forcemerge?max_num_segments=1'
- _reindex
curl -XPUT 'localhost:9200/new-logstash-2017.07.01?pretty' -H 'Content-Type: application/json' -d' { "settings" : { "index" : { "number_of_shards" : 1 } } } '
- Merging multiple indices and limiting number of shards
curl -XPOST 'localhost:9200/_reindex?pretty' -H 'Content-Type: application/json' -d' { "conflicts": "proceed", "source": { "index": "logstash-2017.07*" }, "dest": { "index": "logstash-2017.07", "op_type": "create" } } '
Performance
https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster
https://www.datadoghq.com/blog/elasticsearch-performance-scaling-problems/
https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration-memory.html
https://qbox.io/blog/optimizing-elasticsearch-how-many-shards-per-index
threadpool
https://www.elastic.co/guide/en/elasticsearch/reference/5.2/modules-threadpool.html
Generar templates de index
https://discuss.elastic.co/t/how-change-default-number-of-shards/117985
curl -XPUT "$(hostname -I):9200/myindex/_settings?pretty" -H 'Content-Type: application/json' -d' { "index" : { "number_of_shards" : 2, "number_of_replicas" : 0 } } '
Reparar errores
"FORBIDDEN/12/index read-only / allow delete"
https://discuss.elastic.co/t/forbidden-12-index-read-only-allow-delete-api/110282/4
PUT _settings { "index": { "blocks": { "read_only_allow_delete": "false" } } }
Logstash
Definir zona horaria para logs de IIS
https://discuss.elastic.co/t/not-able-to-convert-iis-log-time-to-australian-time-zone/128875/3
https://discuss.elastic.co/t/gmt-timezone-to-cst-in-logstash/75424/6
date { match => ["log_timestamp", "yyyy-MM-dd HH:mm:ss"] timezone => "Etc/GMT" }
Configurar indice para suricata
https://gist.github.com/elnappo/729cd7929dd5dae5be7eca6cd6014f5e
Plugins
https://blog.ip2location.com/knowledge-base/how-to-use-ip2proxy-filter-plugin-with-elastic-stack/
https://github.com/WedjaaOpen/logstash-filter-spamhaus
Kibana
Opciones de Kibana
https://www.elastic.co/guide/en/kibana/4.5/kibana-server-properties.html
Importar dashboard
curl -X POST -H "Content-Type: application/json" -H "kbn-xsrf: true" -d @export.json http://localhost:5601/api/kibana/dashboards/import
Filebeat
Configurar Filebeat con Docker
https://www.elastic.co/guide/en/beats/filebeat/current/add-docker-metadata.html
processors: - add_host_metadata: ~ - add_cloud_metadata: ~ - add_docker_metadata: host: "unix:///var/run/docker.sock"
Opciones de Filebeat
https://www.elastic.co/guide/en/beats/filebeat/current/how-filebeat-works.html
https://www.elastic.co/guide/en/beats/filebeat/6.2/configuration-filebeat-options.html
Configurar filebeat en Windows
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation.html
https://logit.io/sources/configure/iis
https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-module-iis.html
APM
Instalar APM Server
https://www.elastic.co/guide/en/apm/server/current/running-on-docker.html
https://www.elastic.co/guide/en/apm/server/current/apm-server-configuration.html
https://logz.io/blog/application-performance-monitoring/
Instalar APM Agent en Python
https://www.elastic.co/guide/en/apm/agent/python/1.x/configuration.html
https://www.elastic.co/guide/en/apm/agent/python/current/django-support.html
Instalar APM Agent en Node.js
https://www.elastic.co/guide/en/apm/agent/nodejs/current/express.html#express-initialization
https://blog.jmtalarn.com/express-app-and-a-ghost-blog-together/
https://www.elastic.co/guide/en/apm/agent/nodejs/master/configuration.html#active