De Wiki inetshell
Saltar a: navegación, buscar

ELK

Instalar usando Docker

https://hub.docker.com/r/sebp/elk/

https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html

Elasticsearch

Cluster

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html

  • elasticsearch.yml
node:
  name: ${HOSTNAME}
  data: ${NODE_DATA:true}
  ingest: ${NODE_INGEST:true}
  master: ${NODE_MASTER:true}

cluster:
  name: elasticsearch
  remote.connect: false
  initial_master_nodes: ["elk-docker01"]

network:
  host: 0.0.0.0

discovery:
  zen:
    ping.unicast.hosts: ["elk-docker01","elk-docker02","elk-docker03"]
    minimum_master_nodes: 1

processors: ${PROCESSORS:1}

path:
  repo: /var/backups

Crear indice

PUT filebeat-2019.07
{
    "settings" : {
        "index" : {
            "number_of_shards" : 3, 
            "number_of_replicas" : 0 
        }
    }
}

Eliminar documentos de un indice

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html

POST /filebeat-*/_delete_by_query
{
  "query": {
    "match": {
      "message": "text"
    }
  }
}

Consultar estadisticas

https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster.html

# If no filters are given, the default is to select all nodes
GET /_nodes
# Explicitly select all nodes
GET /_nodes/_all
# Select just the local node
GET /_nodes/_local
# Select the elected master node
GET /_nodes/_master
# Select nodes by name, which can include wildcards
GET /_nodes/node_name_goes_here
GET /_nodes/node_name_goes_*
# Select nodes by address, which can include wildcards
GET /_nodes/10.0.0.3,10.0.0.4
GET /_nodes/10.0.0.*
# Select nodes by role
GET /_nodes/_all,master:false
GET /_nodes/data:true,ingest:true
GET /_nodes/coordinating_only:true
GET /_nodes/master:true,voting_only:false
# Select nodes by custom attribute (e.g. with something like `node.attr.rack: 2` in the configuration file)
GET /_nodes/rack:2
GET /_nodes/ra*:2
GET /_nodes/ra*:2*

Hacer merge a index

https://stackoverflow.com/questions/46153683/elasticsearch-how-to-merge-indexes-into-one-index

  • _forcemerge
curl -XPOST 'localhost:9200/logstash-2017.07*/_forcemerge?max_num_segments=1'
  • _reindex
curl -XPUT 'localhost:9200/new-logstash-2017.07.01?pretty' -H 'Content-Type: application/json' -d'
{
    "settings" : {
        "index" : {
            "number_of_shards" : 1 
        }
    }
}
'
  • Merging multiple indices and limiting number of shards
curl -XPOST 'localhost:9200/_reindex?pretty' -H 'Content-Type: application/json' -d'
{
    "conflicts": "proceed",
    "source": {
        "index": "logstash-2017.07*"
    },
    "dest": {
        "index": "logstash-2017.07",
        "op_type": "create"
    }
}
'

Performance

https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster

https://www.datadoghq.com/blog/elasticsearch-performance-scaling-problems/

https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration-memory.html

https://qbox.io/blog/optimizing-elasticsearch-how-many-shards-per-index

threadpool

https://www.elastic.co/guide/en/elasticsearch/reference/5.2/modules-threadpool.html

https://discuss.elastic.co/t/troubleshooting-high-load-avg-low-cpu-low-i-o-on-a-sizeable-production-cluster/40287

Generar templates de index

https://discuss.elastic.co/t/how-change-default-number-of-shards/117985

curl -XPUT  "$(hostname -I):9200/myindex/_settings?pretty" -H 'Content-Type: application/json' -d'
{
    "index" : {
        "number_of_shards" : 2, 
        "number_of_replicas" : 0 
    }
}
'

Reparar errores

"FORBIDDEN/12/index read-only / allow delete"

https://discuss.elastic.co/t/forbidden-12-index-read-only-allow-delete-api/110282/4

PUT _settings
{
  "index": {
    "blocks": {
      "read_only_allow_delete": "false"
    }
  }
}

Logstash

Definir zona horaria para logs de IIS

https://discuss.elastic.co/t/not-able-to-convert-iis-log-time-to-australian-time-zone/128875/3

https://discuss.elastic.co/t/gmt-timezone-to-cst-in-logstash/75424/6

date {
  match => ["log_timestamp", "yyyy-MM-dd HH:mm:ss"]
  timezone => "Etc/GMT"
}

Configurar indice para suricata

https://gist.github.com/elnappo/729cd7929dd5dae5be7eca6cd6014f5e

Plugins

https://blog.ip2location.com/knowledge-base/how-to-use-ip2proxy-filter-plugin-with-elastic-stack/

https://github.com/WedjaaOpen/logstash-filter-spamhaus

Kibana

Opciones de Kibana

https://www.elastic.co/guide/en/kibana/4.5/kibana-server-properties.html

Importar dashboard

https://stackoverflow.com/questions/47017583/how-to-import-export-a-dashboard-in-kibana-using-a-restful-api

curl -X POST -H "Content-Type: application/json" -H "kbn-xsrf: true" -d @export.json http://localhost:5601/api/kibana/dashboards/import

Filebeat

Configurar Filebeat con Docker

https://www.elastic.co/guide/en/beats/filebeat/current/add-docker-metadata.html

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata:
      host: "unix:///var/run/docker.sock"

Opciones de Filebeat

https://www.elastic.co/guide/en/beats/filebeat/current/how-filebeat-works.html

https://www.elastic.co/guide/en/beats/filebeat/6.2/configuration-filebeat-options.html

https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html#filebeat-input-log-close-options

Configurar filebeat en Windows

https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation.html

https://logit.io/sources/configure/iis

https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-module-iis.html

APM

Instalar APM Server

https://www.elastic.co/guide/en/apm/server/current/running-on-docker.html

https://www.elastic.co/guide/en/apm/server/current/apm-server-configuration.html

https://logz.io/blog/application-performance-monitoring/

Instalar APM Agent en Python

https://www.elastic.co/guide/en/apm/agent/python/1.x/configuration.html

https://www.elastic.co/guide/en/apm/agent/python/current/django-support.html

Instalar APM Agent en Node.js

https://www.elastic.co/guide/en/apm/agent/nodejs/current/express.html#express-initialization

https://blog.jmtalarn.com/express-app-and-a-ghost-blog-together/

https://www.elastic.co/guide/en/apm/agent/nodejs/master/configuration.html#active