Untitled: White board for stacking my life

Curator is tool for managing ES indices.
큐레이터는 엘라스틱서치 인덱스 관리 툴이다.

Installation and configuration

You need to check your Elasticsearch version compatible with curator version.

운영중인 엘라스틱 서치 버전과 큐레이터 버전을 맞춰 주어야 한다.

설치는 pip를 이용하며, 설정 파일은 yml 형식으로 작성한다.

# in the ES master node, ES version 6+
$ pip install -U elasticsearch-curator==5.8.1

# create curator config file
# logging is optional
$ vi ~/.curator/curator.yml
---
client:
  hosts:
    - [ES_IP]
  port: [ES_PORT]
logging:
  loglevel: INFO
  logfile: /[PATH_TO_LOGFILE]/curator.log
  logformat: default

# create curator action yaml file
$ vi delete-old-indices.yml
---

actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 1 months (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      disable_action: False
    filters:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: months
      unit_count: 1

CLI execution

# execute curator command (--dry-run: verbose and execute action without changes)
curator --dry-run delete-old-indices.yml

Filebeat에서 Logstash로 데이터 전송 시 ssl 설정

Create instance file for elasticsearch-certutil

logstash 노드에서 elasticsearch-certutil 명령어와 instance 파일을 사용하여 p12 파일 생성

# vi instance.yml

instances:
  - name: "logstash"
    ip:
      - "192.168.1.2"

Create p12 cert file from elasticsearch-certutil and instance.yml

# /usr/share/elasticsearch/bin/elasticsearch-certutil cert --in instance.yml --silent --out certs.zip
# cd /DIR_TO_LOGSTASH/cert && unzip ~/certs.zip -d ./
# ls
logstash.p12

Create key and crt file from p12

openssl 명령어로 crt, key 파일 생성

# openssl pkcs12 -in logstash.p12 -out logstash.crt -clcerts -nokeys
# openssl pkcs12 -in logstash.p12 -out logstash.key -nocerts -nodes

Edit each configuration

Logstash
vi logstash.yml
input {
  beats {
...
    ssl => true
    ssl_certificate => "/[LOGSTASH]/cert/logstash.crt"
    ssl_key => "/[LOGSTASH]/cert/logstash.key"
...

Filebeat
vi filebeat.yml
output.logstash:
...
  ssl.certificate: "/[FILEBEAT]cert/logstash.crt"
  ssl.key: "/[FILEBEAT]/cert/logstash.key"
  ssl.certificate_authorities: ["/[FILEBEAT]/cert/logstash.crt"]
...

Restart logstash and filebeat service on each server

In logstash server
# sudo kill -SIGHUP [PID_OF_$(ps aux |grep logstash)]

In filebeat server
# sudo service filebeat restart

Check each logstash and filebeat log for connection establishment