🔧

ELK Stack 日志分析

📂 devops ⏱ 3 min 524 words

ELK Stack 日志分析

什么是 ELK Stack

ELK Stack 是由 Elasticsearch、Logstash 和 Kibana 组成的日志分析平台：

Elasticsearch: 搜索和存储引擎
Logstash: 数据收集和处理管道
Kibana: 可视化和分析界面

架构设计

┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│   日志源    │───→│   Logstash  │───→│Elasticsearch│
│  (Filebeat) │    │   (处理)    │    │   (存储)    │
└─────────────┘    └─────────────┘    └─────────────┘
                                              │
                                              ▼
                                       ┌─────────────┐
                                       │   Kibana    │
                                       │  (可视化)   │
                                       └─────────────┘

Docker 部署

docker-compose.yml

version: '3.8'

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.10.0
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
    ports:
      - "9200:9200"
    volumes:
      - es-data:/usr/share/elasticsearch/data
    networks:
      - elk

  logstash:
    image: docker.elastic.co/logstash/logstash:8.10.0
    container_name: logstash
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline
    ports:
      - "5044:5044"
    environment:
      - "LS_JAVA_OPTS=-Xms512m -Xmx512m"
    networks:
      - elk
    depends_on:
      - elasticsearch

  kibana:
    image: docker.elastic.co/kibana/kibana:8.10.0
    container_name: kibana
    ports:
      - "5601:5601"
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    networks:
      - elk
    depends_on:
      - elasticsearch

  filebeat:
    image: docker.elastic.co/beats/filebeat:8.10.0
    container_name: filebeat
    volumes:
      - ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml
      - /var/log:/var/log:ro
    networks:
      - elk
    depends_on:
      - logstash

volumes:
  es-data:

networks:
  elk:

Filebeat 配置

filebeat.yml

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/nginx/access.log
    fields:
      type: nginx-access
    fields_under_root: true

  - type: log
    enabled: true
    paths:
      - /var/log/nginx/error.log
    fields:
      type: nginx-error
    fields_under_root: true

  - type: log
    enabled: true
    paths:
      - /var/log/app/*.log
    fields:
      type: app
    fields_under_root: true

output.logstash:
  hosts: ["logstash:5044"]

logging.level: info
logging.to_files: true

Logstash 配置

pipeline/logstash.conf

input {
  beats {
    port => 5044
  }
}

filter {
  if [type] == "nginx-access" {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    date {
      match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
    geoip {
      source => "clientip"
    }
  }

  if [type] == "app" {
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}" }
    }
    date {
      match => [ "timestamp", "yyyy-MM-dd HH:mm:ss" ]
    }
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "%{type}-%{+YYYY.MM.dd}"
  }
}

Kibana 配置

访问 Kibana

# 打开浏览器
http://localhost:5601

创建索引模式

进入 Management → Stack Management → Index Patterns
创建索引模式：nginx-access-*
选择 @timestamp 作为时间过滤字段

创建可视化

# 创建仪表盘
1. 进入 Dashboard → Create new dashboard
2. 添加可视化组件
3. 保存仪表盘

实践案例

Nginx 日志分析

# logstash.conf
input {
  file {
    path => "/var/log/nginx/access.log"
    start_position => "beginning"
  }
}

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  
  date {
    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
  
  geoip {
    source => "clientip"
  }
  
  useragent {
    source => "user_agent"
    target => "user_agent"
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "nginx-%{+YYYY.MM.dd}"
  }
}

应用日志分析

# Spring Boot 日志
input {
  file {
    path => "/var/log/app/spring-boot.log"
    codec => multiline {
      pattern => "^%{TIMESTAMP_ISO8601}"
      negate => true
      what => "previous"
    }
  }
}

filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} \[%{DATA:thread}\] %{DATA:logger} - %{GREEDYDATA:message}" }
  }
  
  date {
    match => [ "timestamp", "yyyy-MM-dd HH:mm:ss.SSS" ]
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "app-%{+YYYY.MM.dd}"
  }
}

性能优化

Elasticsearch 优化

# elasticsearch.yml
cluster.name: elk-cluster
node.name: node-1
path.data: /usr/share/elasticsearch/data
path.logs: /usr/share/elasticsearch/logs
network.host: 0.0.0.0
discovery.type: single-node

# JVM 设置
# jvm.options
-Xms2g
-Xmx2g

Logstash 优化

# logstash.yml
pipeline.workers: 4
pipeline.batch.size: 125
pipeline.batch.delay: 50

监控

检查服务状态

# Elasticsearch
curl http://localhost:9200/_cluster/health?pretty

# Kibana
curl http://localhost:5601/api/status

# Logstash
curl http://localhost:9600/_node/stats?pretty

日志文件

# Elasticsearch 日志
tail -f /var/log/elasticsearch/elasticsearch.log

# Logstash 日志
tail -f /var/log/logstash/logstash-plain.log

# Kibana 日志
tail -f /var/log/kibana/kibana.log

常见问题

Elasticsearch 启动失败

# 检查内存
free -h

# 检查磁盘空间
df -h

# 检查日志
tail -f /var/log/elasticsearch/elasticsearch.log

Logstash 处理慢

# 增加 worker 数量
pipeline.workers: 8

# 增加 batch 大小
pipeline.batch.size: 250

# 检查慢查询
curl http://localhost:9600/_node/stats?pretty

最佳实践

使用 Filebeat 替代 Logstash 收集日志
设置合理的索引生命周期
使用 ILM 管理索引
定期清理旧数据
监控集群状态

总结

ELK Stack 是一个强大的日志分析平台。通过合理配置 Filebeat、Logstash、Elasticsearch 和 Kibana，可以实现高效的日志收集、存储和可视化分析。