定制开发ELk日志分析系统搭建-巴掌软件-高性价比的软件定制开发解决方案

一、什么是ELK
ELK是 + Logstash + Kibana 定制开发这种架构的简写.

二、ELK常见的
Elasticsearch + Logstash + Kibana
定制开发这是一种最简单的架构。这种架构，通过logstash收集日志，Elasticsearch分析日志，然后在Kibana(web界面)中展示。定制开发这种架构虽然是官网介定制开发绍里的方式，定制开发但是往往在生产中很少使用。

Elasticsearch + Logstash + filebeat + Kibana
定制开发与上一种架构相比，这种架构增加了一个filebeat模块。filebeat是一个轻量的日志收集代理，用来部署在客户端，优势是消耗非常少的资源(较logstash)，所以生产中，往往会采取这种架构方式，但是这种架构有一个缺点，当logstash出现故障，会造成日志的丢失。

Elasticsearch + Logstash + filebeat + redis(也可以是其他中间件，比如rabbitmq（集群化）) + Kibana
这种架构是上面那个架构的完善版，通过增加中间件，来避免数据的丢失。当Logstash出现故障，日志还是存在中间件中，当Logstash再次启动，则会读取中间件中积压的日志。

下面如何搭建：

1.es 搭建

1.Create a file called elasticsearch.repo in the /etc/yum.repos.d/


[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md

2.sudo yum install --enablerepo=elasticsearch elasticsearch

3.配置：vim /etc/elasticsearch/elasticsearch.yml


path.data: /data/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
 
network.host: 0.0.0.0
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
xpack.security.enabled: false

4.修改同目录下:jvm.options 保证 -xms -xmx 保证是系统内存一半以下或者保证自己服务器合适大小

5.创建文件夹并授权如上的 pat.data=

mkdir /data/elasticsearch

chown -R elasticsearch:elasticsearch /data/elasticsearch/

6.启动：


sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
 
sudo systemctl start elasticsearch.service

2. 下载logstash （rabbitmq 中间件安装跳过自己百度搜索）

1.选择自己的操作系统下载进行解压

2.修改配置文件 conf/logstash-sample.conf


# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
 
input {
 #对接 filebeat  我们使用java 链接不用这个
  beats {
    port => 5044
  }
  #对接 tcp
   tcp {
    mode => "server"
    host => "0.0.0.0"
    port => 4560
    codec => json_lines
   }
   #对接rocketmq
   rabbitmq {
        host=>"localhost"                    
        vhost => "/"                     
        port=> 5672                             
        user=>"guest"                        
        password=>"guest"                    
        queue=>"station_Route"                          
        durable=> true                          
        codec=>json
      
    }
}
 
output {
  elasticsearch {
    hosts => ["http://ip:9200"]
    index => "rabbitmq-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
}

3. 启动 logstash -f logstash.conf

3.安装 kibana

1.解压完成修改配置文件 conf/kibana.yml


i18n.locale: "zh-CN"
elasticsearch.hosts: ["http://localhost:9200"]
server.name: "test-kin"
server.port: 5601
 
# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "0.0.0.0"

2.启动

 nohup ./kibana --allow-root >/dev/null &

3.注意防火墙地址

4.Spring boot 项目整合

1.添加依赖


      <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-amqp</artifactId>
            <version>2.5.5</version>
        </dependency>

2.配置yml


spring:
  application:
    name: test
  mvc:
    static-path-pattern: /**
  rabbitmq:
    host: localhost
    port: 5672
    username: guest
    password: guest

3.修改 logback-spring.xml 自定义一个logback拦截器只有当使用Marker 再进行记录


/**
 * @author chenkang
 * @date 2022/5/19 13:24
 */
public class LogStashFilter extends Filter<ILoggingEvent> {
 
    public final static Marker LOGSTASH = MarkerFactory.getMarker("logstash");
 
    @Override
    public FilterReply decide(ILoggingEvent iLoggingEvent) {
        Marker marker = iLoggingEvent.getMarker();
       return Optional.ofNullable(marker).filter(m->m.equals(LOGSTASH)).map(m->FilterReply.ACCEPT).orElse(FilterReply.DENY);
    }
}


 
    <springProperty name="rabbitmqHost" source="spring.rabbitmq.host"/>
    <springProperty name="rabbitmqPort" source="spring.rabbitmq.port"/>
    <springProperty name="rabbitmqUsername" source="spring.rabbitmq.username"/>
    <springProperty name="rabbitmqPassword" source="spring.rabbitmq.password"/>
 
 
 <appender name="AMQP" class="org.springframework.amqp.rabbit.logback.AmqpAppender">
        <!--Layout（纯文本）而不是格式化的JSON -->
        <filter class="com.chenkang.test.config.LogStashFilter" />
        <layout>
            <pattern>
                <![CDATA[%msg]]>
            </pattern>
        </layout>
        <host>${rabbitmqHost}</host>
        <port>${rabbitmqPort}</port>
        <username>${rabbitmqUsername}</username>
        <password>${rabbitmqPassword}</password>
        <declareExchange>false</declareExchange>
        <exchangeType>direct</exchangeType>
        <exchangeName>exchanges.route</exchangeName>
        <routingKeyPattern>route_exchange</routingKeyPattern>
        <generateId>true</generateId>
        <charset>UTF-8</charset>
        <durable>false</durable>
        <deliveryMode>NON_PERSISTENT</deliveryMode>
    </appender>

最后是测试：


        Message message = new Message();
        message.setDeviceCode("code123");
        message.setDeviceName("deviceName3345");
        message.setIndex("1024");
        log.info(LogStashFilter.LOGSTASH,JSON.toJSONString(message));
        log.info(JSON.toJSONString(message));

输出日志就会推送rabbitmq 订阅然后 logstash 消费存储到 es

KIbana 查询数据自己本地实测每秒 1-2 千没啥问题但是10000 时候就崩掉了 logstash 假死