ElasticSearch + xpack 使用
ElasticSearch 是一个高可用开源全文检索和分析组件。提供存储服务,搜索服务,大数据准实时分析等。一般用于提供一些提供复杂搜索的应。我们为什么要选择 ElasticSearch ?因为它是一个准实时的搜索工具,在一般情况下延时少于一秒,它还支持物理上的水平扩展,并拥有一套分布式协调的管理功能操作比较简单,包括一些 restful 风格的API 等等,接下来我们就来进入今天的正题。
前期准备
1.安装 jdk1.8
yum install java
2.配置源
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
配置elasticsearch
cat >/etc/yum.repos.d/elasticsearch.repo <<EOF
[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF
配置kibana
cat >/etc/yum.repos.d/kibana.repo << EOF
[kibana-5.x]
name=Kibana repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF
3.安装
yum install elasticsearch -y
yum install kibana -y
4.修改 es 配置文件
mkdir /data/es-data/logs -p
chown -R elasticsearch.elasticsearch /data/es-data/logs
[root@linux-node2 ~]# grep -v "^#" /etc/elasticsearch/elasticsearch.yml |grep -v "^$"
cluster.name: lx
node.name: linux-node2.lx.com
path.data: /data/es-data
path.logs: /data/es-data/logs
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.56.12", "192.168.56.14"]
discovery.zen.minimum_master_nodes: 1
5.修改内核参数
vim /etc/security/limits.conf
* soft nofile 655350
* hard nofile 655350
vim /etc/sysctl.conf
fs.file-max=655350
调整内存大小
cat /data/elasticsearch/config/jvm.options (可设置为物理内存的一半)
-Xms8g
-Xmx8g
sysctl -w vm.max_map_count=262144
6.修改 kibana 配置文件
[root@linux-node5 elasticsearch]# grep -v "^#" /etc/kibana/kibana.yml |grep -v "^$"
server.port: 5601
server.host: "0.0.0.0"
server.name: "lx"
elasticsearch.url: "http://192.168.56.14:9200"
elasticsearch.username: "elastic"
elasticsearch.password: "changeme"
7.x-pack 破解
编辑配置文件
cat LicenseVerifier.java
package org.elasticsearch.license;
import java.nio.*;
import java.util.*;
import java.security.*;
import org.elasticsearch.common.xcontent.*;
import org.apache.lucene.util.*;
import org.elasticsearch.common.io.*;
import java.io.*;
public class LicenseVerifier
{
public static boolean verifyLicense(final License license, final byte[] encryptedPublicKeyData) {
return true;
}
public static boolean verifyLicense(final License license) {
return true;
}
}
安装 java-devel
yum install java-devel -y
编译生成LicenseVerifier.class
javac -cp "/usr/share/elasticsearch/lib/elasticsearch-5.6.4.jar:/usr/share/elasticsearch/lib/lucene-core-6.6.1.jar:/usr/share/elasticsearch/plugins/x-pack/x-pack-5.6.4.jar" LicenseVerifier.java
查看生成的LicenseVerifier.class类文件
ll LicenseVerifier.class
替换class文件
cp /usr/share/elasticsearch/plugins/x-pack/x-pack-5.6.4.jar /tmp/test
cd /tmp/test
jar xvf x-pack-5.6.4.jar
cd /tmp/test/org/elasticsearch/license
rm LicenseVerifier.class -f
cp /root/LicenseVerifier.class /tmp/test/org/elasticsearch/license/ #拷贝上文编译生成的java类文件
jar cvf x-pack-5.6.4.jar /tmp/test/* #压缩
\cp /tmp/test/x-pack-5.6.4.jar /usr/share/elasticsearch/plugins/x-pack/ #拷贝到原来的x-pack目录
注意:x-pack-5.6.4.jar 应该拷贝到两个 elasticsearch 集群插件,即目录 /usr/share/elasticsearch/plugins/x-pack/
重启 es 集群
systemct restart elasticsearch
获取 license 证书
https://license.elastic.co/registration
通过填写的邮件获取官方发来的邮件获取证书
cat li-xiang-d28260d9-6c96-4dd2-92dc-2f14a9787903-v5.json
{"license":{"uid":"d28260d9-6c96-4dd2-92dc-2f14a9787903","type":"platinum","issue_date_in_millis":1511740800000,"expiry_date_in_millis":1827359999000,"max_nodes":100,"issued_to":"li xiang (ceshi)","issuer":"Web Form","signature":"AAAAAwAAAA2lsE14rcZQLw3V/JuUAAABmC9ZN0hjZDBGYnVyRXpCOW5Bb3FjZDAxOWpSbTVoMVZwUzRxVk1PSmkxaktJRVl5MUYvUWh3bHZVUTllbXNPbzBUemtnbWpBbmlWRmRZb25KNFlBR2x0TXc2K2p1Y1VtMG1UQU9TRGZVSGRwaEJGUjE3bXd3LzRqZ05iLzRteWFNekdxRGpIYlFwYkJiNUs0U1hTVlJKNVlXekMrSlVUdFIvV0FNeWdOYnlESDc3MWhlY3hSQmdKSjJ2ZTcvYlBFOHhPQlV3ZHdDQ0tHcG5uOElCaDJ4K1hob29xSG85N0kvTWV3THhlQk9NL01VMFRjNDZpZEVXeUtUMXIyMlIveFpJUkk2WUdveEZaME9XWitGUi9WNTZVQW1FMG1DenhZU0ZmeXlZakVEMjZFT2NvOWxpZGlqVmlHNC8rWVVUYzMwRGVySHpIdURzKzFiRDl4TmM1TUp2VTBOUlJZUlAyV0ZVL2kvVk10L0NsbXNFYVZwT3NSU082dFNNa2prQ0ZsclZ4NTltbU1CVE5lR09Bck93V2J1Y3c9PQAAAQBE870HCp9jTj22SVdEP2uAFLF6ikLdDJFtSlbHuYOki6rPtWxIcw8y+WWpPUT5e8lYZw0GkB8CYT5hFLXZTrBqTsNbYRZ3ABdHS1BnKBlkfE4PPcvnTTt4HtBCawNHaW0BNWQ2BA2fFj6zX3HyYJ8q5OaQk/il0t5f/TkIuf0yo3Y/F3rzDIXOHieBFnVvfG3EpNB4lo+G+e6vPMeOW86PsF9eKvQ24nucGDK3S4WSpwxbP1gZFuTdmE9zDguJhRHrtJ6k//A/Q0Fbo8gFntWgHNp+1OJEklH+VBBZUWo17UMGnjjxDrGlLTZcsz2BPmk7mC7e8gBQY4z7zJ/SgnsU","start_date_in_millis":1511740800000}}
此证书的时间为1年使用时间,你可以通过下面网站进行换算http://tool.chinaz.com/Tools/unixtime.aspx,目前我申请了一个10 年的时间
将 "type":"basic" 替换为 "type":"platinum" # 基础班变更为铂金版
将 "expiry_date_in_millis":1543363199999 替换为 "expiry_date_in_millis":1827359999000 # 1年变为10年
查看当前的 license
curl -XGET -u elastic:changeme 'http://127.0.0.1:9200/_license'
{"license" : {"status" : "active","uid" : "21389992-4010-4d2c-917b-94b4e3d5a1dc","type" : "trial","issue_date" : "2017-11-27T05:12:27.999Z","issue_date_in_millis" : 1511759547999,"expiry_date" : "2017-12-27T05:12:27.999Z","expiry_date_in_millis" : 1514351547999,"max_nodes" : 1000,"issued_to" : "lx","issuer" : "elasticsearch","start_date_in_millis" : -1}
}
替换 license
curl -XPUT -u elastic:changeme 'http://127.0.0.1:9200/_xpack/license?acknowledge=true' -d @li-xiang-d28260d9-6c96-4dd2-92dc-2f14a9787903-v5.json
重启 es 集群
systemctl restart elasticsearch
查看 license
[root@linux-node5 license]# curl -XGET -u elastic:changeme 'http://127.0.0.1:9200/_license'
{"license" : {"status" : "active","uid" : "d28260d9-6c96-4dd2-92dc-2f14a9787903","type" : "platinum","issue_date" : "2017-11-27T00:00:00.000Z","issue_date_in_millis" : 1511740800000,"expiry_date" : "2027-11-27T23:59:59.000Z","expiry_date_in_millis" : 1827359999000,"max_nodes" : 100,"issued_to" : "li xiang (ceshi)","issuer" : "Web Form","start_date_in_millis" : 1511740800000}
}
配置 X- Pack 告警
本文配置 X- Pack 告警是通过 filebeat 收集 Nginx 的日志来做的模拟
配置邮件报警
1.安装 Nginx
2.给 Nginx 配置 json 格式的数据
log_format json '{"@timestamp":"$time_iso8601",''"@version":"1",''"client":"$remote_addr",''"url":"$uri",''"status":"$status",''"domain":"$host",''"host":"$server_addr",''"size":$body_bytes_sent,''"responsetime":$request_time,''"referer": "$http_referer",''"ua": "$http_user_agent"''}';access_log logs/access.log json;
3.安装 filebeta
安装
yum install -y filebeta
配置filebeta
[root@linux-node4 filebeat]# grep -v "^ #" filebeat.yml|grep -v "^$"|grep -v "^#"
filebeat.prospectors:
- input_type: logpaths:- /usr/local/nginx/logs/access.logjson.keys_under_root: truejson.overwrite_keys: true
output.elasticsearch:hosts: ["localhost:9200"]username: "elastic"password: "changeme"
参考网址:https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#config-json
http://www.iyunw.cn/archives/filebeat-shou-ji-json-ge-shi-de-nginx-ri-zhi-fa-song-gei-elasticsearch/
启动filebeta服务
systemctl status filebeat.service
这里默认进入 es 的索引是 filebeat-{%}
4.登陆 kibana 添加索引
5.模拟一些 404 的页面,让其展示
for i in {1..1000};do curl http://192.168.56.14:/lx-0$i;sleep 1;done
6.添加报警触发器(收集每分钟内超过访问状态大于 400 且个数超过 20 报警)
Watch JSON
{"trigger": {"schedule": {"interval": "1m"}},"input": {"search": {"request": {"search_type": "query_then_fetch","indices": ["filebeat-2017*"],"types": [],"body": {"size": 0,"query": {"bool": {"must": [{"query_string": {"query": "status:>=400"}},{"range": {"@timestamp": {"gte": "now-1m"}}}]}},"sort": [{"@timestamp": {"order": "desc"}}]}}}},"condition": {"compare": {"ctx.payload.hits.total": {"gt": 20}}},"actions": {"elasticsearch": {"throttle_period_in_millis": 60000,"email": {"profile": "outlook","attachments": {"attached_data": {"data": {"format": "json"}}},"priority": "high","to": ["lixiang@xxxxx.com"],"subject": "Nginx {{ctx.payload.hits.total}} errors","body": {"text": "nginx 404 错误过多,请注意查看"}}}}
}
7.这一步特别重要需要在 ES 配置文件设置否则无法报警(因为我公司使用的是 Exchange 客户端)
xpack.notification.email.account:exchange_account:profile: outlookemail_defaults:from: warning@xxxxx.comsmtp:auth: truestarttls.enable: truehost: smtp.partner.outlook.cnport: 587user: 123@xxxxxx.compassword: 233g@123
参考网址:https://www.elastic.co/guide/en/x-pack/5.6/actions-email.html #邮件设置
8. 查看邮件报警状态以及 kibana 中的状态
对于上面的内容大家可以参看下面的 url 来了解其中的含义
参考网址:
https://www.elastic.co/guide/en/x-pack/5.6/xpack-alerting.html 对集群和索引事件的警告 Xpack document
配置 Webhook 报警
对于 webhook 的报警,我们需要前期做一下工作需要了解一个东西 web.py,具体 URL 链接大家可以访问这个地方 http://webpy.org/tutorial3.zh-cn#starting
1.用 web.py 启动一个 webhook 的监听
[root@linux-node5 ~]# cat webhooks.py
#!/usr/bin/evn python
# -*- coding:utf-8 -*-
import web
import os
import demjson
import sys
reload(sys)
sys.setdefaultencoding( "utf-8" )
urls = ('/log_event_watch','abc',
)
class abc:def POST(self):data = web.data()print datacmd = '''curl -G -v "http://abc.com/abc/" --data "user=lixiang" --data "media=all" --data-urlencode "subject=test" --data-urlencode "message=%s"''' %(data)os.system(cmd)
if __name__ == "__main__":app = web.application(urls, globals())app.run()
运行 python webhooks.py 9000
注意 :urls,因为我的 Watcher ID 为 "log_event_watch"
2.再次回到我们的 kibana 界面, "Management"->"Edit"
查看 Watches
查看 Wathch Json 内容
{"trigger": {"schedule": {"interval": "1m"}},"input": {"search": {"request": {"search_type": "query_then_fetch","indices": ["nginx_access*"],"types": [],"body": {"size": 0,"query": {"bool": {"must": [{"query_string": {"query": "status:>=400"}},{"range": {"@timestamp": {"gte": "now-1m"}}}]}},"sort": [{"@timestamp": {"order": "desc"}}]}}}},"condition": {"compare": {"ctx.payload.hits.total": {"gt": 5}}},"actions": {"email": {"throttle_period_in_millis": 60000,"email": {"profile": "outlook","attachments": {"attached_data": {"data": {"format": "json"}}},"priority": "high","to": ["lixiang@xxxxx.com"],"subject": "Nginx {{ctx.payload.hits.total}} errors","body": {"text": "nginx 404 错误过多,请注意查看"}}},"webhook": {"condition": {"compare": {"ctx.payload.hits.total": {"gt": 5}}},"webhook": {"scheme": "http","host": "192.168.56.15","port": 9000,"method": "post","path": "/{{watch_id}}","params": {},"headers": {},"body": "Encountered {{ctx.payload.hits.total}} errors"}}}
}
3.模拟触发报警
for i in {1..200};do curl http://192.168.56.14:/lx-0$i;sleep 2;done
4.查看结果,这里就不贴出来了(微信,短信,邮件都能收到报警)
5.这就是实现报警消息附带 URL 地址
官方参考网址:
https://www.elastic.co/guide/en/x-pack/5.6/watcher-getting-started.html
阅读原文
ElasticSearch + xpack 使用相关推荐
- Elasticsearch+X-pack和Java Transport方式连接
Elasticsearch+X-pack和Java Transport方式连接 一. 软件及依赖包版本 二. 修改X-pack-core.jar 1. 通过idea插件(java-decompiler ...
- Elasticsearch X-Pack许可证过期解决办法
在使用elasticsearch-sql-cli的时候,用SQL查询ES数据时,抛出Bad request [current license is non-compliant for [sql]] 或 ...
- Elasticsearch X-pack证书过期解决方法
目录 证书未过期 重新编译破解x-pack-5.6.2.jar 重启Es服务 证书已过期 重新生成x-pack-5.6.2.jar,步骤同上 创建license.json license.json目录 ...
- 龙叔学ES:Elasticsearch XPACK安全认证
目录 1.什么是Xpack 2.相关安全配置介绍 2.1.xpack.security.enabled 2.2.xpack.security.http.ssl 2.3.xpack.security.t ...
- elasticsearch x-pack license过期
1.注册一个新的license,每一项都要填写,每次可以使用一年,一年到期后再来注册一个新的 2.更新license (官方文档:https://www.elastic.co/guide/en/x-p ...
- elasticsearch xpack license过期
在启动elasticsearch是提示: blocking [cluster:monitor/stats] operation due to expired license. Cluster heal ...
- ElasticSearch破解x-pack 6.0+和更新许可证(License)
概要 x-pack首次安装可以免费使用一年,过期之后登陆会有如下提示: # http://192.168.0.166:5601/login Login is disabled because your ...
- Spring Data Elasticsearch 和 x-pack 用户名/密码验证连接
为什么80%的码农都做不了架构师?>>> 使用Spring Data Elasticsearch连接elasticsearch时,正常情况下只需要在application.pr ...
- X-pack 为 Elasticsearch 安全保驾护航
Elasticsearch 本身不提供任何用户认证与授权方面的操作(甚至其中压根没有 "用户" 的概念),此方面工作的责任被让给了开发者与管理员.某些观点看来,这并非功能缺失,而被 ...
最新文章
- Python学习笔记:基础
- C++中 static 关键字的作用
- python自动上传图片_使用Python实现一个简单的图片上传存储服务
- maven 构建增量包_使用Maven进行增量构建
- LISP 冻结excel窗格_粗暴讲解,2分钟 | 即懂excel 冻结首行、首列和单元格怎么弄?...
- 20220307:力扣第283场周赛(上)
- 压力测试神器stresstester源码分析
- 苹果mac图像后期处理软件:Lightroom Classic
- 从我国第一例计算机病毒是小球病毒,传入我国的第一例计算机病毒是
- 环保线绕电阻器的主要特性和应用分析
- powerbi嵌入到HTML5,如何把Power BI嵌入到Web应用中
- 我有阳光(我爱生活)
- 新建Flutter项目无法导入FlutterActivity
- Android 梯形TextView
- Java程序输出26个大写字母的ASCII对照表
- Oracle 中的Userenv()
- IE8浏览器兼容问题(日常经验总结)
- Docker总结(配合阿里云容器镜像服务)
- Tesla AI Day:特斯拉如何实现自动驾驶的详解
- python界面开发工具哪个好_python图形界面开发用什么
热门文章
- Ubuntu上使终端显示Git分支(oh-my-zsh)
- matlab内存管理
- 【Live555】live555源码详解(九):ServerMediaSession、ServerMediaSubsession、live555MediaServer
- 调用jsapi缺少参数appid_【Python学习笔记】18、函数的参数关键字参数
- python并发编程方法_Python Futures并发编程详解
- python mysql批量insert数据、返回id_Python3 操作 MySQL 插入一条数据并返回主键 id的实例...
- 迪杰斯特拉算法c++_《算法图解》学习记录7--迪杰斯特拉算法
- python 协程和异步的关系_python协程与异步协程
- 【java】暑期需要复习的操作
- 6 OC 中的isa 指针