java 对es的操作

1.创建连接：

        SearchRequest request = new SearchRequest(GlobalConstant.INDEX_NAME_NEW);SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();request.source(searchSourceBuilder);RestHighLevelClient client = null;try {client = getNewClient();SearchResponse search = client.search(request, RequestOptions.DEFAULT);Aggregations aggregations = search.getAggregations();Terms byCompanyAggregation2 = aggregations.get("by_code");//新的ES地址public static volatile String CLUSTER_NODES_NEW;//新的ES地址端口public static volatile Integer CLUSTER_NODES_NEW_PORT;//新的ES日志索引public static volatile String INDEX_NAME_NEW;//报文日志索引public static volatile String API_MSG_INDEX;//新的ES登录账号public static volatile String ES_USER_NAME;//新的ES地址登录密码public static volatile String ES_USER_PWD;public static RestHighLevelClient getNewClient(){HttpHost host=new HttpHost(GlobalConstant.CLUSTER_NODES_NEW, GlobalConstant.CLUSTER_NODES_NEW_PORT, HttpHost.DEFAULT_SCHEME_NAME);RestClientBuilder builder= org.elasticsearch.client.RestClient.builder(host);builder.setMaxRetryTimeoutMillis(300 * 1000);CredentialsProvider credentialsProvider = new BasicCredentialsProvider();credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(GlobalConstant.ES_USER_NAME, GlobalConstant.ES_USER_PWD));builder.setHttpClientConfigCallback(f -> f.setDefaultCredentialsProvider(credentialsProvider));RestClientBuilder.RequestConfigCallback configCallback = new RestClientBuilder.RequestConfigCallback() {@Overridepublic org.apache.http.client.config.RequestConfig.Builder customizeRequestConfig(org.apache.http.client.config.RequestConfig.Builder requestConfigBuilder) {return requestConfigBuilder.setConnectTimeout(30000).setSocketTimeout(300 * 1000);//更改客户端的超时限制默认30秒现在改为5分钟}};builder.setRequestConfigCallback(configCallback);RestHighLevelClient restClient = new RestHighLevelClient( builder);return restClient;}

2.sql查询

2.1设置查询缓存大小：

默认时100mb下面设置成500

 RequestOptions.Builder builder = RequestOptions.DEFAULT.toBuilder();builder.setHttpAsyncResponseConsumerFactory(new HttpAsyncResponseConsumerFactory.HeapBufferedResponseConsumerFactory(500 * 1024 * 1024));RequestOptions COMMON_OPTIONS = builder.build();response = newClient.search(request, COMMON_OPTIONS);

2.2设置查询出来的固定字段数据，超时时间

 //过滤字段String[] excludes = {};String[] includes = {"logData.apiId", "logData.rawRequest", "logData.rawResponse", "logData.gatewayErrorCode", "logData.backendResponseCode"};searchSourceBuilder.fetchSource(includes, excludes);// 限制查询出来的条数       searchSourceBuilder.query(boolQueryBuilder).size(LIMIT_SIZE);searchSourceBuilder.timeout(TimeValue.timeValueSeconds(300));

2.3查询某个区间范围

  boolQueryBuilder.must(QueryBuilders.rangeQuery("collectTime").gte(beginDate.getTime()).lte(endDate.getTime()));

2.4查询某个并的数据

                    boolQueryBuilder.must(QueryBuilders.matchQuery("busiLogType", "eopori"));

2.5 查询含有某个字段和不含有某个字段

一下意思时存在logData.gatewayResponseCode数据和不存在
logData.gatewayErrorCode数据或者为空

    boolQueryBuilder.must(QueryBuilders.wildcardQuery("logData.gatewayResponseCode", "*"));boolQueryBuilder.mustNot(QueryBuilders.wildcardQuery("logData.gatewayErrorCode", "*"));

以上完整可以es的sql为

{"size":120,"timeout":"300s","query":{"bool":{"must":[{"match":{"busiLogType":{"query":"eopori","operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1}}},{"range":{"collectTime":{"from":1647142201124,"to":1649820601124,"include_lower":true,"include_upper":true,"boost":1}}},{"match":{"logData.apiId":{"query":"10240","operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1}}},{"wildcard":{"logData.gatewayResponseCode":{"wildcard":"*","boost":1}}}],"must_not":[{"wildcard":{"logData.gatewayErrorCode":{"wildcard":"*","boost":1}}}],"adjust_pure_negative":true,"boost":1}},"_source":{"includes":["logData.apiId","logData.rawRequest","logData.rawResponse","logData.gatewayErrorCode","logData.backendResponseCode"],"excludes":[]}
}

2.6 or查询和and包含or

should就是or mus 就是and wildcardQuery就是模糊查询

 BoolQueryBuilder requestQueryBuilder = QueryBuilders.boolQuery();requestQueryBuilder.should(QueryBuilders.wildcardQuery("logData.apiId.keyword", "*" + apiId + "*"));requestQueryBuilder.should(QueryBuilders.wildcardQuery("logData.apiName.keyword", "*" + apiId + "*"));boolQueryBuilder.must(requestQueryBuilder);

2.7 分页查询

from 就是从哪里开始查询
size：就是查询数据
sort 就是按照logData.timestamp时间降序排序

  searchSourceBuilder.size(size).from(from).trackTotalHits(true).query(boolQueryBuilder).sort("logData.timestamp", SortOrder.DESC);

2.8 分组

关键词：aggregation
按照：busiLogType分组数量查查询
别名是：by_busiLogType
统计分组的数据

        AggregationBuilder by_busiLogType = AggregationBuilders.terms("by_busiLogType").field("busiLogType").size(10000);
searchSourceBuilderCount.query(boolQueryBuilder).aggregation(by_busiLogType).size(0);

3.查询结果获取数据

将其转为map获取某个字段的数据在这里插入代码片

 SearchHit[] results = search.getHits().getHits();for (SearchHit hit : results) {Map<String, Object>map = hit.getSourceAsMap();Map<String, Object> maplogData = (Map<String, Object>) map.get("logData");maps.add(maplogData);}

将其转为字符串在转为json对象字符串

 if (response.status().equals(RestStatus.OK) && !response.isTimedOut()) {SearchHits hits = response.getHits();List<ApiMsg> apiMsgs = new ArrayList<>();for (SearchHit hit : hits) {String apiMsgStr = hit.getSourceAsString();try {String logDataStr = JSON.parseObject(apiMsgStr).getJSONObject("logData").toJSONString();

获取分组的统计数据：

            Aggregations aggregations = search.getAggregations();Terms byCompanyAggregation = aggregations.get("by_busiLogType");for (Terms.Bucket bucket2 : byCompanyAggregation.getBuckets()) {if (bucket2.getKey() != null && !StringUtils.isEmpty(bucket2.getKey().toString().trim()) && "eoplog".equals(bucket2.getKey().toString())) {//总量total = bucket2.getDocCount();}}

4.引用es包

    <dependency><groupId>org.elasticsearch.client</groupId><artifactId>transport</artifactId><version>6.5.0</version></dependency><dependency><groupId>org.elasticsearch.client</groupId><artifactId>elasticsearch-rest-client</artifactId><version>6.5.0</version></dependency><dependency><groupId>org.elasticsearch.client</groupId><artifactId>elasticsearch-rest-high-level-client</artifactId><version>6.5.0</version></dependency>

5. elastic相关操作

5.1查询

GET aiolog-eop-eop-comm-eopori-orderquery-2022.04.02/_search

5.2 插入

          POST  aiolog-eop-eop-comm-eoplog-orderquery-2022.04.12/_doc
{}

5.3 删除

DELETE aiolog-eop-eop-comm-eopori-orderquery-2022.03.21

5.4查看模板

右键点击
aiolog-eop-eop-comm-eoplog-orderquery-2022.04.12
{}

5.5插入模板

PUT /_template/aiolog-eop-eop
{
“order”: 1,
“index_patterns”: [
“grp-eop3-full-*”
],
“settings”: {},
}

5.6 查询每小时，每分钟，12小时之前每小时统计数据

一下例子统计前12小时，每小时的数据

{"size":0,"query":{"bool":{"must":[{"range":{"collectTime":{"from":1676602231188,"to":1676645431188,"include_lower":false,"include_upper":true,"boost":1}}}],"adjust_pure_negative":true,"boost":1}},"aggregations":{"collectTime":{"date_histogram":{"field":"collectTime","format":"HH:mm","time_zone":"+08:00","interval":"1h","offset":-569811,"order":{"_key":"asc"},"keyed":false,"min_doc_count":0,"extended_bounds":{"min":1676602231188,"max":1676645431188}}}}
}SearchRequest request = new SearchRequest(GlobalConstant.LOGIN_LOG_INDEX);SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();searchSourceBuilder.query(QueryBuilders.boolQuery().must(QueryBuilders.rangeQuery("collectTime").gt(startTime.getTime()).lte(endTime.getTime())));DateHistogramAggregationBuilder collectTimeAgg = AggregationBuilders.dateHistogram("collectTime").dateHistogramInterval(DateHistogramInterval.HOUR).format("YYYY-MM-dd HH:mm:ss").timeZone(DateTimeZone.forID("+08:00")).offset(-betterTime).minDocCount(0).extendedBounds(extendedBounds).field("collectTime");searchSourceBuilder.aggregation(collectTimeAgg).size(0);request.source(searchSourceBuilder);RestHighLevelClient client = null;try {client = getNewClient();SearchResponse search = client.search(request, RequestOptions.DEFAULT);Aggregations aggregations = search.getAggregations();ParsedDateHistogram parsedDateHistogram = aggregations.get("collectTime");//系统IDfor (Histogram.Bucket bucket : parsedDateHistogram.getBuckets()) {Date date = DateUtil.parse(bucket.getKeyAsString());Date minuteDate= DateUtil.offsetHour(date,1);String time = DateUtil.format(minuteDate, "HH:mm");StatisticLoginCount statisticLoginCount=StatisticLoginCount.builder().time(time).count(bucket.getDocCount()).build();statisticLoginCounts.add(statisticLoginCount);}if(CollectionUtils.isNotEmpty(statisticLoginCounts)&&statisticLoginCounts.size()>12){for(int i=statisticLoginCounts.size()-12;i<statisticLoginCounts.size();i++){statisticLoginCountList.add(statisticLoginCounts.get(i));}}} catch (Exception e) {log.error(e.toString());}finally {client.close();}collectTime 是统计的时间字段 range的是规定从某段时间到某段时间查询范围，format是时间格式-interval是没1小时，offset当前时间的每一小时1分向前便宜开始统计，extended_bounds是统计某段时间到某段时间的值，min_doc_count最小值

注意：在这里插入代码片
number_of_shards和number_of_replicas es配置的分区数量
如果只有一个设置一

5.6 es的安装可以参考

https://blog.csdn.net/happyzxs/article/details/89156068
https://www.cnblogs.com/xyddm/p/14188842.html
https://www.elastic.co/cn/downloads/past-releases/#kibana
https://www.cnblogs.com/wwjj4811/p/14700279.html
https://blog.csdn.net/qq_43676531/article/details/113095349
https://blog.csdn.net/he19970408/article/details/107359861/
https://blog.csdn.net/weixin_39887748/article/details/112481376