ElasticSearch使用（嵌套查询、嵌套高亮）

嵌套查询
bool 查询
- must、should关系
- - 1、只有must
  - 2、只有should
  - 3、must和should同时存在
  - 4、怎样设置should必须满足、最小满足条数
查询高亮
- 1. 普通查询高亮
- 2. 嵌套查询高亮

嵌套查询

当ES的一条文档中的数据是一个数组，数组中有多个对象时，普通的查询不再适用，比如：

{"_index" : "t_trace_index-2019.08.11.15","_type" : "_doc","_id" : "c8fc4f6f0bcd^1565435961237^3980","_score" : 11.434727,"_routing" : "c8fc4f6f0bcd^1565435961237^3980","_source" : {"type" : "pdoc","calls" : [{"bizParams" : """{"input":{"CONTROL_IN_PARAM":{"DATA_STATE":["DATA_ALL"]},"OPER_ID":"10034571"}}""","callee" : "test-busi-ocrm-yyt","caller" : "test-busi-cust-yyt","rpc" : "CSF_SEC_OPERM_OPIDQQUERY_002","traceId" : "c8fc4f6f0bcd^1565435961237^3980","ts" : 1565507717426},{"traceId" : "c8fc4f6f0bcd^1565435961237^3980","caller" : "test-busi-cust-yyt","rpc" : "ocrm_ICbossCSV_idCheck","callee" : "test-busi-ocrm-yyt","ts" : 1565507717915,"bizParams" : """{"input":{"CONTROL_IN_PARAM":{"DATA_STATE":["DATA_ALL"]},"RSP_JSON_STR":"{\"IdCheck\":{\"Seq\":\"10020190811151517510040\",\"CustomerName\":\"786382D36B599DF8D3C312272A75AD81\",\"IDCardType\":\"04\",\"IDCardNum\":\"99425ED800E46883C131C544E9F8BAD9\",\"ChannelId\":\"210000000100001\",\"MsOpcode\":\"C8000\"}}"}}"""},{"traceId" : "c8fc4f6f0bcd^1565435961237^3980","caller" : "test-busi-rule-yyt","rpc" : "cust_IOneCardFiveUserCSV_idCheck","callee" : "test-busi-cust-yyt","ts" : 1565507717985,"bizParams" : """{"input":{"CERT_CODE":"523453","CODE_TYPE":"CM_CERT_TYPE_CHECKCODE","CUST_NAME":"对方八年","CERT_TYPE":"3","CONTROL_IN_PARAM":{"DATA_STATE":["DATA_ALL"]}}}"""}],"id" : "c8fc4f6f0bcd^1565435961237^3980","timestamp" : 1565507717426,"trace_join" : {"name" : "pdoc"}}
}

这时候就需要使用嵌套查询（nested）

bool 查询

查询字句	说明	类型
must	文档必须匹配must查询条件（AND）	数组
should	匹配should的一个或多个（OR）	数组
must_not	文档不能匹配该查询条件	数组
filter	过滤，文档需要匹配过滤条件，功能和must一样，不过filter不参与评分	数组

must、should关系

1、只有must

must中条件必须满足

2、只有should

没有must，只有should，那么should中必须至少匹配一个才行

3、must和should同时存在

如果must中条件匹配，那么should可以不匹配，所以如果查询语句中must、should同时存在，又不设置should最小满足条数，那么should条件有时是不会生效的

项目使用场景：
快速定位功能，需要输入关键字和各种过滤条件，关键字有三种类型，查询时，每个关键字都需要搜索一遍，考虑的是关键字使用should，只要有一种关键字类型满足即可，过滤条件使用must，需要全都满足，在没有指定should最小满足条数时，发现关键字可以输入任意值，只要过滤条件满足就可以查询出结果，关键字没有起作用，在增加should最小满足条数为1之后，设置should至少满足一条，查询结果正确

4、怎样设置should必须满足、最小满足条数

可以设置should中必须匹配多少个，参数"minimum_should_match": "1"，也可以设置匹配百分比"minimum_should_match":75%

Java代码

BoolQueryBuilder boolQueryBuilder = new BoolQueryBuilder();
boolQueryBuilder.should(QueryBuilders.nestedQuery("calls", QueryBuilders.boolQuery().must(QueryBuilders.matchPhraseQuery("calls.bizParams", keyword)), ScoreMode.Avg));
boolQueryBuilder.should(QueryBuilders.nestedQuery("calls", QueryBuilders.boolQuery().must(QueryBuilders.termQuery("calls.rpc", keyword)), ScoreMode.Avg));
boolQueryBuilder.should(QueryBuilders.nestedQuery("calls", QueryBuilders.boolQuery().must(QueryBuilders.termQuery("calls.traceId", keyword)), ScoreMode.Avg));
// 设置should必须匹配多少条
boolQueryBuilder.minimumShouldMatch(1);

DSL查询语句

"should": [
{"nested": {"query": {"bool": {"must": [{"match_phrase": {"calls.bizParams": {"query": "input","slop": 0,"zero_terms_query": "NONE","boost": 1}}}],"adjust_pure_negative": true,"boost": 1}},"path": "calls","ignore_unmapped": false,"score_mode": "avg","boost": 1,}
},
{"nested": {"query": {"bool": {"must": [{"term": {"calls.rpc": {"value": "input","boost": 1}}}],"adjust_pure_negative": true,"boost": 1}},"path": "calls","ignore_unmapped": false,"score_mode": "avg","boost": 1}
},
{"nested": {"query": {"bool": {"must": [{"term": {"calls.traceId": {"value": "input","boost": 1}}}],"adjust_pure_negative": true,"boost": 1}},"path": "calls","ignore_unmapped": false,"score_mode": "avg","boost": 1}
}
],
"adjust_pure_negative": true,
"minimum_should_match": "1",
"boost": 1

设置should中三个条件至少有一个匹配才行

查询高亮

1. 普通查询高亮

java代码

HighlightBuilder highlightBuilder = new HighlightBuilder();
highlightBuilder.field("calls.bizParams");
//高亮标签
highlightBuilder.preTags("<em>").postTags("</em>");
//高亮内容长度
highlightBuilder.fragmentSize(200);
// 创建并设置SearchSourceBuilder对象
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.highlighter(highlightBuilder);

DSL查询语句

{"from": 0,"size": 5000,"timeout": "2000ms","query": {"bool": {"should": [{"nested": {"query": {"bool": {"must": [{"match_phrase": {"calls.bizParams": {"query": "cb47bea067fc^1565266299866^564","slop": 0,"zero_terms_query": "NONE","boost": 1}}}],"adjust_pure_negative": true,"boost": 1}},"path": "calls","ignore_unmapped": false,"score_mode": "none","boost": 1}}],"adjust_pure_negative": true,"boost": 1,"minimum_should_match": "1"}},"highlight": {"pre_tags": ["<em>"],"post_tags": ["</em>"],"fragment_size": 200,"fields": {"calls.traceId": {}}}
}

2. 嵌套查询高亮

在嵌套查询时，如果使用普通高亮查询，会有一个问题，因为一个文档数据记录中可能会有多个对象，所以一条文档可能会有多个满足高亮的字段，而普通高亮只会把高亮的字段在数组中罗列出来，并不知道一条高亮字段属于文档中的哪个对象

普通DSL查询语句

{"from": 0,"size": 5000,"timeout": "2000ms","query": {"bool": {"should": [{"nested": {"query": {"bool": {"must": [{"match_phrase": {"calls.traceId": {"query": "cb47bea067fc^1565266299866^564","slop": 0,"zero_terms_query": "NONE","boost": 1}}}],"adjust_pure_negative": true,"boost": 1}},"path": "calls","ignore_unmapped": false,"score_mode": "none","boost": 1}}],"adjust_pure_negative": true,"boost": 1,"minimum_should_match": "1"}},"highlight": {"pre_tags": ["<em>"],"post_tags": ["</em>"],"fragment_size": 200,"fields": {"calls.traceId": {}}}
}

查询结果：

"hits" : {"total" : {"value" : 1,"relation" : "eq"},"max_score" : 2.0,"hits" : [{"_index" : "t_trace_index-2019.08.08.20","_type" : "_doc","_id" : "cb47bea067fc^1565266299866^564","_score" : 2.0,"_routing" : "cb47bea067fc^1565266299866^564","_source" : {"calls" : [{"traceId" : "cb47bea067fc^1565266299866^564"},{"traceId" : "cb47bea067fc^1565266299866^564"}],"id" : "cb47bea067fc^1565266299866^564","timestamp" : 1565268881001,"trace_join" : {"name" : "pdoc"},"type" : "pdoc"},"highlight" : {"calls.traceId" : ["<em>cb47bea067fc^1565266299866^564</em>","<em>cb47bea067fc^1565266299866^564</em>"]}}]
}

这里对traceId字段使用了高亮查询，可以看到这里高亮匹配到了两条，而除了高亮字段外无任何信息，无法知道高亮语句属于文档数组中的哪个对象，这种高亮方法只适用于非嵌套查询，一条文档中一个对象，高亮字段只会有一个

嵌套查询高亮代码

HighlightBuilder highlightBuilder = new HighlightBuilder();
// 高亮字段
highlightBuilder.field("calls.bizParams");
// 高亮标签
highlightBuilder.preTags("<em>").postTags("</em>");
//highlightBuilder.order("score");
// 高亮内容长度
highlightBuilder.fragmentSize(200);
BoolQueryBuilder boolQueryBuilder = new BoolQueryBuilder();
// 嵌套查询使用高亮
boolQueryBuilder.should(QueryBuilders.nestedQuery("calls", QueryBuilders.boolQuery().must(QueryBuilders.matchPhraseQuery("calls.bizParams", keyword)), ScoreMode.Avg).innerHit(new InnerHitBuilder().setHighlightBuilder(highlightBuilder)));

嵌套查询高亮DSL查询语句

"query": {"bool": {"must": [{"nested": {"query": {"bool": {"must": [{"term": {"calls.traceId": {"value": "c8fc4f6f0bcd^1565435961237^3980","boost": 1}}}],"adjust_pure_negative": true,"boost": 1}},"path": "calls","ignore_unmapped": false,"score_mode": "avg","boost": 1}}],"filter": [{"nested": {"query": {"range": {"calls.ts": {"from": 1564589746707,"to": 1565590046000,"include_lower": true,"include_upper": true,"boost": 1}}},"path": "calls","ignore_unmapped": false,"score_mode": "none","boost": 1}}],"should": [{"nested": {"query": {"bool": {"must": [{"match_phrase": {"calls.bizParams": {"query": "input","slop": 0,"zero_terms_query": "NONE","boost": 1}}}],"adjust_pure_negative": true,"boost": 1}},"path": "calls","ignore_unmapped": false,"score_mode": "avg","boost": 1,"inner_hits": {"ignore_unmapped": false,"from": 0,"size": 3,"version": false,"seq_no_primary_term": false,"explain": false,"track_scores": false,"highlight": {"pre_tags": ["<em>"],"post_tags": ["</em>"],"fragment_size": 200,"fields": {"calls.bizParams": {}}}}}},{"nested": {"query": {"bool": {"must": [{"term": {"calls.rpc": {"value": "input","boost": 1}}}],"adjust_pure_negative": true,"boost": 1}},"path": "calls","ignore_unmapped": false,"score_mode": "avg","boost": 1}},{"nested": {"query": {"bool": {"must": [{"term": {"calls.traceId": {"value": "input","boost": 1}}}],"adjust_pure_negative": true,"boost": 1}},"path": "calls","ignore_unmapped": false,"score_mode": "avg","boost": 1}}],"adjust_pure_negative": true,"minimum_should_match": "1","boost": 1}
}

查询结果，为了减小篇幅，删除无意义字段：

{"took" : 42,"timed_out" : false,"_shards" : {"total" : 430,"successful" : 430,"skipped" : 0,"failed" : 0},"hits" : {"total" : {"value" : 1,"relation" : "eq"},"max_score" : 11.434727,"hits" : [{"_index" : "t_trace_index-2019.08.11.15","_type" : "_doc","_id" : "c8fc4f6f0bcd^1565435961237^3980","_score" : 11.434727,"_routing" : "c8fc4f6f0bcd^1565435961237^3980","_source" : {"type" : "pdoc","calls" : [{"bizParams" : """{"input":{"CONTROL_IN_PARAM":{"DATA_STATE":["DATA_ALL"]},"OPER_ID":"10034571"}}""","rpc" : "CSF_SEC_OPERM_OPIDQQUERY_002","traceId" : "c8fc4f6f0bcd^1565435961237^3980","ts" : 1565507717426},{"traceId" : "c8fc4f6f0bcd^1565435961237^3980","rpc" : "ocrm_ICbossCSV_idCheck","ts" : 1565507717915,"bizParams" : """{"input":{"CONTROL_IN_PARAM":{"DATA_STATE":["DATA_ALL"]},"RSP_JSON_STR":"{\"IdCheck\":{\"Seq\":\"10020190811151517510040\",\"CustomerName\":\"786382D36B599DF8D3C312272A75AD81\",\"IDCardType\":\"04\",\"IDCardNum\":\"99425ED800E46883C131C544E9F8BAD9\",\"ChannelId\":\"210000000100001\",\"MsOpcode\":\"C8000\"}}"}}"""},{"traceId" : "c8fc4f6f0bcd^1565435961237^3980","rpc" : "cust_IOneCardFiveUserCSV_idCheck","ts" : 1565507717985,"bizParams" : """{"input":{"CERT_CODE":"523453","CODE_TYPE":"CM_CERT_TYPE_CHECKCODE","CUST_NAME":"对方八年","CERT_TYPE":"3","CONTROL_IN_PARAM":{"DATA_STATE":["DATA_ALL"]}}}"""},{"traceId" : "c8fc4f6f0bcd^1565435961237^3980","rpc" : "cust_IBackListCSV_queryBlackInfo","bizParams" : """{"input":{"CERT_CODE":"523453","START_NUM":-1,"CERT_TYPE":3,"PARTY_NAME":"","END_NUM":-1,"CONTROL_IN_PARAM":{"DATA_STATE":["DATA_ALL"]}}}""","ts" : 1565507717360},{"traceId" : "c8fc4f6f0bcd^1565435961237^3980","rpc" : "cust_IRealNameCSV_getRealNameCountByPhoneOrIden","ts" : 1565507717409,"bizParams" : """{"input":{"CERT_CODE":"523453","BILL_ID":"","CERT_TYPE":"3","CONTROL_IN_PARAM":{"DATA_STATE":["DATA_ALL"]}}}"""},{"traceId" : "c8fc4f6f0bcd^1565435961237^3980","rpc" : "CSF_RULE_COMMON_RULESCHECK_001","ts" : 1565507717366,"bizParams" : """{"input":{"RULE_MAP":{"ORDER_OBJ_MAP":{"CUST_DC":{"CUST_CERT_TYPE":"3","CUST_NAME":"对方八年","CUST_CERT_CODE":"523453"}}},"CONTROL_IN_PARAM":{"DATA_STATE":["DATA_ALL"]},"RULE_IDS":"100001049"}}"""},{"traceId" : "c8fc4f6f0bcd^1565435961237^3980","rpc" : "CSF_RULE_COMMON_RULESCHECK_001","ts" : 1565507717992,"bizParams" : """{"input":{"RULE_MAP":{"ORDER_OBJ_MAP":{"CUST_DC":{"CUST_CERT_TYPE":"3","CUST_NAME":"对方八年","CUST_CERT_CODE":"523453"}}},"CONTROL_IN_PARAM":{"DATA_STATE":["DATA_ALL"]},"RULE_IDS":"100060110"}}"""},{"traceId" : "c8fc4f6f0bcd^1565435961237^3980","rpc" : "/web-crm/","ts" : 1565507718017}],"id" : "c8fc4f6f0bcd^1565435961237^3980","timestamp" : 1565507717426,"trace_join" : {"name" : "pdoc"}},"inner_hits" : {"calls" : {"hits" : {"total" : {"value" : 7,"relation" : "eq"},"max_score" : 9.85075E-6,"hits" : [{"_index" : "t_trace_index-2019.08.11.15","_type" : "_doc","_id" : "c8fc4f6f0bcd^1565435961237^3980","_nested" : {"field" : "calls","offset" : 0},"_score" : 9.85075E-6,"_source" : {"bizParams" : """{"input":{"CONTROL_IN_PARAM":{"DATA_STATE":["DATA_ALL"]},"OPER_ID":"10034571"}}""","rpc" : "CSF_SEC_OPERM_OPIDQQUERY_002","traceId" : "c8fc4f6f0bcd^1565435961237^3980","ts" : 1565507717426},"highlight" : {"calls.bizParams" : ["""{"<em>input</em>":{"CONTROL_IN_PARAM":{"DATA_STATE":["DATA_ALL"]},"OPER_ID":"10034571"}}"""]}},{"_index" : "t_trace_index-2019.08.11.15","_type" : "_doc","_id" : "c8fc4f6f0bcd^1565435961237^3980","_nested" : {"field" : "calls","offset" : 1},"_score" : 9.85075E-6,"_source" : {"traceId" : "c8fc4f6f0bcd^1565435961237^3980","rpc" : "ocrm_ICbossCSV_idCheck","ts" : 1565507717915,"bizParams" : """{"input":{"CONTROL_IN_PARAM":{"DATA_STATE":["DATA_ALL"]},"RSP_JSON_STR":"{\"IdCheck\":{\"Seq\":\"10020190811151517510040\",\"CustomerName\":\"786382D36B599DF8D3C312272A75AD81\",\"IDCardType\":\"04\",\"IDCardNum\":\"99425ED800E46883C131C544E9F8BAD9\",\"ChannelId\":\"210000000100001\",\"MsOpcode\":\"C8000\"}}"}}"""},"highlight" : {"calls.bizParams" : ["""{"<em>input</em>":{"CONTROL_IN_PARAM":{"DATA_STATE":["DATA_ALL"]},"RSP_JSON_STR":"{\"IdCheck\":{\"Seq\":\"10020190811151517510040\",\"CustomerName\":\"786382D36B599DF8D3C312272A75AD81\",\"IDCardType\":\"04\",\""""]}},{"_index" : "t_trace_index-2019.08.11.15","_type" : "_doc","_id" : "c8fc4f6f0bcd^1565435961237^3980","_nested" : {"field" : "calls","offset" : 2},"_score" : 9.85075E-6,"_source" : {"traceId" : "c8fc4f6f0bcd^1565435961237^3980","rpc" : "cust_IOneCardFiveUserCSV_idCheck","ts" : 1565507717985,"bizParams" : """{"input":{"CERT_CODE":"523453","CODE_TYPE":"CM_CERT_TYPE_CHECKCODE","CUST_NAME":"对方八年","CERT_TYPE":"3","CONTROL_IN_PARAM":{"DATA_STATE":["DATA_ALL"]}}}"""},"highlight" : {"calls.bizParams" : ["""{"<em>input</em>":{"CERT_CODE":"523453","CODE_TYPE":"CM_CERT_TYPE_CHECKCODE","CUST_NAME":"对方八年","CERT_TYPE":"3","CONTROL_IN_PARAM":{"DATA_STATE":["DATA_ALL"]}}}"""]}}]}}}}]}
}

这里对bizParams字段使用嵌套高亮查询，注意看这里的结果和前面查询结果的区别，可以看到查询结果中有inner_hits这个字段

"inner_hits" : {"calls" : {"hits" : {"total" : {"value" : 7,"relation" : "eq"},"max_score" : 9.85075E-6,"hits" : [{"_index" : "t_trace_index-2019.08.11.15","_type" : "_doc","_id" : "c8fc4f6f0bcd^1565435961237^3980","_nested" : {"field" : "calls","offset" : 0},"_score" : 9.85075E-6,"_source" : {"bizParams" : """{"input":{"CONTROL_IN_PARAM":{"DATA_STATE":["DATA_ALL"]},"OPER_ID":"10034571"}}""","rpc" : "CSF_SEC_OPERM_OPIDQQUERY_002","traceId" : "c8fc4f6f0bcd^1565435961237^3980","ts" : 1565507717426},"highlight" : {"calls.bizParams" : ["""{"<em>input</em>":{"CONTROL_IN_PARAM":{"DATA_STATE":["DATA_ALL"]},"OPER_ID":"10034571"}}"""]}},{"_index" : "t_trace_index-2019.08.11.15","_type" : "_doc","_id" : "c8fc4f6f0bcd^1565435961237^3980","_nested" : {"field" : "calls","offset" : 1},"_score" : 9.85075E-6,"_source" : {"traceId" : "c8fc4f6f0bcd^1565435961237^3980","rpc" : "ocrm_ICbossCSV_idCheck","ts" : 1565507717915,"bizParams" : """{"input":{"CONTROL_IN_PARAM":{"DATA_STATE":["DATA_ALL"]},"RSP_JSON_STR":"{\"IdCheck\":{\"Seq\":\"10020190811151517510040\",\"CustomerName\":\"786382D36B599DF8D3C312272A75AD81\",\"IDCardType\":\"04\",\"IDCardNum\":\"99425ED800E46883C131C544E9F8BAD9\",\"ChannelId\":\"210000000100001\",\"MsOpcode\":\"C8000\"}}"}}"""},"highlight" : {"calls.bizParams" : ["""{"<em>input</em>":{"CONTROL_IN_PARAM":{"DATA_STATE":["DATA_ALL"]},"RSP_JSON_STR":"{\"IdCheck\":{\"Seq\":\"10020190811151517510040\",\"CustomerName\":\"786382D36B599DF8D3C312272A75AD81\",\"IDCardType\":\"04\",\""""]}},{"_index" : "t_trace_index-2019.08.11.15","_type" : "_doc","_id" : "c8fc4f6f0bcd^1565435961237^3980","_nested" : {"field" : "calls","offset" : 2},"_score" : 9.85075E-6,"_source" : {"traceId" : "c8fc4f6f0bcd^1565435961237^3980","rpc" : "cust_IOneCardFiveUserCSV_idCheck","ts" : 1565507717985,"bizParams" : """{"input":{"CERT_CODE":"523453","CODE_TYPE":"CM_CERT_TYPE_CHECKCODE","CUST_NAME":"对方八年","CERT_TYPE":"3","CONTROL_IN_PARAM":{"DATA_STATE":["DATA_ALL"]}}}"""},"highlight" : {"calls.bizParams" : ["""{"<em>input</em>":{"CERT_CODE":"523453","CODE_TYPE":"CM_CERT_TYPE_CHECKCODE","CUST_NAME":"对方八年","CERT_TYPE":"3","CONTROL_IN_PARAM":{"DATA_STATE":["DATA_ALL"]}}}"""]}}]}}
}

这里面就是高亮命中的信息，每有一条高亮数据命中，在inner_hits的hit数组中，都会有一条命中对象，相当于把数组中命中的对象字段高亮之后生成了一个新的对象，包括：

位置信息：

"_nested" : {"field" : "calls","offset" : 0
}

offset：高亮命中数据在查询命中结果中的位置，这里表示高亮对象在查询结果中排在第一位，就可以根据这个位置对原来对象中的属性进行修改，例如使用高亮后字段替换原来的字段

高亮后的字段：

"highlight" : {"calls.bizParams" : ["""{"<em>input</em>":{"CONTROL_IN_PARAM":{"DATA_STATE":["DATA_ALL"]},"OPER_ID":"10034571"}}"""]
}

…
等其他的信息，这个可以按需了解