Elasticsearch之Query DSL语法入门

1． query DSL入门

1.1 DSL

query string 后边的参数原来越多，搜索条件越来越复杂，不能满足需求。

GET /book/_search?q=name:java&size=10&from=0&sort=price:desc

DSL:Domain Specified Language，特定领域的语言

es特有的搜索语言，可在请求体中携带搜索条件，功能强大。

查询全部 GET /book/_search

GET /book/_search
{"query": { "match_all": {} }
}

排序 GET /book/_search?sort=price:desc

GET /book/_search
{"query" : {"match" : {"name" : " java"}},"sort": [{ "price": "desc" }]
}

分页查询 GET /book/_search?size=10&from=0

GET  /book/_search
{"query": { "match_all": {} },"from": 0,"size": 1
}

指定返回字段 GET /book/ _search? _source=name,studymodel

GET /book/_search
{"query": { "match_all": {} },"_source": ["name", "studymodel"]
}

通过组合以上各种类型查询，实现复杂查询。

1.2． Query DSL语法

{QUERY_NAME: {ARGUMENT: VALUE,ARGUMENT: VALUE,...}
}

{QUERY_NAME: {FIELD_NAME: {ARGUMENT: VALUE,ARGUMENT: VALUE,...}}
}

GET /test_index/_search
{"query": {"match": {"test_field": "test"}}
}

1.3 组合多个搜索条件

搜索需求：title必须包含elasticsearch，content可以包含elasticsearch也可以不包含，author_id必须不为111

sql where and or !=

初始数据：

POST /website/_doc/1
{"title": "my hadoop article","content": "hadoop is very bad","author_id": 111
}

POST /website/_doc/2
{"title": "my elasticsearch  article","content": "es is very bad","author_id": 112
}
POST /website/_doc/3
{"title": "my elasticsearch article","content": "es is very goods","author_id": 111
}

搜索：

GET /website/_doc/_search
{"query": {"bool": {"must": [{"match": {"title": "elasticsearch"}}],"should": [{"match": {"content": "elasticsearch"}}],"must_not": [{"match": {"author_id": 111}}]}}
}

{"took" : 488,"timed_out" : false,"_shards" : {"total" : 1,"successful" : 1,"skipped" : 0,"failed" : 0},"hits" : {"total" : {"value" : 1,"relation" : "eq"},"max_score" : 0.47000363,"hits" : [{"_index" : "website","_type" : "_doc","_id" : "2","_score" : 0.47000363,"_source" : {"title" : "my elasticsearch  article","content" : "es is very bad","author_id" : 112}}]}
}

更复杂的搜索需求：

select * from test_index where name='tom' or (hired =true and (personality ='good' and rude != true ))

GET /test_index/_search
{"query": {"bool": {"must": { "match":{ "name": "tom" }},"should": [{ "match":{ "hired": true }},{ "bool": {"must":{ "match": { "personality": "good" }},"must_not": { "match": { "rude": true }}}}],"minimum_should_match": 1}}
}

2． full-text search 全文检索

2.1 全文检索

重新创建book索引

PUT /book/
{"settings": {"number_of_shards": 1,"number_of_replicas": 0},"mappings": {"properties": {"name":{"type": "text","analyzer": "ik_max_word","search_analyzer": "ik_smart"},"description":{"type": "text","analyzer": "ik_max_word","search_analyzer": "ik_smart"},"studymodel":{"type": "keyword"},"price":{"type": "double"},"timestamp": {"type": "date","format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"},"pic":{"type":"text","index":false}}}
}

插入数据

PUT /book/_doc/1
{
"name": "Bootstrap开发",
"description": "Bootstrap是由Twitter推出的一个前台页面开发css框架，是一个非常流行的开发框架，此框架集成了多种页面效果。此开发框架包含了大量的CSS、JS程序代码，可以帮助开发者（尤其是不擅长css页面开发的程序人员）轻松的实现一个css，不受浏览器限制的精美界面css效果。",
"studymodel": "201002",
"price":38.6,
"timestamp":"2019-08-25 19:11:35",
"pic":"group1/M00/00/00/wKhlQFs6RCeAY0pHAAJx5ZjNDEM428.jpg",
"tags": [ "bootstrap", "dev"]
}

PUT /book/_doc/2
{
"name": "java编程思想",
"description": "java语言是世界第一编程语言，在软件开发领域使用人数最多。",
"studymodel": "201001",
"price":68.6,
"timestamp":"2019-08-25 19:11:35",
"pic":"group1/M00/00/00/wKhlQFs6RCeAY0pHAAJx5ZjNDEM428.jpg",
"tags": [ "java", "dev"]
}

PUT /book/_doc/3
{
"name": "spring开发基础",
"description": "spring 在java领域非常流行，java程序员都在用。",
"studymodel": "201001",
"price":88.6,
"timestamp":"2019-08-24 19:11:35",
"pic":"group1/M00/00/00/wKhlQFs6RCeAY0pHAAJx5ZjNDEM428.jpg",
"tags": [ "spring", "java"]
}

搜索

GET  /book/_search
{"query" : {"match" : {"description" : "java程序员"}}
}

2.2 _score初探

{"took" : 1,"timed_out" : false,"_shards" : {"total" : 1,"successful" : 1,"skipped" : 0,"failed" : 0},"hits" : {"total" : {"value" : 2,"relation" : "eq"},"max_score" : 2.137549,"hits" : [{"_index" : "book","_type" : "_doc","_id" : "3","_score" : 2.137549,"_source" : {"name" : "spring开发基础","description" : "spring 在java领域非常流行，java程序员都在用。","studymodel" : "201001","price" : 88.6,"timestamp" : "2019-08-24 19:11:35","pic" : "group1/M00/00/00/wKhlQFs6RCeAY0pHAAJx5ZjNDEM428.jpg","tags" : ["spring","java"]}},{"_index" : "book","_type" : "_doc","_id" : "2","_score" : 0.57961315,"_source" : {"name" : "java编程思想","description" : "java语言是世界第一编程语言，在软件开发领域使用人数最多。","studymodel" : "201001","price" : 68.6,"timestamp" : "2019-08-25 19:11:35","pic" : "group1/M00/00/00/wKhlQFs6RCeAY0pHAAJx5ZjNDEM428.jpg","tags" : ["java","dev"]}}]}
}

结果分析

1、建立索引时, description字段 term倒排索引

java 2,3

程序员 3

2、搜索时，直接找description中含有java的文档 2,3，并且3号文档含有两个java字段，一个程序员，所以得分高，排在前面。2号文档含有一个java，排在后面。

3． DSL 语法练习

3.1 match_all

查询所有文档

GET /book/_search
{"query": {"match_all": {}}
}

3.2 match

match query 知道分词器的存在，会对field进行分词操作，然后再查询

GET /book/_search
{"query": { "match": { "description": "java程序员"}}
}

如果想不分词直接查找，可以加.keyword

GET /book/_search
{"query": { "match": { "description.keyword": "java程序员 西二旗"}}
}

3.3 multi_match

可以指定多个字段

GET /book/_search
{"query": {"multi_match": {"query": "java程序员","fields": ["name", "description"]}}
}

3.4、range query

范围查询

GET /book/_search
{"query": {"range": {"price": {"gte": 80,"lte": 90}}}
}

3.5、term query

term query会去倒排索引中寻找确切的term，它并不知道分词器的存在。这种查询适合keyword 、numeric、date

GET /book/_search
{"query": {"term": {"description": "java程序员"}}
}

3.6、terms query

查询某个字段里含有多个关键词的文档

GET /book/_search
{"query": { "terms": { "tags": [ "search", "full_text", "dev" ] }}
}

3.7、exist query

查询有某些字段值的文档

GET /_search
{"query": {"exists": {"field": "join_date"}}
}

3. 8、Fuzzy query

返回包含与搜索词类似的词的文档，该词由Levenshtein编辑距离度量。

包括以下几种情况：

更改角色（box→fox）
删除字符（aple→apple）
插入字符（sick→sic）
调换两个相邻字符（ACT→CAT）

GET /book/_search
{"query": {"fuzzy": {"description": {"value": "jave"}}}
}

3.9、ids

GET /book/_search
{"query": {"ids" : {"values" : ["1", "4", "100"]}}
}

3.10、prefix 前缀查询

GET /book/_search
{"query": {"prefix": {"description": {"value": "spring"}}}
}

3.11、regexp query 正则查询

GET /book/_search
{"query": {"regexp": {"description": {"value": "j.*a","flags" : "ALL","max_determinized_states": 10000,"rewrite": "constant_score"}}}
}

4． Filter

4.1 filter与query示例

需求：用户查询description中有"java程序员"，并且价格大于80小于90的数据。

GET /book/_search
{"query": {"bool": {"must": [{"match": {"description": "java程序员"}},{"range": {"price": {"gte": 80,"lte": 90}}}]}}
}

使用filter:

GET /book/_search
{"query": {"bool": {"must": [{"match": {"description": "java程序员"}}],"filter": {"range": {"price": {"gte": 80,"lte": 90}}}}}
}

4.2 filter与query对比

filter，仅仅只是按照搜索条件过滤出需要的数据而已，不计算任何相关度分数，对相关度没有任何影响。

query，会去计算每个document相对于搜索条件的相关度，并按照相关度进行排序。

应用场景：

一般来说，如果你是在进行搜索，需要将最匹配搜索条件的数据先返回，那么用query 如果你只是要根据一些条件筛选出一部分数据，不关注其排序，那么用filter

4.3 filter与query性能

filter，不需要计算相关度分数，不需要按照相关度分数进行排序，同时还有内置的自动cache最常使用filter的数据

query，相反，要计算相关度分数，按照分数进行排序，而且无法cache结果

5．定位错误语法

验证错误语句：

GET /book/_validate/query?explain
{"query": {"mach": {"description": "java程序员"}}
}

{"valid" : false,"error" : "org.elasticsearch.common.ParsingException: no [query] registered for [mach]"
}

正确

GET /book/_validate/query?explain
{"query": {"match": {"description": "java程序员"}}
}

{"_shards" : {"total" : 1,"successful" : 1,"failed" : 0},"valid" : true,"explanations" : [{"index" : "book","valid" : true,"explanation" : "description:java description:程序员"}]
}

一般用在那种特别复杂庞大的搜索下，比如你一下子写了上百行的搜索，这个时候可以先用validate api去验证一下，搜索是否合法。

合法以后，explain就像mysql的执行计划，可以看到搜索的目标等信息。

6．定制排序规则

6.1 默认排序规则

默认情况下，是按照_score降序排序的

然而，某些情况下，可能没有用到_score，比如说filter

GET book/_search
{"query": {"bool": {"must": [{"match": {"description": "java程序员"}}]}}
}

当然，也可以是constant_score

6.2 定制排序规则

相当于sql中order by ?sort=sprice:desc

GET /book/_search
{"query": {"constant_score": {"filter" : {"term" : {"studymodel" : "201001"}}}},"sort": [{"price": {"order": "asc"}}]
}

7． Text字段排序问题

如果对一个text field进行排序，结果往往不准确，因为分词后是多个单词，再排序就不是我们想要的结果了。

通常解决方案是，将一个text field建立两次索引，一个分词，用来进行搜索；一个不分词，用来进行排序。

fielddate:true

PUT /website
{"mappings": {"properties": {"title": {"type": "text","fields": {"keyword": {"type": "keyword"}        }      },"content": {"type": "text"},"post_date": {"type": "date"},"author_id": {"type": "long"}}}
}

插入数据

PUT /website/_doc/1
{"title": "first article","content": "this is my second article","post_date": "2019-01-01","author_id": 110
}

PUT /website/_doc/2
{"title": "second article","content": "this is my second article","post_date": "2019-01-01","author_id": 110
}

PUT /website/_doc/3
{"title": "third article","content": "this is my third article","post_date": "2019-01-02","author_id": 110
}

搜索

GET /website/_search
{"query": {"match_all": {}},"sort": [{"title.keyword": {"order": "desc"}}]
}

8． Scroll分批查询

场景：下载某一个索引中1亿条数据，到文件或是数据库。

不能一下全查出来，系统内存溢出。所以使用scoll滚动搜索技术，一批一批查询。

scoll搜索会在第一次搜索的时候，保存一个当时的视图快照，之后只会基于该旧的视图快照提供数据搜索，如果这个期间数据变更，是不会让用户看到的

每次发送scroll请求，我们还需要指定一个scoll参数，指定一个时间窗口，每次搜索请求只要在这个时间窗口内能完成就可以了。

搜索

GET /book/_search?scroll=1m
{"query": {"match_all": {}},"size": 3
}

{"_scroll_id" : "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAMOkWTURBNDUtcjZTVUdKMFp5cXloVElOQQ==","took" : 3,"timed_out" : false,"_shards" : {"total" : 1,"successful" : 1,"skipped" : 0,"failed" : 0},"hits" : {"total" : {"value" : 3,"relation" : "eq"},"max_score" : 1.0,"hits" : []}
}

获得的结果会有一个scoll_id，下一次再发送scoll请求的时候，必须带上这个scoll_id

GET /_search/scroll
{"scroll": "1m", "scroll_id" : "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAMOkWTURBNDUtcjZTVUdKMFp5cXloVElOQQ=="
}

与分页区别：

分页给用户看的 deep paging

scroll是用户系统内部操作，如下载批量数据，数据转移。零停机改变索引映射。