ElasticSearch教程——自定义分词器（转学习使用）

一、分词器

Elasticsearch中，内置了很多分词器（analyzers），例如standard（标准分词器）、english（英文分词）和chinese（中文分词），默认是standard.

standard tokenizer：以单词边界进行切分
standard token filter：什么都不做
lowercase token filter：将所有字母转换为小写
stop token filer（默认被禁用）：移除停用词，比如a the it等等

二、修改分词器设置

启用english，停用词token filter

PUT /my_index
{"settings": {"analysis": {"analyzer": {"es_std":{"type":"standard","stopwords":"_english_"}}}}
}

三、标准分词测试代码

GET /my_index/_analyze
{"analyzer": "standard","text":"a dog is in the house"
}

结果：

{"tokens": [{"token": "a","start_offset": 0,"end_offset": 1,"type": "<ALPHANUM>","position": 0},{"token": "dog","start_offset": 2,"end_offset": 5,"type": "<ALPHANUM>","position": 1},{"token": "is","start_offset": 6,"end_offset": 8,"type": "<ALPHANUM>","position": 2},{"token": "in","start_offset": 9,"end_offset": 11,"type": "<ALPHANUM>","position": 3},{"token": "the","start_offset": 12,"end_offset": 15,"type": "<ALPHANUM>","position": 4},{"token": "house","start_offset": 16,"end_offset": 21,"type": "<ALPHANUM>","position": 5}]
}

四、设置的英文分词测试代码

GET /my_index/_analyze
{"analyzer": "es_std","text":"a dog is in the house"}

结果：

{"tokens": [{"token": "dog","start_offset": 2,"end_offset": 5,"type": "<ALPHANUM>","position": 1},{"token": "house","start_offset": 16,"end_offset": 21,"type": "<ALPHANUM>","position": 5}]
}

五、自定义分词器

PUT /my_index
{"settings": {"analysis": {"char_filter": {"&_to_and": {"type": "mapping","mappings": ["&=> and"]}},"filter": {"my_stopwords": {"type": "stop","stopwords": ["the", "a"]}},"analyzer": {"my_analyzer": {"type": "custom","char_filter": ["html_strip", "&_to_and"],"tokenizer": "standard","filter": ["lowercase", "my_stopwords"]}}}}
}

测试：

GET /my_index/_analyze
{"text": "tom&jerry are a friend in the house, <a>, HAHA!!","analyzer": "my_analyzer"
}

结果：

{"tokens": [{"token": "tomandjerry","start_offset": 0,"end_offset": 9,"type": "<ALPHANUM>","position": 0},{"token": "are","start_offset": 10,"end_offset": 13,"type": "<ALPHANUM>","position": 1},{"token": "friend","start_offset": 16,"end_offset": 22,"type": "<ALPHANUM>","position": 3},{"token": "in","start_offset": 23,"end_offset": 25,"type": "<ALPHANUM>","position": 4},{"token": "house","start_offset": 30,"end_offset": 35,"type": "<ALPHANUM>","position": 6},{"token": "haha","start_offset": 42,"end_offset": 46,"type": "<ALPHANUM>","position": 7}]
}

六、type中的使用

PUT /my_index/_mapping/my_type
{"properties": {"content": {"type": "text","analyzer": "my_analyzer"}}
}

转载于:https://www.cnblogs.com/yfb918/p/10718712.html

ElasticSearch教程——自定义分词器（转学习使用）相关推荐

elasticsearch教程--中文分词器作用和使用
概述本文都是基于elasticsearch安装教程中的elasticsearch安装目录(/opt/environment/elasticsearch-6.4.0)为范例环境准备 ·全新最小 ...
Elasticsearch教程 IK分词器安装
首先说说分词器,分词,英语分词很好分,就是按固定的英文的空格,或者"-".中文分词就稍微有点复杂了,而 Elasticsearch 默认是带分词器了,我们来测试一下. http: ...
elasticsearch分词练习、自定义分词器练习
elasticsearch分词练习.自定义分词器练习分词练习自定义分词器分词练习准备数据 post metric_zabbix/log {"@message":" ...
【Es】ElasticSearch 自定义分词器
1.分词器转载:https://blog.csdn.net/gwd1154978352/article/details/83343933 分词器首先看文章:[Elasticsearch]Elasti ...
Elasticsearch 分布式搜索引擎 -- 自动补全（拼音分词器、自定义分词器、自动补全查询、实现搜索框自动补全）
文章目录 1. 自动补全 1.1 拼音分词器 1.2.1 自定义分词器 1.2.2 小结 1.2 自动补全 1.3 实现酒店搜索框自动补全 1.3.1 修改酒店映射结构 1.3.2 修改HotelDo ...
Elasticsearch配置拼音分词和自定义分词器
下载elasticsearch-analysis-pinyin拼音分词器 https://codeload.github.com/medcl/elasticsearch-analysis-pinyi ...
ElasticSearch 自定义分词器Analyzer示例
一.前提概述接下来定义一个index,并在该index中使用自定义分词器. 假设在ES中有这么一个index,用来存储用户在什么时间调用了哪个接口,调用的入参是什么的这么一个信息(即服务网关日志), ...
59、Docker ElasticSearch安装拼音分词器及自定义分词器
一.使用拼音分词器 1.拼音分词器 2.docker下安装拼音分词器插件 3.测试拼音分词器 # 测试拼音分词器 POST /_analyze { "text": [" ...
ElasticSearch入门：ES分词器与自定义分词器
ES入门:ES分词器与自定义分词器分词器的简单介绍不同分词器的效果对比自定义分词器的应用分词器的简单介绍分词器是es中的一个组件,通俗意义上理解,就是将一段文本按照一定的逻辑,分析成多个词语 ...

ElasticSearch教程——自定义分词器（转学习使用）

ElasticSearch教程——自定义分词器（转学习使用）相关推荐

最新文章

热门文章