数据清洗-使用es的ingest

通常es产品里面，数据清洗的logstash，这里使用另外的ingest做简单的数据处理

这里是根据逗号分隔数据，变成数组

PUT spring_blogs/_doc/1
{"title":"Introducing spring framework......","tags":"spring,spring boot,spring cloud","content":"You konw, for spring framework"
}POST _ingest/pipeline/_simulate
{"pipeline": {"description": "to split blog tags","processors": [{"split": {"field": "tags","separator": ","}}]},"docs": [{"_index": "index","_id": "id","_source": {"title": "Introducing spring framework......","tags": "spring,spring boot,spring cloud","content": "You konw, for spring framework"}},{"_index": "index","_id": "idxx","_source": {"title": "Introducing cloud computering","tags": "docker,k8s,ingrest","content": "You konw, for cloud"}}]
}

# 为ES添加一个 Pipeline
PUT _ingest/pipeline/spring_blog_pipeline
{"description": "a spring blog pipeline","processors": [{"split": {"field": "tags","separator": ","}},{"set":{"field": "views","value": 0}}]
}#查看Pipleline
GET _ingest/pipeline/spring_blog_pipeline#测试pipeline
POST _ingest/pipeline/spring_blog_pipeline/_simulate
{"docs": [{"_source": {"title": "Introducing cloud computering","tags": "docker,k8s,ingrest","content": "You konw, for cloud"}}]
}DELETE spring_blogsPUT spring_blogs/_doc/1
{"title":"Introducing spring framework......","tags":"spring,spring boot,spring cloud","content":"You konw, for spring framework"
}#使用pipeline更新数据
PUT spring_blogs/_doc/2?pipeline=spring_blog_pipeline
{"title": "Introducing cloud computering","tags": "docker,k8s,ingrest","content": "You konw, for cloud"
}POST spring_blogs/_search#增加update_by_query的条件
POST spring_blogs/_update_by_query?pipeline=spring_blog_pipeline
{"query": {"bool": {"must_not": {"exists": {"field": "views"}}}}
}

最终处理后的数据

{"took" : 0,"timed_out" : false,"_shards" : {"total" : 1,"successful" : 1,"skipped" : 0,"failed" : 0},"hits" : {"total" : {"value" : 2,"relation" : "eq"},"max_score" : 1.0,"hits" : [{"_index" : "spring_blogs","_id" : "2","_score" : 1.0,"_source" : {"title" : "Introducing cloud computering","content" : "You konw, for cloud","views" : 0,"tags" : ["docker","k8s","ingrest"]}},{"_index" : "spring_blogs","_id" : "1","_score" : 1.0,"_source" : {"title" : "Introducing spring framework......","content" : "You konw, for spring framework","views" : 0,"tags" : ["spring","spring boot","spring cloud"]}}]}
}

还可以使用Script Prcessor，这种自由度更大，可以处理稍微复杂点数据

POST _ingest/pipeline/_simulate
{"pipeline": {"description": "to split spring blog tags","processors": [{"split": {"field": "tags","separator": ","}},{"script": {"source": """if(ctx.containsKey("title")){ctx.content_length = ctx.title.length();}else{ctx.content_length=0;}"""}},{"set": {"field": "views","value": 0}}]},"docs": [{"_index": "index","_id": "id","_source": {"title": "Introducing spring framework......","tags": "spring,spring boot,spring cloud","content": "You konw, for spring framework"}},{"_index": "index","_id": "idxx","_source": {"title": "Introducing cloud computering","tags": "docker,k8s,ingrest","content": "You konw, for cloud"}}]
}

数据清洗-使用es的ingest相关推荐

filebeat收集日志到elsticsearch中并使用ingest node的pipeline处理
filebeat收集日志到elsticsearch中一.需求二.实现 1.filebeat.yml 配置文件的编写 2.创建自定义的索引模板 3.加密连接到es用户的密码 1.创建keystore ...
Elasticsearch调优实践
背景 Elasticsearch(ES)作为NOSQL+搜索引擎的有机结合体,不仅有近实时的查询能力,还具有强大的聚合分析能力.因此在全文检索.日志分析.监控系统.数据分析等领域ES均有广泛应用. ...
spark配置elasticsearch属性汇总(基于es7)
必要设置 es.resource Elasticsearch资源位置,在该位置读取和写入数据.需要格式 <index>/<type> es.resource.read(默认为e ...
【Elasticsearch】es Ingest 节点
1.概述 1.1 问题引出来自星球同学的提问: "Ingest node什么场景会遇到它? 一直没搜到它是在什么场景工作的?" 的确我们比较关心集群的节点角色的划分.包括: 集群 ...
ES 检索 word、pdf 文档插件 ingest attachment 的管道配置和文档结构映射
一.安装 ingest attachment 插件安装方法:https://blog.csdn.net/catoop/article/details/124468788 二.定义文本抽取管道 1.单 ...
data es集群master_Kubernetes Helm3 部署 ElasticSearch amp; Kibana 7 集群
一.简介 Elasticsearch 是一个分布式的搜索和分析引擎,可以用于全文检索.结构化检索和分析,并能将这三者结合起来.设计用于云计算中,能够达到实时搜索,稳定,可靠,快速,安装使用方便,轻松扩 ...
linux下载python的es库,Elasticsearch py客户端库安装及使用方法解析
一.介绍 elasticsearch-py是一个官方提供的low-level的elasticsearch python客户端库.为什么说它是一个low-level的客户端库呢?因为它只是对elasti ...
自建ES迁移至阿里云ES（使用脚本及logstash同步）
搭建测试3节点ES集群,官网下载所需的版本进行解压运行,ECS与阿里云ES在同一地域可用区,并在相同的VPC内. # mkdir /usr/java # tar xvf jdk-8u162-linux ...
centos7 如何安装部署k8s_五步教你如何使用k8s快速部署ES
前言今天小编打算用五步教大家如何使用k8s快速部署ES,有兴趣的小伙伴可以了解一下~ 由于是使用本地存储,所以需要先创建pv 1.创建存储类 local-elasticsearch.yaml kin ...

数据清洗-使用es的ingest

数据清洗-使用es的ingest相关推荐

最新文章

热门文章