pyspark pipline

关于pip line的介绍

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Created on Thu Jun  7 16:49:03 2018@author: luogan
"""from pyspark.ml import Pipeline
from pyspark.ml.classification import LogisticRegression
from pyspark.ml.feature import HashingTF, Tokenizerfrom pyspark.sql import SparkSessionspark= SparkSession\.builder \.appName("dataFrame") \.getOrCreate()# Prepare training documents from a list of (id, text, label) tuples.
training = spark.createDataFrame([(0, "a b c d e spark", 1.0),(1, "b d", 0.0),(2, "spark f g h", 1.0),(3, "hadoop mapreduce", 0.0)
], ["id", "text", "label"])# Configure an ML pipeline, which consists of three stages: tokenizer, hashingTF, and lr.
tokenizer = Tokenizer(inputCol="text", outputCol="words")
hashingTF = HashingTF(inputCol=tokenizer.getOutputCol(), outputCol="features")
lr = LogisticRegression(maxIter=10, regParam=0.001)
pipeline = Pipeline(stages=[tokenizer, hashingTF, lr])# Fit the pipeline to training documents.
model = pipeline.fit(training)# Prepare test documents, which are unlabeled (id, text) tuples.
test = spark.createDataFrame([(4, "spark i j k"),(5, "l m n"),(6, "spark hadoop spark"),(7, "apache hadoop")
], ["id", "text"])# Make predictions on test documents and print columns of interest.
prediction = model.transform(test)
selected = prediction.select("id", "text", "probability", "prediction")
for row in selected.collect():rid, text, prob, prediction = rowprint("(%d, %s) --> prob=%s, prediction=%f" % (rid, text, str(prob), prediction))

(4, spark i j k) --> prob=[0.15554371384424398,0.844456286155756], prediction=1.000000
(5, l m n) --> prob=[0.8307077352111738,0.16929226478882617], prediction=0.000000
(6, spark hadoop spark) --> prob=[0.06962184061952888,0.9303781593804711], prediction=1.000000
(7, apache hadoop) --> prob=[0.9815183503510166,0.018481649648983405], prediction=0.000000

pyspark pipline相关推荐

pyspark入门---机器学习实战预测婴儿出生率（二）使用ML库
机器学习实战预测婴儿出生率 1.加载数据 2.创建转换器 3.创建预测器 4.创建管道 5.训练模型 6.使用BinaryClassificationEvaluator对模型评估 7.模型保存与调用 ...
pyspark汇总小结
20220402 Spark报Total size of serialized results of 12189 tasks is bigger than spark.driver.maxResult ...
spark- PySparkSQL之PySpark解析Json集合数据
PySparkSQL之PySpark解析Json集合数据数据样本 12341234123412342|asefr-3423|[{"name":"spark", ...
jupyter笔记本_如何为Jupyter笔记本电脑设置PySpark
jupyter笔记本 by Tirthajyoti Sarkar 由Tirthajyoti Sarkar 如何为Jupyter笔记本电脑设置PySpark (How to set up PySpark ...
手把手教你实现PySpark机器学习项目——回归算法
作者 | hecongqing 来源 | AI算法之心(ID:AIHeartForYou) [导读]PySpark作为工业界常用于处理大数据以及分布式计算的工具,特别是在算法建模时起到了非常大的作用. ...
pyspark常用API
union 和unionall union 纵向合并dataframe In this Spark article, you will learn how to union two or more d ...
利用PySpark进行迁移学习的多类图像分类
在本文中,我们将演示计算机视觉问题,它具有结合两种最先进技术的能力:深度学习和Apache Spark.我们将利用深度学习管道的强大功能来解决多类图像分类问题. PySpark 是 Spark 为 P ...
pyspark dataframe数据连接（join）、转化为pandas dataframe、基于多个字段删除冗余数据
pyspark dataframe数据连接(join).转化为pandas dataframe.基于多个字段删除冗余数据目录 pyspark dataframe数据连接(join).转化为panda ...
基于关联规则（Variational Autoencoders）疾病预测系统实战：(pyspark FPGrowth实现频繁项集挖掘、最后给出预测模型topK准确率和召回率)
基于关联规则(Variational Autoencoders)疾病预测系统实战:(pyspark FPGrowth实现频繁项集挖掘.最后给出预测模型topK准确率和召回率) 目录

pyspark pipline

pyspark pipline相关推荐

最新文章

热门文章