spark 运行自带python示例的方式

命令行直接执行：

[root@master spark-2.2.0-bin-hadoop2.7]# bin/spark-submit examples/src/main/python/ml/kmeans_example.py

此外，也可以将代码拷贝到pyspark中执行，如下所示：

Welcome to____              __/ __/__  ___ _____/ /___\ \/ _ \/ _ `/ __/  '_//__ / .__/\_,_/_/ /_/\_\   version 2.2.0/_/Using Python version 2.7.5 (default, Nov  6 2016 00:28:07)
SparkSession available as 'spark'.
>>> from pyspark.ml.clustering import KMeans
set)
wssse = model.computeCost(dataset)
print("Within Set Sum of Squared Errors = " + str(wssse))centers = model.clusterCenters()>>> from pyspark.sql import SparkSession
>>> dataset = spark.read.format("libsvm").load("/home/spark/spark-2.2.0-bin-hadoop2.7/data/mllib/sample_kmeans_data.txt")
>>> kmeans = KMeans().setK(2).setSeed(1)
>>> model = kmeans.fit(dataset)18/10/24 04:23:19 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
18/10/24 04:23:19 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
>>> wssse = model.computeCost(dataset)
>>> print("Within Set Sum of Squared Errors = " + str(wssse))
Within Set Sum of Squared Errors = 0.12
>>>
>>> centers = model.clusterCenters()
>>> for center in centers:print(center)
...
[0.1 0.1 0.1]
[9.1 9.1 9.1]

spark 运行自带python示例的方式相关推荐

python 全局_全局关键字，带Python示例
python 全局 Python全局关键字 (Python global keyword) global is a keyword (case-sensitive) in python, it is ...
python true_True关键字，带Python示例
python true Python True关键字 (Python True keyword) True is a keyword (case-sensitive) in python, it is ...
python none_None关键字，带Python示例
python none Python None关键字 (Python None keyword) Note: None is not a keyword – we are just consideri ...
python函数示例_带Python示例的complex（）函数
python函数示例 Python complex()函数 (Python complex() function) complex() function is a library function i ...
python函数示例_带Python示例的float（）函数
python函数示例 Python float()函数 (Python float() function) float() function is a library function in Pyth ...
flink安装以及运行自带wordcount示例（单机版，无hadoop环境）
1.下载安装包到/opt目录 2.解压安装包 tar zxf flink-1.6.1-bin-hadoop26-scala_2.11.tgz 3.启动flink cd /opt/flink-1.6.1 ...
tau nb_math.tau常数，带Python示例
tau nb Python math.tau常数 (Python math.tau constant) math.tau constant is a predefined constant, whic ...
spark运行自带例子_运行spark自带的例子出错及解决
以往都是用java运行spark的没问题,今天用scala在eclipse上运行spark的代码倒是出现了错误 ,记录首先是当我把相关的包导入好后,Run,报错: Exception in thre ...
时间模块，带Python示例
Python时间模块 (Python time Module) The time module is a built-in module in Python and it has various fu ...

spark 运行自带python示例的方式

spark 运行自带python示例的方式相关推荐

最新文章

热门文章