pyspark:

AttributeError: 'NoneType' object has no attribute 'setCallSite'

我草，是pyspark的bug。解决方法：

print("Approximately joining on distance smaller than 0.6:")distance_min = model.approxSimilarityJoin(imsi_proc_df, imsi_proc_df, 1e6, distCol="JaccardDistance") \.select(col("datasetA.id").alias("idA"),col("datasetB.id").alias("idB"),col("JaccardDistance")) #.filter("idA=idB")print(distance_min.show())print("*"*88)print(imsi_proc_df.show())key = Vectors.sparse(53, [1, 3], [1.0, 1.0])print(model.approxNearestNeighbors(imsi_proc_df, key, 2).show())print("start calculate find botnet!")print("*"*99)print("time start:", time.time())print(type(distance_min), dir(distance_min))print(dir(distance_min.toLocalIterator))

    ############################################## add this line to solvedistance_min.sql_ctx.sparkSession._jsparkSession = spark_app._jsparkSessiondistance_min._sc = spark_app._sc    #############################################similarity_val_rdd = distance_min.toLocalIterator #.collect()print("time end:", time.time())print(similarity_val_rdd)print("*"*99)try:G = ConnectedGraph()ddos_ue_list = []for item in similarity_val_rdd():imsi, imsi2, jacard_similarity_val = item["idA"], item["idB"], item["JaccardDistance"]print("???", imsi, imsi2, jacard_similarity_val)

Description

reproducing the bug from the example in the documentation:

import pyspark
from pyspark.ml.linalg import Vectors
from pyspark.ml.stat import Correlation
spark = pyspark.sql.SparkSession.builder.getOrCreate()
dataset = [[Vectors.dense([1, 0, 0, -2])],[Vectors.dense([4, 5, 0, 3])],[Vectors.dense([6, 7, 0, 8])],[Vectors.dense([9, 0, 0, 1])]]
dataset = spark.createDataFrame(dataset, ['features'])
df = Correlation.corr(dataset, 'features', 'pearson') df.collect()

This produces the following stack trace:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-92-e7889fa5d198> in <module>()11 dataset = spark.createDataFrame(dataset, ['features'])12 df = Correlation.corr(dataset, 'features', 'pearson')
---> 13 df.collect()/opt/spark/python/pyspark/sql/dataframe.py in collect(self)530         [Row(age=2, name=u'Alice'), Row(age=5, name=u'Bob')] 531 """ --> 532 with SCCallSiteSync(self._sc) as css: 533 sock_info = self._jdf.collectToPython() 534 return list(_load_from_socket(sock_info, BatchedSerializer(PickleSerializer()))) /opt/spark/python/pyspark/traceback_utils.py in __enter__(self) 70 def __enter__(self): 71 if SCCallSiteSync._spark_stack_depth == 0: ---> 72 self._context._jsc.setCallSite(self._call_site) 73 SCCallSiteSync._spark_stack_depth += 1 74 AttributeError: 'NoneType' object has no attribute 'setCallSite'

Analysis:

Somehow the dataframe properties `df.sql_ctx.sparkSession._jsparkSession`, and `spark._jsparkSession` do not match with the ones available in the spark session.

The following code fixes the problem (I hope this helps you narrowing down the root cause)

df.sql_ctx.sparkSession._jsparkSession = spark._jsparkSession
df._sc = spark._scdf.collect()>>> [Row(pearson(features)=DenseMatrix(4, 4, [1.0, 0.0556, nan, 0.4005, 0.0556, 1.0, nan, 0.9136, nan, nan, 1.0, nan, 0.4005, 0.9136, nan, 1.0], False))]

转载于:https://www.cnblogs.com/bonelee/p/10976253.html

pyspark AttributeError: 'NoneType' object has no attribute 'setCallSite'相关推荐

AttributeError: 'NoneType' object has no attribute 'sc' 解决方法（二）
上一次本以为可以解决了这个问题,然而并没有那么地简单.博主最近在edx网站学习pyspark,想打一下视频上的代码,结果报错了,依旧是报了"AttributeError:'NoneType' ...
Traceback (most recent call last): File AttributeError: 'NoneType' object has no attribute 'group'
Traceback (most recent call last):File "<stdin>", line 1, in <module> Attribut ...
Keras问题“AttributeError: 'NoneType' object has no attribute 'update”解决
BUG 在使用Keras训练模型时,在每个epoch完成后save_model时会报错 "AttributeError: 'NoneType' object has no attribute ...
Pywinauto 应用后端类型选择错误：AttributeError: ‘NoneType‘ object has no attribute ‘backend‘. 原因及解决办法
AttributeError: 'NoneType' object has no attribute 'backend'. 错误原因: 选择的应用后端类型不对. windows 上应用的后端类型有两种 ...
成功解决AttributeError: ‘NoneType‘ object has no attribute ‘shape‘
成功解决AttributeError: 'NoneType' object has no attribute 'shape' 目录解决问题解决思路解决方法解决问题 multiplier = [ ...
成功解决AttributeError: 'NoneType' object has no attribute '__array_interface__'
成功解决AttributeError: 'NoneType' object has no attribute '__array_interface__' 目录解决问题解决思路解决方法原因及思路 ...
AttributeError: 'NoneType' object has no attribute 'grid'报错解决方案
1问题描述: 当我们在使用tkinter时经常遇到AttributeError: 'NoneType' object has no attribute 'grid'的报错 2.原因分析: import ...
Python学习笔记：‘’AttributeError: NoneType object has no attribute‘’
前言最近在学习python,犯了很多低级错误,总结一下目录文章目录前言目录问题出处方案问题 AttributeError: 'NoneType' object has no attr ...
解决AttributeError AttributeError: 'NoneType' object has no attribute 'filename'
原因忘记上传文件表单需要加属性 enctype="multipart/form-data" 否则报错!AttributeError AttributeError: 'NoneTy ...

pyspark AttributeError: 'NoneType' object has no attribute 'setCallSite'

AttributeError: 'NoneType' object has no attribute 'setCallSite'

我草，是pyspark的bug。解决方法：

Description

pyspark AttributeError: 'NoneType' object has no attribute 'setCallSite'相关推荐

最新文章

热门文章