pyspark:

AttributeError: 'NoneType' object has no attribute 'setCallSite'

我草，是pyspark的bug。解决方法：

print("Approximately joining on distance smaller than 0.6:")distance_min = model.approxSimilarityJoin(imsi_proc_df, imsi_proc_df, 1e6, distCol="JaccardDistance") \.select(col("datasetA.id").alias("idA"),col("datasetB.id").alias("idB"),col("JaccardDistance")) #.filter("idA=idB")print(distance_min.show())print("*"*88)print(imsi_proc_df.show())key = Vectors.sparse(53, [1, 3], [1.0, 1.0])print(model.approxNearestNeighbors(imsi_proc_df, key, 2).show())print("start calculate find botnet!")print("*"*99)print("time start:", time.time())print(type(distance_min), dir(distance_min))print(dir(distance_min.toLocalIterator))

    ############################################## add this line to solvedistance_min.sql_ctx.sparkSession._jsparkSession = spark_app._jsparkSessiondistance_min._sc = spark_app._sc    #############################################similarity_val_rdd = distance_min.toLocalIterator #.collect()print("time end:", time.time())print(similarity_val_rdd)print("*"*99)try:G = ConnectedGraph()ddos_ue_list = []for item in similarity_val_rdd():imsi, imsi2, jacard_similarity_val = item["idA"], item["idB"], item["JaccardDistance"]print("???", imsi, imsi2, jacard_similarity_val)

Description

reproducing the bug from the example in the documentation:

import pyspark
from pyspark.ml.linalg import Vectors
from pyspark.ml.stat import Correlation
spark = pyspark.sql.SparkSession.builder.getOrCreate()
dataset = [[Vectors.dense([1, 0, 0, -2])],[Vectors.dense([4, 5, 0, 3])],[Vectors.dense([6, 7, 0, 8])],[Vectors.dense([9, 0, 0, 1])]]
dataset = spark.createDataFrame(dataset, ['features'])
df = Correlation.corr(dataset, 'features', 'pearson') df.collect()

This produces the following stack trace:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-92-e7889fa5d198> in <module>()11 dataset = spark.createDataFrame(dataset, ['features'])12 df = Correlation.corr(dataset, 'features', 'pearson')
---> 13 df.collect()/opt/spark/python/pyspark/sql/dataframe.py in collect(self)530         [Row(age=2, name=u'Alice'), Row(age=5, name=u'Bob')] 531 """ --> 532 with SCCallSiteSync(self._sc) as css: 533 sock_info = self._jdf.collectToPython() 534 return list(_load_from_socket(sock_info, BatchedSerializer(PickleSerializer()))) /opt/spark/python/pyspark/traceback_utils.py in __enter__(self) 70 def __enter__(self): 71 if SCCallSiteSync._spark_stack_depth == 0: ---> 72 self._context._jsc.setCallSite(self._call_site) 73 SCCallSiteSync._spark_stack_depth += 1 74 AttributeError: 'NoneType' object has no attribute 'setCallSite'

Analysis:

Somehow the dataframe properties `df.sql_ctx.sparkSession._jsparkSession`, and `spark._jsparkSession` do not match with the ones available in the spark session.

The following code fixes the problem (I hope this helps you narrowing down the root cause)

df.sql_ctx.sparkSession._jsparkSession = spark._jsparkSession
df._sc = spark._scdf.collect()>>> [Row(pearson(features)=DenseMatrix(4, 4, [1.0, 0.0556, nan, 0.4005, 0.0556, 1.0, nan, 0.9136, nan, nan, 1.0, nan, 0.4005, 0.9136, nan, 1.0], False))]

转载于:https://www.cnblogs.com/bonelee/p/10976253.html

pyspark AttributeError: 'NoneType' object has no attribute 'setCallSite'相关推荐

  1. AttributeError: 'NoneType' object has no attribute 'sc' 解决方法（二）

    上一次本以为可以解决了这个问题,然而并没有那么地简单.博主最近在edx网站学习pyspark,想打一下视频上的代码,结果报错了,依旧是报了"AttributeError:'NoneType' ...

  2. Traceback (most recent call last): File AttributeError: 'NoneType' object has no attribute 'group'

    Traceback (most recent call last):File "<stdin>", line 1, in <module> Attribut ...

  3. Keras问题“AttributeError: 'NoneType' object has no attribute 'update”解决

    BUG 在使用Keras训练模型时,在每个epoch完成后save_model时会报错 "AttributeError: 'NoneType' object has no attribute ...

  4. Pywinauto 应用后端类型选择错误：AttributeError: ‘NoneType‘ object has no attribute ‘backend‘. 原因及解决办法

    AttributeError: 'NoneType' object has no attribute 'backend'. 错误原因: 选择的应用后端类型不对. windows 上应用的后端类型有两种 ...

  5. 成功解决AttributeError: ‘NoneType‘ object has no attribute ‘shape‘

    成功解决AttributeError: 'NoneType' object has no attribute 'shape' 目录 解决问题 解决思路 解决方法 解决问题 multiplier = [ ...

  6. 成功解决AttributeError: 'NoneType' object has no attribute '__array_interface__'

    成功解决AttributeError: 'NoneType' object has no attribute '__array_interface__' 目录 解决问题 解决思路 解决方法 原因及思路 ...

  7. AttributeError: 'NoneType' object has no attribute 'grid'报错解决方案

    1问题描述: 当我们在使用tkinter时经常遇到AttributeError: 'NoneType' object has no attribute 'grid'的报错 2.原因分析: import ...

  8. Python学习笔记：‘’AttributeError: NoneType object has no attribute‘’

    前言 最近在学习python,犯了很多低级错误,总结一下 目录 文章目录 前言 目录 问题 出处 方案 问题 AttributeError: 'NoneType' object has no attr ...

  9. 解决AttributeError AttributeError: 'NoneType' object has no attribute 'filename'

    原因忘记上传文件 表单需要加属性 enctype="multipart/form-data" 否则报错!AttributeError AttributeError: 'NoneTy ...

最新文章

  1. 用JAVA语言创建链表的方法
  2. 使用IDEA编写Python程序，插件安装python插件，安装python SDK运行Python程序
  3. RabbitMQ基本概念（三）-Centos7下安装RabbitMQ3.6.1
  4. PyTorch 和 TensorFlow 引领未来
  5. CloudStack 4.4学习总结之注册ISO
  6. 高可用web架构: LVS+keepalived+nginx+apache+php+eaccelerator（+nfs可选 可不选）
  7. JS验证身份证、营业执照、组织机构代码等
  8. 十大著名黑客—— 凯文-米特尼克
  9. 苹果CMS V10后台登录验证码错误解决方法
  10. ①读后感之《当我们谈论爱情时我们在谈论什么》┊(美)雷蒙德.卡佛
  11. pip‘s dependency resolver does not currently take into account 解决办法
  12. axure sketch 对比_Sketch to Axure RP插件下载
  13. Non-UTF-8 code starting with ‘\xb5‘ in file D:\eclipse\Python\test\__init__.
  14. 智能手环guard日志获取-兔盯云
  15. form-create-desniger 自定义拖拽表单
  16. iOS APP下载安装时，如果出现此时无法下载安装APP的字样时，一些解决思路
  17. python爬取桌面壁纸（初学者），一看就会
  18. matplotlib 基础_子图创建
  19. 新学：DES加密 小知识点（S盒，初始置换）
  20. Spinner ArrayAdapter

热门文章

  1. ad域推送软件_Python3.6.4 软件安装教程
  2. python3.4.4怎么打代码_Python3.4里面__name和__main__的作用
  3. 在html表单中设置数组的方法是什么,js如何处理表单中的数组方式？
  4. linux线程怎样实时性,高效轻型线程怎么提高Linux实时性能？
  5. opencore0.6.3_大杨随笔2020.11.3
  6. python快速入门第3版 数据_Python 快速入门 第3版
  7. div.php织梦自定义表判断不能为空,织梦自定义表单字段为必填项的教程
  8. git show查看本地文件的修改部分，git diff 如何比较文件，git grep 搜索
  9. React Native开发错误警告处理总结（已解决 ！持续更新）
  10. Java性能优化推荐书！JAVA-注解的基本原理