一、问题描述

from pyspark.sql.types import StringType@udf(returnType = StringType())
def bad_funify(s):return s + " is fun!"countries2 = spark.createDataFrame([("Thailand", 3), (None, 4)], ["country", "id"])
countries2.withColumn("fun_country", bad_funify("country")).show()

用一个udf想让df（有country和id两个字段）生成新的一列fun_country（内容是字符串，内容为country xx is fun），但是df中有的country字段内容没有数据（注意类型是None而不是null），结果报错如下：

PythonException: An exception was thrown from the Python worker. Please see the stack trace below.
Traceback (most recent call last):File "/usr/lib/spark-current/python/lib/pyspark.zip/pyspark/worker.py", line 619, in mainprocess()File "/usr/lib/spark-current/python/lib/pyspark.zip/pyspark/worker.py", line 611, in processserializer.dump_stream(out_iter, outfile)File "/usr/lib/spark-current/python/lib/pyspark.zip/pyspark/serializers.py", line 211, in dump_streamself.serializer.dump_stream(self._batched(iterator), stream)File "/usr/lib/spark-current/python/lib/pyspark.zip/pyspark/serializers.py", line 132, in dump_streamfor obj in iterator:File "/usr/lib/spark-current/python/lib/pyspark.zip/pyspark/serializers.py", line 200, in _batchedfor item in iterator:File "/usr/lib/spark-current/python/lib/pyspark.zip/pyspark/worker.py", line 452, in mapperresult = tuple(f(*[a[o] for o in arg_offsets]) for (arg_offsets, f) in udfs)File "/usr/lib/spark-current/python/lib/pyspark.zip/pyspark/worker.py", line 452, in <genexpr>result = tuple(f(*[a[o] for o in arg_offsets]) for (arg_offsets, f) in udfs)File "/usr/lib/spark-current/python/lib/pyspark.zip/pyspark/worker.py", line 87, in <lambda>return lambda *a: f(*a)File "/usr/lib/spark-current/python/lib/pyspark.zip/pyspark/util.py", line 74, in wrapperreturn f(*args, **kwargs)File "<ipython-input-1051-5a6c51e7c332>", line 5, in bad_funify
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

二、解决方案

这是个很蠢的问题。其实如果country为空值时，fun_country应该也是空的，所以就简单加多个判断的逻辑即可。修改udf为good_funity后：

@udf(returnType=StringType())
def good_funify(s):return None if s == None else s + " is fun!"
countries2.withColumn("fun_country", good_funify("country")).show()+--------+---+----------------+
| country| id|     fun_country|
+--------+---+----------------+
|Thailand|  3|Thailand is fun!|
|    null|  4|            null|
+--------+---+----------------+

Reference

[1] Navigating None and null in PySpark

解决报错TypeError：unsupported operand type(s) for +: ‘NoneType‘ and ‘str‘相关推荐

apex安装报错：TypeError: unsupported operand type(s) for +: ‘NoneType‘ and ‘str‘
参考解决方法:TypeError: unsupported operand type(s) for +: 'NoneType' and 'str' · Issue #990 · NVIDIA/apex ...
解决pip安装时出现报错TypeError: unsupported operand type(s) for -=: ‘Retry‘ and ‘int‘
我在Linux下使用pip安装时出现报错: Exception: Traceback (most recent call last):File "/usr/lib/python2.7/dis ...
python报错TypeError: unsupported operand type(s) for -: ‘decimal.Decimal‘ and ‘float‘的解决方法
问题描述在编写python代码时,进行小数位相减时出现 TypeError: unsupported operand type(s) for -: 'decimal.Decimal' and 'fl ...
预测数据时数据类型是object导致报错TypeError: unsupported operand type(s) for -: ‘str‘ and ‘float‘
解决方法更换数据类型: data:pd.DataFrame = data.astype('int64') # 或是: data:pd.DataFrame = data.astype('float') ...
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
遇到这个问题,因为在文件中写入中文,目前的做法是open时指定encoding为utf-8,问题解决. 修改文件名为中文也有类似问题,原因是系统默认不支持中文.
opencv小笔记（TypeError: unsupported operand type(s) for +: ‘NoneType‘ and ‘NoneType‘）
今天在学习OpenCV的算数操作时,进行图像加法时,运行了下列程序 import numpy as np import cv2 as cv import matplotlib.pyplot as pl ...
成功解决TypeError: unsupported operand type(s) for %: 'NoneType' and 'dict'
成功解决TypeError: unsupported operand type(s) for %: 'NoneType' and 'dict' 目录解决问题解决思路解决方法解决问题 TypeE ...
解决pandas（Python）的报错：unsupported operand type(s) for -: ‘datetime.date’ and ‘Timestamp’
在使用Pandas包的时候,遇到时间加减出现的报错 'datetime.date'是datetime的一种时间格式: 'Timestamp'(注意是大写的字母)是Pandas的一种时间格式. 这两个虽 ...
Python3报错：TypeError: unsupported operand type(s) for +: ‘int‘ and ‘str‘
其实错误提示已经很明确了,"类型错误:不支持操作类型为整数和字符串",这里需要解释的最关键的东西是"+","+"在python中有两个作用, ...

解决报错TypeError：unsupported operand type(s) for +: ‘NoneType‘ and ‘str‘

文章目录

一、问题描述

二、解决方案

Reference

解决报错TypeError：unsupported operand type(s) for +: ‘NoneType‘ and ‘str‘相关推荐

最新文章

热门文章