一、问题描述

创建sparkcontext和SparkSession，连接spark集群时报错，如题ValueError: Cannot run multiple SparkContexts at once; existing SparkContext。

from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql.functions import min, max
from pyspark.sql.functions import monotonically_increasing_id, lit, col, struct
from pyspark.ml.evaluation import RegressionEvaluator
from pyspark.ml.recommendation import ALS
from pyspark.sql.functions import udf
from pyspark import SparkContext, SparkConf
from pyspark.sql.types import StructType, StructField, IntegerType, StringType, ArrayType, DoubleType, FloatType
from pyspark.ml.feature import Word2Vec
import os
import pandas as pd
import numpy as np
from tqdm import tqdm
import heapq# 创建 sparkcontext
# sc.stop()
# local[*]时最大线程数
conf = SparkConf().setAppName("data_process_first").setMaster("local[*]")
sc = SparkContext(conf=conf)# 创建SparkSession，连接spark集群
spark = SparkSession.builder.appName('mypyspark_test1') \.master("local")\.config("spark.driver.memory","30G")\.config("spark.executor.memory","30G")\.getOrCreate()

二、解决方法

因为之前已经启动了sparkContext，不能重复启动，所以在SparkConf()上一句运行sc.stop()即可，即关闭spark集群。

sparkContext即完成了一个spark集群的连接，可以在该集群上创建RDD和广播变量。创建时至少传入master（上面栗子local[*]是使用当前所有的线程运行）和appname两个参数。

Reference

[1] Spark启动时的master参数以及Spark的部署方式

解决ValueError: Cannot run multiple SparkContexts at once； existing SparkContext相关推荐

「golang」panic: commands out of sync. Did you run multiple statements at once
[问题现象] go使用go-sql-driver连接mariaDB,如下,提示: panic: commands out of sync. Did you run multiple statement ...
成功解决ValueError: Data is not binary and pos_label is not specified
成功解决ValueError: Data is not binary and pos_label is not specified 目录解决问题解决思路解决方法解决问题 raise Value ...
成功解决ValueError: DataFrame.dtypes for data must be int, float or bool.Did not expect the data types
成功解决ValueError: DataFrame.dtypes for data must be int, float or bool. Did not expect the data types ...
成功解决ValueError: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216 from C h
成功解决ValueError: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216 from C h ...
成功解决ValueError: Unable to add relationship because child variable ‘ID‘ in ‘cats_df‘ is also its inde
成功解决ValueError: Unable to add relationship because child variable 'ID' in 'cats_df' is also its inde ...
成功解决ValueError: fill value must be in categories
成功解决ValueError: fill value must be in categories 目录解决问题解决思路解决方法解决问题 ValueError: fill value must ...
成功解决 ValueError: fill value must be in categories
成功解决 ValueError: fill value must be in categories 目录解决问题解决思路解决方法解决问题 raise ValueError("fill ...
成功解决ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() o
成功解决ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() o ...
成功解决ValueError: Shape of passed values is (33, 1), indices imply (33, 2)
成功解决ValueError: Shape of passed values is (33, 1), indices imply (33, 2) 目录解决问题解决思路解决方法解决问题 Valu ...
成功解决ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any(
成功解决ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any( ...

解决ValueError: Cannot run multiple SparkContexts at once； existing SparkContext

一、问题描述

二、解决方法

Reference

解决ValueError: Cannot run multiple SparkContexts at once； existing SparkContext相关推荐

最新文章

热门文章