Spark provides three main locations to configure the system:

• Environment variables for launching Spark workers, which can be set either in your driver program or in theconf/spark-env.shscript.
• Java system properties, which control internal configuration parameters and can be set either programmatically (by callingSystem.setPropertybefore creating aSparkContext) or through theSPARK_JAVA_OPTSenvironment variable inspark-env.sh.
• Logging configuration, which is done throughlog4j.properties.

Spark提供了三种主要本地设置来配置系统：

• 环境变量 用来加载Spark的workers，可以在你的驱动程序或theconf/spark-env.shscript中设定。
• Java系统属性 控制内部配置参数，可以通过编程方式设置(通过在创建SparkContext之前调用System.setProperty)或者通过inspark-env.sh中的SPARK_JAVA_OPTS环境变量。
• 日志配置 通过log4j.properties来设置。

Environment Variables

Spark determines how to initialize the JVM on worker nodes, or even on the local node when you runspark-shell, by running theconf/spark-env.shscript in the directory where it is installed. This script does not exist by default in the Git repository, but but you can create it by copyingconf/spark-env.sh.template. Make sure that you make the copy executable.

Insidespark-env.sh, you must set at least the following two variables:

• SCALA_HOME, to point to your Scala installation, orSCALA_LIBRARY_PATHto point to the directory for Scala library JARs (if you install Scala as a Debian or RPM package, there is noSCALA_HOME, but these libraries are in a separate path, typically /usr/share/java; look forscala-library.jar).
• MESOS_NATIVE_LIBRARY, if you are running on a Mesos cluster.

环境变量

Spark决定worker节点上如何初始化JVM，当你在本地运行spark-shell时也是这样，通过运行Spark安装目录下的conf/spark-env.sh脚本即可启动spark-shell。该脚本在Git库中默认不存在，但你可以通过复制conf/spark-env.sh.template来创建一个。确保复制后的脚本有执行权限。

在spark-env.sh中，你必需至少设置下面两个变量：

• SCALA_HOME 指定你的Scala安装位置，或使用SCALA_LIBRARY_PATH指定Scala的jar库位置（如果你以Debian或RPM包方式安装Scala，无法设置SCALA_HOME，但库在一个特定目录里，通常是/usr/share/java，可以通过搜索scala-library.jar获取）。
• MESOS_NATIVE_LIBRARY 如果你正运行在一个Mesos集群上设置该项。

• SPARK_JAVA_OPTS, to add JVM options. This includes any system properties that you’d like to pass with-D.
• SPARK_CLASSPATH, to add elements to Spark’s classpath.
• SPARK_LIBRARY_PATH, to add search directories for native libraries.
• SPARK_MEM, to set the amount of memory used per node. This should be in the same format as the JVM’s -Xmx option, e.g.300mor1g. Note that this option will soon be deprecated in favor of thespark.executor.memorysystem property, so we recommend using that in new code.

Beware that if you do set these variables inspark-env.sh, they will override the values set by user programs, which is undesirable; if you prefer, you can choose to havespark-env.shset them only if the user program hasn’t, as follows:

if [ -z "$SPARK_JAVA_OPTS" ] ; thenSPARK_JAVA_OPTS="-verbose:gc"
fi

另外,这里有4个另外的用于控制执行的变量。这些变量需要在运行Job的驱动程序的上下文设置,而不是在spark-env.sh,因为这些变量将会被自动传递给workers。在每一个job里设置这些变量可以使得不同的job对这些变量有不同的配置。

• SPARK_JAVA_OPTS, 添加JVM选项。这包含了你用-D传递的一些系统属性。
• SPARK_CLASSPATH, 向Spark的classpath添加元素
• SPARK_LIBRARY_PATH, 添加本地库的搜索路径
• SPARK_MEM, 设置每一个节点使用的内存大小。这应该和JVM的-Xmx选项类似的形式，比如300m或者1g。需要注意为了推行spark.executor.memorysystem属性，这个选项很快将不被推荐使用,所以我们推荐你使用在代码中使用spark.executor.memorysystem。

注意,如果在spark-env.sh文件中设置这些变量，他们将会被用户程序中设置的值所覆盖。如果你愿意，你可以选择只有在用户程序中没有设置的情况下在spark-env.sh中如下设置他们：

if [ -z "$SPARK_JAVA_OPTS" ] ; thenSPARK_JAVA_OPTS="-verbose:gc"
fi

System Properties

To set a system property for configuring Spark, you need to either pass it with a -D flag to the JVM (for examplejava -Dspark.cores.max=5 MyProgram) or callSystem.setPropertyin your code before creating your Spark context, as follows:

System.setProperty("spark.cores.max", "5")
val sc = new SparkContext(...)

Most of the configurable system properties control internal settings that have reasonable default values. However, there are at least five properties that you will commonly want to control:

系统属性

为了设置Spark的系统属性，你需要给JVM传递一个-D标志的参数 (比如，java -Dspark.cores.max=5 MyProgram) 或者是在你创建Spark上下文的时候调用System.setProperty()方法，就像下面这样:

System.setProperty("spark.cores.max", "5")
val sc = new SparkContext(...)

大多数的可配置系统会在内部设置一个比较合理的默认值。但是，至少下面的5个属性你应该自己去设置的：

Configuring Logging

Spark uses log4j for logging. You can configure it by adding alog4j.propertiesfile in theconfdirectory. One way to start is to copy the existinglog4j.properties.templatelocated there.

日志配置

Spark使用 log4j 作为它的日志实现。 你可以在conf文件夹中增加一个log4j.properties配置文件去配置日志。开始的时候，你可以复制conf文件夹中已经存在一个log4j.properties.template模板，重命名为log4j.properties。