/*** A unit of execution. We have two kinds of Task's in Spark:**  - [[org.apache.spark.scheduler.ShuffleMapTask]]*  - [[org.apache.spark.scheduler.ResultTask]]** A Spark job consists of one or more stages. The very last stage in a job consists of multiple* ResultTasks, while earlier stages consist of ShuffleMapTasks. A ResultTask executes the task* and sends the task output back to the driver application. A ShuffleMapTask executes the task* and divides the task output to multiple buckets (based on the task's partitioner).** @param stageId id of the stage this task belongs to* @param partitionId index of the number in the RDD*/

/*** A running job in the DAGScheduler. Jobs can be of two types: a result job, which computes a* ResultStage to execute an action, or a map-stage job, which computes the map outputs for a* ShuffleMapStage before any downstream stages are submitted. The latter is used for adaptive* query planning, to look at map output statistics before submitting later stages. We distinguish* between these two types of jobs using the finalStage field of this class.** Jobs are only tracked for "leaf" stages that clients directly submitted, through DAGScheduler's* submitJob or submitMapStage methods. However, either type of job may cause the execution of* other earlier stages (for RDDs in the DAG it depends on), and multiple jobs may share some of* these previous stages. These dependencies are managed inside DAGScheduler.** @param jobId A unique ID for this job.* @param finalStage The stage that this job computes (either a ResultStage for an action or a*   ShuffleMapStage for submitMapStage).* @param callSite Where this job was initiated in the user's program (shown on UI).* @param listener A listener to notify if tasks in this job finish or the job fails.* @param properties Scheduling properties attached to the job, such as fair scheduler pool name.*/

/*** A stage is a set of parallel tasks all computing the same function that need to run as part* of a Spark job, where all the tasks have the same shuffle dependencies. Each DAG of tasks run* by the scheduler is split up into stages at the boundaries where shuffle occurs, and then the* DAGScheduler runs these stages in topological order.** Each Stage can either be a shuffle map stage, in which case its tasks' results are input for* other stage(s), or a result stage, in which case its tasks directly compute a Spark action* (e.g. count(), save(), etc) by running a function on an RDD. For shuffle map stages, we also* track the nodes that each output partition is on.** Each Stage also has a firstJobId, identifying the job that first submitted the stage.  When FIFO* scheduling is used, this allows Stages from earlier jobs to be computed first or recovered* faster on failure.** Finally, a single stage can be re-executed in multiple attempts due to fault recovery. In that* case, the Stage object will track multiple StageInfo objects to pass to listeners or the web UI.* The latest one will be accessible through latestInfo.** @param id Unique stage ID* @param rdd RDD that this stage runs on: for a shuffle map stage, it's the RDD we run map tasks*   on, while for a result stage, it's the target RDD that we ran an action on* @param numTasks Total number of tasks in stage; result stages in particular may not need to*   compute all partitions, e.g. for first(), lookup(), and take().* @param parents List of stages that this stage depends on (through shuffle dependencies).* @param firstJobId ID of the first job this stage was part of, for FIFO scheduling.* @param callSite Location in the user program associated with this stage: either where the target*   RDD was created, for a shuffle map stage, or where the action for a result stage was called.*/

/*** Spark executor, backed by a threadpool to run tasks.** This can be used with Mesos, YARN, and the standalone scheduler.* An internal RPC interface (at the moment Akka) is used for communication with the driver,* except in the case of Mesos fine-grained mode.*/

/*** A Resilient Distributed Dataset (RDD), the basic abstraction in Spark. Represents an immutable,* partitioned collection of elements that can be operated on in parallel. This class contains the* basic operations available on all RDDs, such as `map`, `filter`, and `persist`. In addition,* [[org.apache.spark.rdd.PairRDDFunctions]] contains operations available only on RDDs of key-value* pairs, such as `groupByKey` and `join`;* [[org.apache.spark.rdd.DoubleRDDFunctions]] contains operations available only on RDDs of* Doubles; and* [[org.apache.spark.rdd.SequenceFileRDDFunctions]] contains operations available on RDDs that* can be saved as SequenceFiles.* All operations are automatically available on any RDD of the right type (e.g. RDD[(Int, Int)]* through implicit.** Internally, each RDD is characterized by five main properties:**  - A list of partitions*  - A function for computing each split*  - A list of dependencies on other RDDs*  - Optionally, a Partitioner for key-value RDDs (e.g. to say that the RDD is hash-partitioned)*  - Optionally, a list of preferred locations to compute each split on (e.g. block locations for*    an HDFS file)** All of the scheduling and execution in Spark is done based on these methods, allowing each RDD* to implement its own way of computing itself. Indeed, users can implement custom RDDs (e.g. for* reading data from a new storage system) by overriding these functions. Please refer to the* [[http://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf Spark paper]] for more details* on RDD internals.*/

/*** An identifier for a partition in an RDD.*/
trait Partition extends Serializable {/*** Get the partition's index within its parent RDD*/def index: Int// A better default implementation of HashCodeoverride def hashCode(): Int = index
}

 /*** Choose a partitioner to use for a cogroup-like operation between a number of RDDs.** If any of the RDDs already has a partitioner, choose that one.** Otherwise, we use a default HashPartitioner. For the number of partitions, if* spark.default.parallelism is set, then we'll use the value from SparkContext* defaultParallelism, otherwise we'll use the max number of upstream partitions.** Unless spark.default.parallelism is set, the number of partitions will be the* same as the number of partitions in the largest upstream RDD, as this should* be least likely to cause out-of-memory errors.** We use two method parameters (rdd, others) to enforce callers passing at least 1 RDD.*/def defaultPartitioner(rdd: RDD[_], others: RDD[_]*): Partitioner = {val bySize = (Seq(rdd) ++ others).sortBy(_.partitions.size).reversefor (r <- bySize if r.partitioner.isDefined && r.partitioner.get.numPartitions > 0) {return r.partitioner.get}if (rdd.context.conf.contains("spark.default.parallelism")) {new HashPartitioner(rdd.context.defaultParallelism)} else {new HashPartitioner(bySize.head.partitions.size)}}