由于近期准备深入研究一下Spark的核心源码,所以开了这一系列用来记录自己研究spark源码的过程!

想要读源码,那么第一步肯定导入spark源码啦(笔者使用的是IntelliJ IDEA),在网上找了一圈,尝试了好几种方法都没有成功,最终通过自己摸索出了一种非常简单的方式(只需要两步即可!

环境要求

  1. IntelliJ IDEA(Community版本即可)
  2. maven(当然jdk是不可少的)

具体信息如下:

C:\Users\Administrator>mvn -version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)
Maven home: D:\java\apache-maven-3.3.9\bin\..
Java version: 1.8.0_151, vendor: Oracle Corporation
Java home: D:\java\jdk-1.8u151\jre
Default locale: zh_CN, platform encoding: GBK
OS name: "windows 7", version: "6.1", arch: "amd64", family: "dos"

  

顺便贴一下maven的settings.xml

<?xml version="1.0" encoding="UTF-8"?><!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
--><!--| This is the configuration file for Maven. It can be specified at two levels:||  1. User Level. This settings.xml file provides configuration for a single user,|                 and is normally provided in ${user.home}/.m2/settings.xml.||                 NOTE: This location can be overridden with the CLI option:||                 -s /path/to/user/settings.xml||  2. Global Level. This settings.xml file provides configuration for all Maven|                 users on a machine (assuming they're all using the same Maven|                 installation). It's normally provided in|                 ${maven.conf}/settings.xml.||                 NOTE: This location can be overridden with the CLI option:||                 -gs /path/to/global/settings.xml|| The sections in this sample file are intended to give you a running start at| getting the most out of your Maven installation. Where appropriate, the default| values (values used when the setting is not specified) are provided.||-->
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd"><!-- localRepository| The path to the local repository maven will use to store artifacts.|| Default: ${user.home}/.m2/repository<localRepository>D:\Mavenworkspace\m2\repository</localRepository>-->
<localRepository>D:\java\mavenRepository</localRepository><!-- interactiveMode| This will determine whether maven prompts you when it needs input. If set to false,| maven will use a sensible default value, perhaps based on some other setting, for| the parameter in question.|| Default: true<interactiveMode>true</interactiveMode>--><!-- offline| Determines whether maven should attempt to connect to the network when executing a build.| This will have an effect on artifact downloads, artifact deployment, and others.|| Default: false<offline>false</offline>--><!-- pluginGroups| This is a list of additional group identifiers that will be searched when resolving plugins by their prefix, i.e.| when invoking a command line like "mvn prefix:goal". Maven will automatically add the group identifiers| "org.apache.maven.plugins" and "org.codehaus.mojo" if these are not already contained in the list.|--><pluginGroups><!-- pluginGroup| Specifies a further group identifier to use for plugin lookup.<pluginGroup>com.your.plugins</pluginGroup>--></pluginGroups><!-- proxies| This is a list of proxies which can be used on this machine to connect to the network.| Unless otherwise specified (by system property or command-line switch), the first proxy| specification in this list marked as active will be used.|--><proxies><!-- proxy| Specification for one proxy, to be used in connecting to the network.|<proxy><id>optional</id><active>true</active><protocol>http</protocol><username>proxyuser</username><password>proxypass</password><host>proxy.host.net</host><port>80</port><nonProxyHosts>local.net|some.host.com</nonProxyHosts></proxy>--></proxies><!-- servers| This is a list of authentication profiles, keyed by the server-id used within the system.| Authentication profiles can be used whenever maven must make a connection to a remote server.|--><servers><!-- server| Specifies the authentication information to use when connecting to a particular server, identified by| a unique name within the system (referred to by the 'id' attribute below).|| NOTE: You should either specify username/password OR privateKey/passphrase, since these pairings are|       used together.|<server><id>deploymentRepo</id><username>repouser</username><password>repopwd</password></server>--><!-- Another sample, using keys to authenticate.<server><id>siteServer</id><privateKey>/path/to/private/key</privateKey><passphrase>optional; leave empty if not used.</passphrase></server>--></servers><!-- mirrors| This is a list of mirrors to be used in downloading artifacts from remote repositories.|| It works like this: a POM may declare a repository to use in resolving certain artifacts.| However, this repository may have problems with heavy traffic at times, so people have mirrored| it to several places.|| That repository definition will have a unique id, so we can create a mirror reference for that| repository, to be used as an alternate download site. The mirror site will be the preferred| server for that repository.|--><mirrors><!-- mirror| Specifies a repository mirror site to use instead of a given repository. The repository that| this mirror serves has an ID that matches the mirrorOf element of this mirror. IDs are used| for inheritance and direct lookup purposes, and must be unique across the set of mirrors.|<mirror><id>mirrorId</id><mirrorOf>repositoryId</mirrorOf><name>Human Readable Name for this Mirror.</name><url>http://my.repository.com/repo/path</url></mirror>--><mirror><id>nexus-osc</id><mirrorOf>central</mirrorOf><name>Nexus osc</name><url>http://maven.aliyun.com/nexus/content/groups/public/</url></mirror><mirror><id>osc_thirdparty</id><mirrorOf>thirdparty</mirrorOf><url>http://maven.aliyun.com/nexus/content/repositories/central</url></mirror></mirrors><!-- profiles| This is a list of profiles which can be activated in a variety of ways, and which can modify| the build process. Profiles provided in the settings.xml are intended to provide local machine-| specific paths and repository locations which allow the build to work in the local environment.|| For example, if you have an integration testing plugin - like cactus - that needs to know where| your Tomcat instance is installed, you can provide a variable here such that the variable is| dereferenced during the build process to configure the cactus plugin.|| As noted above, profiles can be activated in a variety of ways. One way - the activeProfiles| section of this document (settings.xml) - will be discussed later. Another way essentially| relies on the detection of a system property, either matching a particular value for the property,| or merely testing its existence. Profiles can also be activated by JDK version prefix, where a| value of '1.4' might activate a profile when the build is executed on a JDK version of '1.4.2_07'.| Finally, the list of active profiles can be specified directly from the command line.|| NOTE: For profiles defined in the settings.xml, you are restricted to specifying only artifact|       repositories, plugin repositories, and free-form properties to be used as configuration|       variables for plugins in the POM.||--><profiles><!-- profile| Specifies a set of introductions to the build process, to be activated using one or more of the| mechanisms described above. For inheritance purposes, and to activate profiles via <activatedProfiles/>| or the command line, profiles have to have an ID that is unique.|| An encouraged best practice for profile identification is to use a consistent naming convention| for profiles, such as 'env-dev', 'env-test', 'env-production', 'user-jdcasey', 'user-brett', etc.| This will make it more intuitive to understand what the set of introduced profiles is attempting| to accomplish, particularly when you only have a list of profile id's for debug.|| This profile example uses the JDK version to trigger activation, and provides a JDK-specific repo.<profile><id>jdk-1.4</id><activation><jdk>1.4</jdk></activation><repositories><repository><id>jdk14</id><name>Repository for JDK 1.4 builds</name><url>http://www.myhost.com/maven/jdk14</url><layout>default</layout><snapshotPolicy>always</snapshotPolicy></repository></repositories></profile>-->
<profile><profile>         <id>jdk-1.8</id><activation><activeByDefault>true</activeByDefault><jdk>1.8</jdk></activation><properties><maven.compiler.source>1.8</maven.compiler.source><maven.compiler.target>1.8</maven.compiler.target><maven.compiler.compilerVersion>1.8</maven.compiler.compilerVersion></properties><repositories><repository><id>nexus</id><name>local private nexus</name><url>http://maven.aliyun.com/nexus/content/repositories/central</url><releases><enabled>true</enabled></releases><snapshots><enabled>false</enabled></snapshots></repository></repositories><pluginRepositories><pluginRepository><id>nexus</id><name>local private nexus</name><url>http://maven.aliyun.com/nexus/content/repositories/central</url><releases><enabled>true</enabled></releases><snapshots><enabled>false</enabled></snapshots></pluginRepository></pluginRepositories></profile><profile>         <id>osc</id><activation><activeByDefault>true</activeByDefault></activation><repositories>            <repository><id>osc</id><url>http://maven.aliyun.com/nexus/content/repositories/central</url></repository><repository><id>osc_thirdparty</id><url>http://maven.aliyun.com/nexus/content/repositories/central</url></repository></repositories><pluginRepositories><pluginRepository><id>osc</id><url>http://maven.aliyun.com/nexus/content/repositories/central</url></pluginRepository></pluginRepositories></profile></profile><!--| Here is another profile, activated by the system property 'target-env' with a value of 'dev',| which provides a specific path to the Tomcat instance. To use this, your plugin configuration| might hypothetically look like:|| ...| <plugin>|   <groupId>org.myco.myplugins</groupId>|   <artifactId>myplugin</artifactId>||   <configuration>|     <tomcatLocation>${tomcatPath}</tomcatLocation>|   </configuration>| </plugin>| ...|| NOTE: If you just wanted to inject this configuration whenever someone set 'target-env' to|       anything, you could just leave off the <value/> inside the activation-property.|<profile><id>env-dev</id><activation><property><name>target-env</name><value>dev</value></property></activation><properties><tomcatPath>/path/to/tomcat/instance</tomcatPath></properties></profile>--></profiles><!-- activeProfiles| List of profiles that are active for all builds.|<activeProfiles><activeProfile>alwaysActiveProfile</activeProfile><activeProfile>anotherAlwaysActiveProfile</activeProfile></activeProfiles>-->
</settings>

settings.xml

好了,一旦环境准备就绪,那就速战速决吧!

第一步:从github上下载源代码

先选择你想要阅读的spark版本,笔者这里选择的是spark1.3版本

接着直接下载zip包到本地解压(当然也可以使用git拉下来啦)

第二步:使用IDEA导入spark源码

打开你的IntelliJ IDEA ,File -> Open 选中你源码解压后的文件夹即可! (不需要使用Import

到这里基本已经大功告成!接下来只需要等待maven解决各种依赖即可(大概需要半个小时,大家耐心一点)

成功后的界面如下(提示:可以使用ctrl + N 搜索你想要阅读的类文件):

转载于:https://www.cnblogs.com/LiCheng-/p/8128003.html

Spark源码剖析(一):如何将spark源码导入到IDEA中相关推荐

  1. java.lang 源码剖析_java.lang.Void类源码解析

    在一次源码查看ThreadGroup的时候,看到一段代码,为以下: /* * @throws NullPointerException if the parent argument is {@code ...

  2. STL源码剖析 学C语言,STL源码剖析(一)

    SLT简介 STL(Standard Template Library),即标准模板库,是一个高效的C++程序库.包含了诸多在计算机科学领域里经常使用的基本数据结构和基本算法.为广大C++程序员们提供 ...

  3. Oracle调用接口(OCI)源码剖析(2):执行SQL语句并获取结果

    概述 接着上一篇文章<Oracle调用接口(OCI)源码剖析(1):创建数据库连接>,我们继续对OCI中执行SQL语句并获取结果的源码进行剖析.该操作主要是由两个函数完成的:CDbExec ...

  4. Spring中AOP源码剖析

    Spring中AOP源码剖析 关键词 aop的增强发生在后置处理器中(没有循环依赖) 最终增强是通过 递归调用 ,层层增强 一.环境准备 1.1 bean和接口 public class AopBea ...

  5. 阿里中间件seata源码剖析六:TCC模式中2阶段提交实现

    目录 TM通知TC事务状态 TC通知RM分支事务提交 RM处理TC提交事务请求 总结 上篇文章中,我们以TCC模式的demo为例,讲解了seata中全局事务的开启.在这个demo中,TM作为一个全局事 ...

  6. is_heap和is_heap_until 用法与源码剖析

    一:用法示例 is_heap一共两个重载: default (1) template <class RandomAccessIterator>   bool is_heap ( Rando ...

  7. 【有点狂的手撕STL】STL源码剖析精读 000

    STL源码剖析精读 前言 通过刷题感受到了C++中STL的妙用,十分的想要提高自己对于STL的理解以及运用能力,因此开设此专栏,并希望能够带领大家一起感受C++中STL的魅力. 一.STL简介 STL ...

  8. spark 源码分析之十八 -- Spark存储体系剖析

    本篇文章主要剖析BlockManager相关的类以及总结Spark底层存储体系. 总述 先看 BlockManager相关类之间的关系如下: 我们从NettyRpcEnv 开始,做一下简单说明. Ne ...

  9. Spark存储机制源码剖析

    一.Shuffle结果的写入和读取 通过之前的文章Spark源码解读之Shuffle原理剖析与源码分析我们知道,一个Shuffle操作被DAGScheduler划分为两个stage,第一个stage是 ...

  10. Spark一级资源调度Shedule机制及SpreadOut模式源码深入剖析

    本套系列博客从真实商业环境抽取案例进行总结和分享,并给出Spark源码解读及商业实战指导,请持续关注本套博客.版权声明:本套Spark源码解读及商业实战归作者(秦凯新)所有,禁止转载,欢迎学习. Sp ...

最新文章

  1. 上市 10 天就遭破解!AirTag 还能买吗?
  2. 【收藏】蘑菇博客mogu_blog项目文档
  3. Pytorch:GAN生成对抗网络实现MNIST手写数字的生成
  4. 创业产品经理需要懂技术吗?
  5. python 获取唯一值_从Python列表中获取唯一值
  6. 10个python爬虫入门实例(小结)
  7. 《系统集成项目管理》第一章 信息化知识
  8. 完全背包问题(模板)
  9. 【人性】豆豆三部曲之《背叛》、《遥远的救世主》和《天幕红尘》
  10. 你真的了解“手机端的 C/S架构 向 B/S架构 迁移”吗
  11. 为什么打开edge浏览器,就出来qq导航,hao123页面等等!
  12. 《卓有成效的管理者》——学习心得(五)
  13. 对 Mac 原生终端模拟器的增强
  14. 山东省初中计算机考试分数段,山东中考等级录取
  15. java duck的屏幕保护程序
  16. nyoj 125 盗梦空间
  17. DASH视频系统(服务器播放器)搭建
  18. mysql视图高级设置_MySQL 视图高级属性知多少?
  19. Windows下Nextcloud文件同步备份(使用robocopy)
  20. windows 下 MAC地址绑定

热门文章

  1. 微软宣布 Windows 10X:用于可折叠 PC 的新操作系统
  2. SQL Server和Oracle的差异
  3. Linux调出git页面,Linux 显示 git 分支 及 完整路径
  4. java count 在哪一类里_count 是java关键字吗
  5. 全文搜索引擎有哪些?_你想要拥有自己的搜索引擎吗?
  6. php mysql 网站_实验六:MYSQL+PHP的网站搭建
  7. null索引表键值_MySQL 的B+树索引
  8. linux cnc 软件手册,最新LINUXCNC源程序原理说明.docx
  9. hht时频谱 matlab 乱序_MATLAB信号频谱分析FFT详解
  10. html可折叠边栏,html – 仅使用CSS的可折叠灵活宽边栏