安装虚拟机管理软件vagrant:

https://www.vagrantup.com/downloads.html

安装VirtualBox, vagrant默认使用VirtualBox:

https://www.virtualbox.org/wiki/Downloads

使用windows的powershell:

初始化进入centos

PS C:\Users\geng\env> vagrant centos/7
PS C:\Users\geng\env> vagrant init centos/7
PS C:\Users\geng\env> vagrant up

PS C:\Users\geng> cd .\env\
PS C:\Users\geng\env>
PS C:\Users\geng\env>
PS C:\Users\geng\env> ls
PS C:\Users\geng\env> vagrant centos/7
Usage: vagrant [options] <command> [<args>]-v, --version                    Print the version and exit.-h, --help                       Print this help.Common commands:box             manages boxes: installation, removal, etc.cloud           manages everything related to Vagrant Clouddestroy         stops and deletes all traces of the vagrant machineglobal-status   outputs status Vagrant environments for this userhalt            stops the vagrant machinehelp            shows the help for a subcommandinit            initializes a new Vagrant environment by creating a Vagrantfileloginpackage         packages a running vagrant environment into a boxplugin          manages plugins: install, uninstall, update, etc.port            displays information about guest port mappingspowershell      connects to machine via powershell remotingprovision       provisions the vagrant machinepush            deploys code in this environment to a configured destinationrdp             connects to machine via RDPreload          restarts vagrant machine, loads new Vagrantfile configurationresume          resume a suspended vagrant machinesnapshot        manages snapshots: saving, restoring, etc.ssh             connects to machine via SSHssh-config      outputs OpenSSH valid configuration to connect to the machinestatus          outputs status of the vagrant machinesuspend         suspends the machineup              starts and provisions the vagrant environmentupload          upload to machine via communicatorvalidate        validates the Vagrantfileversion         prints current and latest Vagrant versionwinrm           executes commands on a machine via WinRMwinrm-config    outputs WinRM configuration to connect to the machineFor help on any individual command run `vagrant COMMAND -h`Additional subcommands are available, but are either more advanced
or not commonly used. To see all subcommands, run the command
`vagrant list-commands`.
PS C:\Users\geng\env> vagrant init centos/7
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.
PS C:\Users\geng\env> vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Box 'centos/7' could not be found. Attempting to find and install...default: Box Provider: virtualboxdefault: Box Version: >= 0
==> default: Loading metadata for box 'centos/7'default: URL: https://vagrantcloud.com/centos/7
==> default: Adding box 'centos/7' (v1901.01) for provider: virtualboxdefault: Downloading: https://vagrantcloud.com/centos/boxes/7/versions/1901.01/providers/virtualbox.boxdefault: Download redirected to host: cloud.centos.orgdefault:
==> default: Successfully added box 'centos/7' (v1901.01) for 'virtualbox'!
==> default: Importing base box 'centos/7'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'centos/7' version '1901.01' is up to date...
==> default: Setting the name of the VM: env_default_1551574680676_41983
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...default: Adapter 1: nat
==> default: Forwarding ports...default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...default: SSH address: 127.0.0.1:2222default: SSH username: vagrantdefault: SSH auth method: private keydefault:default: Vagrant insecure key detected. Vagrant will automatically replacedefault: this with a newly generated keypair for better security.default:default: Inserting generated public key within guest...default: Removing insecure key from the guest if it's present...default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...default: No guest additions were detected on the base box for this VM! Guestdefault: additions are required for forwarded ports, shared folders, host onlydefault: networking, and more. If SSH fails on this machine, please installdefault: the guest additions and repackage the box to continue.default:default: This is not an error message; everything may continue to work properly,default: in which case you may ignore this message.
==> default: Rsyncing folder: /cygdrive/c/Users/geng/env/ => /vagrant
PS C:\Users\geng\env>
PS C:\Users\geng\env>
PS C:\Users\geng\env> vagrant ssh

进入centos, 安装jdk:

进入centos:

PS C:\Users\geng\env> vagrant ssh

更新软件：

[vagrant@localhost ~]$ sudo yum update

安装java:

[vagrant@localhost ~]$ sudo yum install java-1.8.0-openjdk

安装spark:

https://www.apache.org/dyn/closer.lua/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz

[vagrant@localhost ~]$ curl -O https://www.apache.org/dyn/closer.lua/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz[vagrant@localhost ~]$ tar zxvf spark-2.4.0-bin-hadoop2.7.tgz

运行spark:

[vagrant@localhost opt]$ cd spark-2.4.0-bin-hadoop2.7/
[vagrant@localhost spark-2.4.0-bin-hadoop2.7]$ bin/pyspark
Python 2.7.5 (default, Oct 30 2018, 23:45:53)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
2019-03-03 09:21:13 WARN  Utils:66 - Your hostname, localhost.localdomain resolves to a loopback address: 127.0.0.1; using 10.0.2.15 instead (on interface eth0)
2019-03-03 09:21:13 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address
2019-03-03 09:21:14 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Welcome to____              __/ __/__  ___ _____/ /___\ \/ _ \/ _ `/ __/  '_//__ / .__/\_,_/_/ /_/\_\   version 2.4.0/_/Using Python version 2.7.5 (default, Oct 30 2018 23:45:53)
SparkSession available as 'spark'.

测试spark:

>>> rdd = sc.parallelize([1,2,3,4,5])
>>>
>>>
>>>
>>> rdd.map(lambda x: x+1).reduce(lambda x,y : x+y)
20

window 10下 Spark 安装简单使用相关推荐

Window 10下 Sqlite安装教程
Window 10下 Sqlite安装教程下载官网下载地址:https://sqlite.org/download.html 找到适配自己电脑系统的dll和tools下载. 安装配置按照如下几个 ...
Window 10下JAVA环境配置
Window 10下JAVA环境配置 1.下载结束如图: 注意:根据自己windows系统的版本以及位数下载对应的JDK 2.配置环境将上面下载的JDK安装好,我安装在了F盘,路径如下:F:\J ...
linux14.10硬盘安装,Windows 10下硬盘安装Ubuntu 14.10图文教程
著名的Linux发行版 Ubuntu 系统已经释出最新的 14.10 版,同时大家也可能不知道其实它已经十岁了.与半年前的14.10相比,这次Ubuntu 14.10系统升级的修改幅度很小,外观界面几 ...
Windows 10下编译安装Hadoop2.6
转自:https://www.linuxidc.com/Linux/2016-08/134131.htm Windows 10下安装Hadoop2.6,Windows10下编译64位Hadoop2.x ...
Redis在Window服务下的安装
Redis 安装 1.首先在Windows下下载安装Redis 下载地址:https://github.com/MicrosoftArchive/redis/releases 根据你电脑系统的实际情况 ...
window 10 下配置 cygwin ；并且安装 snaphu --SNAP解缠
写在前面的话:一定要看完,一定要看完,一定要看完,再按照此步骤安装. cygwin 下载网址:Cygwin 安装教程:Cygwin安装教程(图文)-百度经验 (baidu.com) snaphu 下载 ...
python pypcap 安装失败_解决Windows 10下python安装pypcap失败的问题
先说结论,直接使用pcap-ct. 因为项目需求需要在windows下抓包,用python编程.所以就找相关的轮子.找到了三个: 1.pypcap(pip install pypcap) 2.libp ...
mysql 5.6.23 源码包安装报错_Ubuntu 14.10下编译安装MySQL 5.6.23
1. 安装环境: Ubuntu Server 14.10 MySQL-5.6.23.tar.gz 2. 安装必备的工具 sudo apt-get install make bison g++ buil ...
suse 10 下mysql安装
一 . 安装MYSQL 1. 下载MYSQL的安装文件 MySQL-server-5.0.22-0.i386.rpm MySQL-client-5.0.22-0.i386.rpm 下载地址可以到官方网 ...

window 10下 Spark 安装简单使用

文章目录

安装虚拟机管理软件vagrant:

安装VirtualBox, vagrant默认使用VirtualBox:

使用windows的powershell:

进入centos, 安装jdk:

安装spark:

运行spark:

测试spark:

window 10下 Spark 安装简单使用相关推荐

最新文章

热门文章