在CDH sandbox中安装并测试livy

安装部分

1、下载Livy安装包，如livy-0.5.0-incubating-bin.zip
2、上传到/opt/livy目录
3、解压安装包 unzip livy-0.5.0-incubating-bin.zip
4、配置Livy使用需要的spark环境变量

   export SPARK_HOME=/usr/lib/sparkexport HADOOP_CONF_DIR=/etc/hadoop/conf

5、启动livy

    cd  /opt/livy/livy-0.5.0-incubating-bin/bin./livy-server start

Livy默认使用端口8998，可以查看http://<IP>:8998/ui

测试部分（通过livy使用spark-shell）

1、创建pyspark会话

[root@quickstart ~]# curl -X POST --data '{"kind": "pyspark"}' -H "Content-Type:application/json" localhost:8998/sessions
{"id":1,"appId":null,"owner":null,"proxyUser":null,"state":"starting","kind":"pyspark","appInfo":{"driverLogUrl":null,"sparkUiUrl":null},"log":["stdout: ","\nstderr: "]}

2、查看会话状态

[root@quickstart ~]# curl localhost:8998/sessions/1 | python -m json.tool% Total    % Received % Xferd  Average Speed   Time    Time     Time  CurrentDload  Upload   Total   Spent    Left  Speed
107  1290  107  1290    0     0  43949      0 --:--:-- --:--:-- --:--:-- 75882
{"appId": null, "appInfo": {"driverLogUrl": null, "sparkUiUrl": null}, "id": 1, "kind": "pyspark", "log": ["18/11/25 22:54:38 INFO spark.SparkContext: Added JAR file:/opt/livy/livy-0.5.0-incubating-bin/repl_2.10-jars/livy-repl_2.10-0.5.0-incubating.jar at spark://192.168.64.154:55570/jars/livy-repl_2.10-0.5.0-incubating.jar with timestamp 1543215278500", "18/11/25 22:54:38 INFO executor.Executor: Starting executor ID driver on host localhost", "18/11/25 22:54:38 INFO executor.Executor: Using REPL class URI: spark://192.168.64.154:55570/classes", "18/11/25 22:54:38 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 51955.", "18/11/25 22:54:38 INFO netty.NettyBlockTransferService: Server created on 51955", "18/11/25 22:54:38 INFO storage.BlockManagerMaster: Trying to register BlockManager", "18/11/25 22:54:38 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:51955 with 534.5 MB RAM, BlockManagerId(driver, localhost, 51955)", "18/11/25 22:54:38 INFO storage.BlockManagerMaster: Registered BlockManager", "18/11/25 22:54:38 INFO driver.SparkEntries: Spark context finished initialization in 6592ms", "18/11/25 22:54:41 INFO driver.SparkEntries: Created SQLContext."], "owner": null, "proxyUser": null, "state": "idle"
}

state为idle表示该会话存活，已经准备就绪，可以向该会话提交任务了

3、提交任务

[root@quickstart ~]# curl localhost:8998/sessions/1/statements -X POST -H 'Content-Type: application/json' -d '{"code":"1 + 1"}'
{"id":0,"code":"1 + 1","state":"waiting","output":null,"progress":0.0}

从返回可以看出，生成了一个statement，id为0，我们可以查看该statement的运行状态以及结果。

4、查看任务结果

[root@quickstart ~]# curl localhost:8998/sessions/1/statements/0
{"id":0,"code":"1 + 1","state":"available","output":{"status":"ok","execution_count":0,"data":{"text/plain":"2"}},"progress":1.0}

5、继续提交任务

[root@quickstart ~]# curl localhost:8998/sessions/1/statements -X POST -H 'Content-Type: application/json' -d '{"code":"a = 10"}'
{"id":1,"code":"a = 10","state":"available","output":{"status":"ok","execution_count":1,"data":{"text/plain":""}},"progress":1.0}

从返回可以看出，生成了一个statement，id为1

6、执行a+1操作

[root@quickstart ~]# curl localhost:8998/sessions/1/statements -X POST -H 'Content-Type: application/json' -d '{"code":"a + 1"}'
{"id":2,"code":"a + 1","state":"available","output":{"status":"ok","execution_count":2,"data":{"text/plain":"11"}},"progress":1.0}

从返回可以看出，生成了一个statement，id为2，执行的代码为a+1，返回的结果为11
也可以通过如下接口查看执行结果

[root@quickstart bin]# curl localhost:8998/sessions/1/statements/2
{"id":2,"code":"a + 1","state":"available","output":{"status":"ok","execution_count":2,"data":{"text/plain":"11"}},"progress":1.0}

其它

1、通过livy还可以提交spark代码、执行python文件等，详细请参考 https://blog.csdn.net/dockj/article/details/53328800

2、在livy界面中查看session列表

3、查看session 1 的详细信息

4、在postman中通过rest API查看