本地部署

安装

在官网安装Flink，并解压到/usr/local/flink

sudo tar -zxf flink-1.6.2-bin-hadoop27-scala_2.11.tgz -C /usr/local
cd /usr/local

54388226982

修改文件名字，并设置权限

sudo mv ./flink-*/ ./flink
sudo chown -R hadoop:hadoop ./flink

修改配置文件

Flink对于本地模式是开箱即用的，如果要修改Java运行环境，可修改conf/flink-conf.yaml中的env.java.home，设置为本地java的绝对路径

添加环境变量

vim ~/.bashrc
export FLINK_HOME=/usr/local/flink
export PATH=$FLINK_HOME/bin:$PATH

54388242695

启动Flink

start-cluster.sh

可以通过观察logs目录下的日志来检测系统是否正在运行了

tail log/flink--jobmanager-.log

54388315301

JobManager同时会在8081端口上启动一个web前端，通过http://localhost:8081来访问

54388290147

可以发现flink已经正常启动

运行示例

使用Maven创建Flink项目，在pom.xml中添加以下依赖：

    <dependencies><dependency><groupId>org.apache.flink</groupId><artifactId>flink-java</artifactId><version>1.6.2</version></dependency><dependency><groupId>org.apache.flink</groupId><artifactId>flink-streaming-java_2.11</artifactId><version>1.6.2</version></dependency><dependency><groupId>org.apache.flink</groupId><artifactId>flink-clients_2.11</artifactId><version>1.6.2</version></dependency></dependencies>

批处理运行WordCount

官方示例

可以直接在/usr/local/flink/examples/batch中运行WordCount程序，并且这里还有更多示例：

54388437325

运行：

flink run WordCount.jar

54388443638

代码

WordCountData

提供原始数据

import org.apache.flink.api.java.DataSet;
import org.apache.flink.api.java.ExecutionEnvironment;public class WordCountData {public static final String[] WORDS=new String[]{"To be, or not to be,--that is the question:--", "Whether \'tis nobler in the mind to suffer", "The slings and arrows of outrageous fortune", "Or to take arms against a sea of troubles,", "And by opposing end them?--To die,--to sleep,--", "No more; and by a sleep to say we end", "The heartache, and the thousand natural shocks", "That flesh is heir to,--\'tis a consummation", "Devoutly to be wish\'d. To die,--to sleep;--", "To sleep! perchance to dream:--ay, there\'s the rub;", "For in that sleep of death what dreams may come,", "When we have shuffled off this mortal coil,", "Must give us pause: there\'s the respect", "That makes calamity of so long life;", "For who would bear the whips and scorns of time,", "The oppressor\'s wrong, the proud man\'s contumely,", "The pangs of despis\'d love, the law\'s delay,", "The insolence of office, and the spurns", "That patient merit of the unworthy takes,", "When he himself might his quietus make", "With a bare bodkin? who would these fardels bear,", "To grunt and sweat under a weary life,", "But that the dread of something after death,--", "The undiscover\'d country, from whose bourn", "No traveller returns,--puzzles the will,", "And makes us rather bear those ills we have", "Than fly to others that we know not of?", "Thus conscience does make cowards of us all;", "And thus the native hue of resolution", "Is sicklied o\'er with the pale cast of thought;", "And enterprises of great pith and moment,", "With this regard, their currents turn awry,", "And lose the name of action.--Soft you now!", "The fair Ophelia!--Nymph, in thy orisons", "Be all my sins remember\'d."};public WordCountData() {}public static DataSet<String> getDefaultTextLineDataset(ExecutionEnvironment env){return env.fromElements(WORDS);}
}

WordCountTokenizer

切分句子

import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.util.Collector;public class WordCountTokenizer implements FlatMapFunction<String, Tuple2<String,Integer>>{public WordCountTokenizer(){}public void flatMap(String value, Collector<Tuple2<String, Integer>> out) throws Exception {String[] tokens = value.toLowerCase().split("\\W+");int len = tokens.length;for(int i = 0; i<len;i++){String tmp = tokens[i];if(tmp.length()>0){out.collect(new Tuple2<String, Integer>(tmp,Integer.valueOf(1)));}}}
}

WordCount

主函数

import org.apache.flink.api.java.DataSet;
import org.apache.flink.api.java.ExecutionEnvironment;
import org.apache.flink.api.java.operators.AggregateOperator;
import org.apache.flink.api.java.utils.ParameterTool;public class WordCount {public WordCount(){}public static void main(String[] args) throws Exception {ParameterTool params = ParameterTool.fromArgs(args);ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();env.getConfig().setGlobalJobParameters(params);Object text;//如果没有指定输入路径，则默认使用WordCountData中提供的数据if(params.has("input")){text = env.readTextFile(params.get("input"));}else{System.out.println("Executing WordCount example with default input data set.");System.out.println("Use -- input to specify file input.");text = WordCountData.getDefaultTextLineDataset(env);}AggregateOperator counts = ((DataSet)text).flatMap(new WordCountTokenizer()).groupBy(new int[]{0}).sum(1);//如果没有指定输出，则默认打印到控制台if(params.has("output")){counts.writeAsCsv(params.get("output"),"\n", " ");env.execute();}else{System.out.println("Printing result to stdout. Use --output to specify output path.");counts.print();}}
}

首先打包成JAR包，这里需要使用-c指定main函数：

flink run -c WordCount WordCount.jar

流处理运行WordCount

官方示例

可以直接在/usr/local/flink/examples/streaming中运行WordCount程序，并且这里还有更多示例：

54388669798

代码

SocketWindowWordCount

import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.java.utils.ParameterTool;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;import java.sql.Time;
import java.util.stream.Collector;public class SocketWindowWordCount {public static void main(String[] args) throws Exception {// the port to connect tofinal int port;try {final ParameterTool params = ParameterTool.fromArgs(args);port = params.getInt("port");} catch (Exception e) {System.err.println("No port specified. Please run 'SocketWindowWordCount --port <port>'");return;}// get the execution environmentfinal StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();// get input data by connecting to the socketDataStream<String> text = env.socketTextStream("localhost", port, "\n");// parse the data, group it, window it, and aggregate the countsDataStream<WordWithCount> windowCounts = text.flatMap(new FlatMapFunction<String, WordWithCount>() {@Overridepublic void flatMap(String value, Collector<WordWithCount> out) {for (String word : value.split("\\s")) {out.collect(new WordWithCount(word, 1L));}}}).keyBy("word").timeWindow(Time.seconds(5), Time.seconds(1)).reduce(new ReduceFunction<WordWithCount>() {@Overridepublic WordWithCount reduce(WordWithCount a, WordWithCount b) {return new WordWithCount(a.word, a.count + b.count);}});// print the results with a single thread, rather than in parallelwindowCounts.print().setParallelism(1);env.execute("Socket Window WordCount");}// Data type for words with countpublic static class WordWithCount {public String word;public long count;public WordWithCount() {}public WordWithCount(String word, long count) {this.word = word;this.count = count;}@Overridepublic String toString() {return word + " : " + count;}}
}

首先打包成JAR包，然后启动netcat：

nc -l 9000

将终端启动netcat作为输入流：

提交Jar包：

flink run -c SocketWindowWordCount WordCountSteaming.jar --port 9000

这样终端会一直等待netcat的输入流

54388822906

在netcat中输入字符流：

54388825265

可以在WebUI中查看运行结果：

54388897680

作者：zealscott
链接：https://www.jianshu.com/p/bbaa8d72cfcf/
来源：简书
简书著作权归作者所有，任何形式的转载都请联系作者获得授权并注明出处。

Flink安装及使用相关推荐

2021年大数据Flink（三）：Flink安装部署 Local本地模式
目录 Flink安装部署 Local本地模式原理操作测试 Flink安装部署 Flink支持多种安装模式 - Local-本地单机模式,学习测试时使用 - Standalone-独立集群模式,F ...
Flink安装与编程实践（Flink1.9.1）
Flink安装与编程实践(Flink1.9.1) 1.安装Flink Flink的运行需要Java环境的支持,因此,在安装Flink之前,请先参照相关资料安装Java环境(比如Java8).然后,到F ...
学习笔记Flink（三）—— Flink安装启动与监控
一.Linux环境准备 Centos7, 1 CPU , 2G Memory ,20G Disk , Virtual System Hostname : node110.centos.com .nod ...
第一天：什么是Flink、WordCount入门、Flink安装、并行度
1. 初识 Flink 在当前数据量激增的时代,各种业务场景都有大量的业务数据产生,对于这些不断产的数据应该如何进行有效的处理,成为当下大多数公司所面临的问题.目前比较流行的大数据处理引擎 Apach ...
大数据Flink安装部署
目录 1 Local本地模式 1.1 原理 1.2 操作 1.3 测试 2 Standalone独立集群模式 2.1 原理 2.2 操作 2.3 测试 3 Standalone-HA高可用集群模式 3 ...
0001-Flink安装---Flink安装(Standlone模式)
一.获取安装包,上传至服务器并解压在安装Flink之前,我们先要获取到flink-1.10.1-bin-scala_2.12.tgz.zip安装包. (注意:Flin有两个版本分别是flink-1. ...
凌波微步Flink——Flink安装及简单实用
转载请注明出处:http://blog.csdn.net/dongdong9223/article/details/88819199 本文出自[我是干勾鱼的博客] Ingredients: Java: ...
flink安装以及运行自带wordcount示例（单机版，无hadoop环境）
1.下载安装包到/opt目录 2.解压安装包 tar zxf flink-1.6.1-bin-hadoop26-scala_2.11.tgz 3.启动flink cd /opt/flink-1.6.1 ...
Flink安装及运行说明
目录 Flink部署Linux集群版 Flink任务提交方式 Flink运行架构 Flink部署Linux集群版修改flink-conf.yaml,指定master节点地址修改masters,配置 ...

Flink安装及使用

本地部署

安装

修改配置文件

添加环境变量

启动Flink

运行示例

批处理运行WordCount

官方示例

代码

流处理运行WordCount

官方示例

代码

Flink安装及使用相关推荐

最新文章

热门文章