尧图网页设计 
目录1.flink集群搭建1集群规划2下载并解压安装包① 下载安装包flink-1.17.1-bin-scala_2.12.tgz将该jar包上传到hadoop202节点服务器的/opt/software路径上。② 解压flink-1.17.1-bin-scala_2.12.tgz到/opt/module路径3修改集群配置① 修改flink-conf.yaml② 修改workers文件指定hadoop202、hadoop203和hadoop204为TaskManager③ 修改masters文件④ 在flink-conf.yaml文件中还可以对集群中的JobManager和TaskManager组件进行优化配置主要配置项如下4将flink安装目录拷贝给另外两个节点服务器① 拷贝flink到203和204服务器② 修改hadoop203的 taskmanager.host③ 修改hadoop204的 taskmanager.host④ 配置环境变量yarn运行模式会用到5启动集群① 在hadoop202节点服务器上执行start-cluster.sh启动Flink集群② 查看进程访问web ui2. 单作业模式Per-Job Mode3.YARN运行模式重点1相关准备和配置① 配置环境变量1.flink集群搭建1集群规划节点服务器hadoop202hadoop203hadoop204角色JobManagerTaskManagerTaskManagerTaskManager2下载并解压安装包Downloads | Apache FlinkApache Flink® Downloads # Apache Flink # Apache Flink® 1.20.0 是我们最新的稳定版本。Apache Flink 2.0-preview1 # Apache Flink 2.0-preview1 (asc, sha512)Apache Flink 2.0-preview1 Source Release (asc, sha512)Apache Flink 1.20.0 # Apache Flink 1.20.0 (asc, sha512)Apache Flink 1.20.0 Source Release (asc, sha512)Release Notes # Please have a look at the Release Notes for Apache Flink 1.20.0 if you plan to upgrade your Flink setup from a previous version.https://flink.apache.org/zh/downloads/① 下载安装包flink-1.17.1-bin-scala_2.12.tgz将该jar包上传到hadoop202节点服务器的/opt/software路径上。② 解压flink-1.17.1-bin-scala_2.12.tgz到/opt/module路径进入/opt/software目录解压flinktar -zxvf flink-1.17.1-bin-scala_2.12.tgz -C /opt/module/3修改集群配置① 修改flink-conf.yaml进入目录/opt/module/flink-1.17.1/confvim flink-conf.yaml修改后# JobManager节点地址 jobmanager.rpc.address: hadoop202 jobmanager.bind-host: 0.0.0.0 rest.address: hadoop202 rest.bind-address: 0.0.0.0 # TaskManager节点地址.需要配置为当前机器名 taskmanager.bind-host: 0.0.0.0 taskmanager.host: hadoop202② 修改workers文件指定hadoop202、hadoop203和hadoop204为TaskManager在目录/opt/module/flink-1.17.1/conf[tjmhadoop202 conf]$ vim workers添加如下内容hadoop202 hadoop203 hadoop204③ 修改masters文件在目录/opt/module/flink-1.17.1/confvim masters④ 在flink-conf.yaml文件中还可以对集群中的JobManager和TaskManager组件进行优化配置主要配置项如下jobmanager.memory.process.size对JobManager进程可使用到的全部内存进行配置包括JVM元空间和其他开销默认为1600M可以根据集群规模进行适当调整。taskmanager.memory.process.size对TaskManager进程可使用到的全部内存进行配置包括JVM元空间和其他开销默认为1728M可以根据集群规模进行适当调整。taskmanager.numberOfTaskSlots对每个TaskManager能够分配的Slot数量进行配置默认为1可根据TaskManager所在的机器能够提供给Flink的CPU数量决定。所谓Slot就是TaskManager中具体运行一个任务所分配的计算资源。4.parallelism.defaultFlink任务执行的并行度默认为1。优先级低于代码中进行的并行度配置和任务提交时使用参数指定的并行度数量。4将flink安装目录拷贝给另外两个节点服务器① 拷贝flink到203和204服务器# 拷贝到203服务器 scp -r /opt/module/flink-1.17.1/ roothadoop203:/opt/module/ # 拷贝到204服务器 scp -r /opt/module/flink-1.17.1/ roothadoop204:/opt/module/② 修改hadoop203的 taskmanager.host目录/opt/module/flink-1.17.1/confvim flink-conf.yaml修改如下内容#TaskManager节点地址.需要配置为当前机器名taskmanager.host: hadoop203③ 修改hadoop204的 taskmanager.host目录/opt/module/flink-1.17.1/confvim flink-conf.yaml修改如下内容#TaskManager节点地址.需要配置为当前机器名taskmanager.host: hadoop204④ 配置环境变量yarn运行模式会用到vim /etc/profile.d/my_env.sh#与flink有关 export HADOOP_CONF_DIR$HADOOP_HOME/etc/hadoop export HADOOP_CLASSPATHhadoop classpath# 使环境变量生效 source /etc/profile重写启动Hadoop集群包括HDFS和YARN。hadoop202重启hdfs# 停止hdfs stop-dfs.sh # 启动hdfs start-dfs.shhadoop203重启yarn# 停止yarn stop-yarn.sh # 启动yarn start-yarn.sh5启动集群① 在hadoop202节点服务器上执行start-cluster.sh启动Flink集群目录/opt/module/flink-1.17.1bin/start-cluster.sh② 查看进程访问web ui[tjmhadoop202 flink-1.17.1]$ jps 6306 DataNode 9988 Jps 6182 NameNode 6776 NodeManager 5435 JobHistoryServer 9901 TaskManagerRunner 9566 StandaloneSessionClusterEntrypointhttp://hadoop202:8081/用root用户启动flink集群是正常的估计是tjm用户的权限不够[tjmhadoop202 flink-1.17.1]$ bin/stop-cluster.sh Stopping taskexecutor daemon (pid: 24832) on host hadoop202. No taskexecutor daemon (pid: 16210) is running anymore on hadoop203. No taskexecutor daemon (pid: 20565) is running anymore on hadoop204. No standalonesession daemon (pid: 24438) is running anymore on hadoop202. [tjmhadoop202 flink-1.17.1]$ su - Password: Last login: Sun Nov 10 07:46:10 PST 2024 from 192.168.226.1 on pts/1 [roothadoop202 ~]# cd /opt/module/flink-1.17.1/ [roothadoop202 flink-1.17.1]# bin/start-cluster.sh Starting cluster. Starting standalonesession daemon on host hadoop202. roothadoop202s password: Starting taskexecutor daemon on host hadoop202. roothadoop203s password: Starting taskexecutor daemon on host hadoop203. roothadoop204s password: Starting taskexecutor daemon on host hadoop204. [roothadoop202 flink-1.17.1]# jps 26689 TaskManagerRunner 17073 TaskManagerRunner 12131 NodeManager 16676 StandaloneSessionClusterEntrypoint 26775 Jps 5435 JobHistoryServer 13260 NameNode 13421 DataNode2.单作业模式Per-Job Mode3.YARN运行模式重点YARN上部署的过程是客户端把Flink应用提交给Yarn的ResourceManagerYarn的ResourceManager会向Yarn的NodeManager申请容器。在这些容器上Flink会部署JobManager和TaskManager的实例从而启动集群。Flink会根据运行在JobManger上的作业所需要的Slot数量动态分配TaskManager资源。1相关准备和配置在将Flink任务部署至YARN集群之前需要确认集群是否安装有Hadoop保证Hadoop版本至少在2.2以上并且集群中安装有HDFS服务。① 配置环境变量参考1.4.4章节vim /etc/profile.d/my_env.sh增加环境变量配置如下HADOOP_HOME/opt/module/hadoop-3.3.4 export PATH$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin export HADOOP_CONF_DIR$HADOOP_HOME/etc/hadoop export HADOOP_CLASSPATHhadoop classpath② 启动Hadoop集群包括HDFS和YARN# hadoop202运行 start-dfs.sh # hadoop203运行 start-yarn.sh 【实时数仓（二）】flink-1.17.1集群搭建
目录1.flink集群搭建1集群规划2下载并解压安装包① 下载安装包flink-1.17.1-bin-scala_2.12.tgz将该jar包上传到hadoop202节点服务器的/opt/software路径上。② 解压flink-1.17.1-bin-scala_2.12.tgz到/opt/module路径3修改集群配置① 修改flink-conf.yaml② 修改workers文件指定hadoop202、hadoop203和hadoop204为TaskManager③ 修改masters文件④ 在flink-conf.yaml文件中还可以对集群中的JobManager和TaskManager组件进行优化配置主要配置项如下4将flink安装目录拷贝给另外两个节点服务器① 拷贝flink到203和204服务器② 修改hadoop203的 taskmanager.host③ 修改hadoop204的 taskmanager.host④ 配置环境变量yarn运行模式会用到5启动集群① 在hadoop202节点服务器上执行start-cluster.sh启动Flink集群② 查看进程访问web ui2. 单作业模式Per-Job Mode3.YARN运行模式重点1相关准备和配置① 配置环境变量1.flink集群搭建1集群规划节点服务器hadoop202hadoop203hadoop204角色JobManagerTaskManagerTaskManagerTaskManager2下载并解压安装包Downloads | Apache FlinkApache Flink® Downloads # Apache Flink # Apache Flink® 1.20.0 是我们最新的稳定版本。Apache Flink 2.0-preview1 # Apache Flink 2.0-preview1 (asc, sha512)Apache Flink 2.0-preview1 Source Release (asc, sha512)Apache Flink 1.20.0 # Apache Flink 1.20.0 (asc, sha512)Apache Flink 1.20.0 Source Release (asc, sha512)Release Notes # Please have a look at the Release Notes for Apache Flink 1.20.0 if you plan to upgrade your Flink setup from a previous version.https://flink.apache.org/zh/downloads/① 下载安装包flink-1.17.1-bin-scala_2.12.tgz将该jar包上传到hadoop202节点服务器的/opt/software路径上。② 解压flink-1.17.1-bin-scala_2.12.tgz到/opt/module路径进入/opt/software目录解压flinktar -zxvf flink-1.17.1-bin-scala_2.12.tgz -C /opt/module/3修改集群配置① 修改flink-conf.yaml进入目录/opt/module/flink-1.17.1/confvim flink-conf.yaml修改后# JobManager节点地址 jobmanager.rpc.address: hadoop202 jobmanager.bind-host: 0.0.0.0 rest.address: hadoop202 rest.bind-address: 0.0.0.0 # TaskManager节点地址.需要配置为当前机器名 taskmanager.bind-host: 0.0.0.0 taskmanager.host: hadoop202② 修改workers文件指定hadoop202、hadoop203和hadoop204为TaskManager在目录/opt/module/flink-1.17.1/conf[tjmhadoop202 conf]$ vim workers添加如下内容hadoop202 hadoop203 hadoop204③ 修改masters文件在目录/opt/module/flink-1.17.1/confvim masters④ 在flink-conf.yaml文件中还可以对集群中的JobManager和TaskManager组件进行优化配置主要配置项如下jobmanager.memory.process.size对JobManager进程可使用到的全部内存进行配置包括JVM元空间和其他开销默认为1600M可以根据集群规模进行适当调整。taskmanager.memory.process.size对TaskManager进程可使用到的全部内存进行配置包括JVM元空间和其他开销默认为1728M可以根据集群规模进行适当调整。taskmanager.numberOfTaskSlots对每个TaskManager能够分配的Slot数量进行配置默认为1可根据TaskManager所在的机器能够提供给Flink的CPU数量决定。所谓Slot就是TaskManager中具体运行一个任务所分配的计算资源。4.parallelism.defaultFlink任务执行的并行度默认为1。优先级低于代码中进行的并行度配置和任务提交时使用参数指定的并行度数量。4将flink安装目录拷贝给另外两个节点服务器① 拷贝flink到203和204服务器# 拷贝到203服务器 scp -r /opt/module/flink-1.17.1/ roothadoop203:/opt/module/ # 拷贝到204服务器 scp -r /opt/module/flink-1.17.1/ roothadoop204:/opt/module/② 修改hadoop203的 taskmanager.host目录/opt/module/flink-1.17.1/confvim flink-conf.yaml修改如下内容#TaskManager节点地址.需要配置为当前机器名taskmanager.host: hadoop203③ 修改hadoop204的 taskmanager.host目录/opt/module/flink-1.17.1/confvim flink-conf.yaml修改如下内容#TaskManager节点地址.需要配置为当前机器名taskmanager.host: hadoop204④ 配置环境变量yarn运行模式会用到vim /etc/profile.d/my_env.sh#与flink有关 export HADOOP_CONF_DIR$HADOOP_HOME/etc/hadoop export HADOOP_CLASSPATHhadoop classpath# 使环境变量生效 source /etc/profile重写启动Hadoop集群包括HDFS和YARN。hadoop202重启hdfs# 停止hdfs stop-dfs.sh # 启动hdfs start-dfs.shhadoop203重启yarn# 停止yarn stop-yarn.sh # 启动yarn start-yarn.sh5启动集群① 在hadoop202节点服务器上执行start-cluster.sh启动Flink集群目录/opt/module/flink-1.17.1bin/start-cluster.sh② 查看进程访问web ui[tjmhadoop202 flink-1.17.1]$ jps 6306 DataNode 9988 Jps 6182 NameNode 6776 NodeManager 5435 JobHistoryServer 9901 TaskManagerRunner 9566 StandaloneSessionClusterEntrypointhttp://hadoop202:8081/用root用户启动flink集群是正常的估计是tjm用户的权限不够[tjmhadoop202 flink-1.17.1]$ bin/stop-cluster.sh Stopping taskexecutor daemon (pid: 24832) on host hadoop202. No taskexecutor daemon (pid: 16210) is running anymore on hadoop203. No taskexecutor daemon (pid: 20565) is running anymore on hadoop204. No standalonesession daemon (pid: 24438) is running anymore on hadoop202. [tjmhadoop202 flink-1.17.1]$ su - Password: Last login: Sun Nov 10 07:46:10 PST 2024 from 192.168.226.1 on pts/1 [roothadoop202 ~]# cd /opt/module/flink-1.17.1/ [roothadoop202 flink-1.17.1]# bin/start-cluster.sh Starting cluster. Starting standalonesession daemon on host hadoop202. roothadoop202s password: Starting taskexecutor daemon on host hadoop202. roothadoop203s password: Starting taskexecutor daemon on host hadoop203. roothadoop204s password: Starting taskexecutor daemon on host hadoop204. [roothadoop202 flink-1.17.1]# jps 26689 TaskManagerRunner 17073 TaskManagerRunner 12131 NodeManager 16676 StandaloneSessionClusterEntrypoint 26775 Jps 5435 JobHistoryServer 13260 NameNode 13421 DataNode2.单作业模式Per-Job Mode3.YARN运行模式重点YARN上部署的过程是客户端把Flink应用提交给Yarn的ResourceManagerYarn的ResourceManager会向Yarn的NodeManager申请容器。在这些容器上Flink会部署JobManager和TaskManager的实例从而启动集群。Flink会根据运行在JobManger上的作业所需要的Slot数量动态分配TaskManager资源。1相关准备和配置在将Flink任务部署至YARN集群之前需要确认集群是否安装有Hadoop保证Hadoop版本至少在2.2以上并且集群中安装有HDFS服务。① 配置环境变量参考1.4.4章节vim /etc/profile.d/my_env.sh增加环境变量配置如下HADOOP_HOME/opt/module/hadoop-3.3.4 export PATH$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin export HADOOP_CONF_DIR$HADOOP_HOME/etc/hadoop export HADOOP_CLASSPATHhadoop classpath② 启动Hadoop集群包括HDFS和YARN# hadoop202运行 start-dfs.sh # hadoop203运行 start-yarn.sh