Spark3.1.1 Docker镜像中修改/etc/hosts
2024-06-06 17:32:52
Dockerfile
注释掉spark_uid
- #ARG spark_uid=185
- #USER ${spark_uid}
[root@localhost spark-3.1.1-bin-hadoop2.7]# cat kubernetes/dockerfiles/spark/Dockerfile
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
ARG java_image_tag=11-jre-slimFROM openjdk:${java_image_tag}#ARG spark_uid=185# Before building the docker image, first build and make a Spark distribution following
# the instructions in http://spark.apache.org/docs/latest/building-spark.html.
# If this docker file is being used in the context of building your images from a Spark
# distribution, the docker build command should be invoked from the top level directory
# of the Spark distribution. E.g.:
# docker build -t spark:latest -f kubernetes/dockerfiles/spark/Dockerfile .RUN set -ex && \sed -i 's/http:\/\/deb.\(.*\)/https:\/\/deb.\1/g' /etc/apt/sources.list && \apt-get update && \ln -s /lib /lib64 && \apt install -y bash tini libc6 libpam-modules krb5-user libnss3 procps sudo && \mkdir -p /opt/spark && \mkdir -p /opt/spark/examples && \mkdir -p /opt/spark/work-dir && \touch /opt/spark/RELEASE && \rm /bin/sh && \ln -sv /bin/bash /bin/sh && \echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su && \chgrp root /etc/passwd && chmod ug+rw /etc/passwd && \rm -rf /var/cache/apt/*COPY jars /opt/spark/jars
COPY bin /opt/spark/bin
COPY sbin /opt/spark/sbin
COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/
COPY kubernetes/dockerfiles/spark/decom.sh /opt/
COPY examples /opt/spark/examples
COPY kubernetes/tests /opt/spark/tests
COPY data /opt/spark/dataCOPY jdk1.8.0_201-amd64 /usr/java/default
COPY hadoop-2.9.2 /opt/bigdata/hadoop-2.9.2/
#COPY hbase-2.2.6 /opt/bigdata/hbase-2.2.6
ENV JAVA_HOME=/usr/java/default
ENV HADOOP_HOME /opt/bigdata/hadoop-2.9.2/
ENV SPARK_HOME /opt/spark
#ENV HBASE_HOME /opt/bigdata/hbase-2.2.6
#ENV SPARK_EXTRA_CLASSPATH hdfs://dmgeo/spark/dmgeo-geospark/dmgeo-geospark-tiler/external_jars
ADD hosts /tmp
#it does not work
#CMD cat /tmp/hosts >> /etc/hostsWORKDIR /opt/spark/work-dir
RUN chmod g+w /opt/spark/work-dir
RUN chmod a+x /opt/decom.shENTRYPOINT ["/opt/entrypoint.sh" ]# Specify the User that the actual main process will run as
#USER ${spark_uid}
[root@localhost spark-3.1.1-bin-hadoop2.7]#
entrypoint.sh
[root@localhost spark-3.1.1-bin-hadoop2.7]# cat kubernetes/dockerfiles/spark/entrypoint.sh
#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
## echo commands to the terminal output
set -ex# Check whether there is a passwd entry for the container UID
myuid=$(id -u)
mygid=$(id -g)
# turn off -e for getent because it will return error code in anonymous uid case
set +e
uidentry=$(getent passwd $myuid)
set -e# If there is no passwd entry for the container UID, attempt to create one
if [ -z "$uidentry" ] ; thenif [ -w /etc/passwd ] ; thenecho "$myuid:x:$myuid:$mygid:${SPARK_USER_NAME:-anonymous uid}:$SPARK_HOME:/bin/false" >> /etc/passwdelseecho "Container ENTRYPOINT failed to add passwd entry for anonymous UID"fi
fiSPARK_CLASSPATH="$SPARK_CLASSPATH:${SPARK_HOME}/jars/*"
env | grep SPARK_JAVA_OPT_ | sort -t_ -k4 -n | sed 's/[^=]*=\(.*\)/\1/g' > /tmp/java_opts.txt
readarray -t SPARK_EXECUTOR_JAVA_OPTS < /tmp/java_opts.txtif [ -n "$SPARK_EXTRA_CLASSPATH" ]; thenSPARK_CLASSPATH="$SPARK_CLASSPATH:$SPARK_EXTRA_CLASSPATH"
fiif ! [ -z ${PYSPARK_PYTHON+x} ]; thenexport PYSPARK_PYTHON
fi
if ! [ -z ${PYSPARK_DRIVER_PYTHON+x} ]; thenexport PYSPARK_DRIVER_PYTHON
fi# If HADOOP_HOME is set and SPARK_DIST_CLASSPATH is not set, set it here so Hadoop jars are available to the executor.
# It does not set SPARK_DIST_CLASSPATH if already set, to avoid overriding customizations of this value from elsewhere e.g. Docker/K8s.
if [ -n "${HADOOP_HOME}" ] && [ -z "${SPARK_DIST_CLASSPATH}" ]; thenexport SPARK_DIST_CLASSPATH="$($HADOOP_HOME/bin/hadoop classpath)"
fiif ! [ -z ${HADOOP_CONF_DIR+x} ]; thenSPARK_CLASSPATH="$HADOOP_CONF_DIR:$SPARK_CLASSPATH";
fiif ! [ -z ${SPARK_CONF_DIR+x} ]; thenSPARK_CLASSPATH="$SPARK_CONF_DIR:$SPARK_CLASSPATH";
elif ! [ -z ${SPARK_HOME+x} ]; thenSPARK_CLASSPATH="$SPARK_HOME/conf:$SPARK_CLASSPATH";
ficase "$1" indriver)shift 1CMD=("$SPARK_HOME/bin/spark-submit"--conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS"--deploy-mode client"$@");;executor)shift 1CMD=(${JAVA_HOME}/bin/java"${SPARK_EXECUTOR_JAVA_OPTS[@]}"-Xms$SPARK_EXECUTOR_MEMORY-Xmx$SPARK_EXECUTOR_MEMORY-cp "$SPARK_CLASSPATH:$SPARK_DIST_CLASSPATH"org.apache.spark.executor.CoarseGrainedExecutorBackend--driver-url $SPARK_DRIVER_URL--executor-id $SPARK_EXECUTOR_ID--cores $SPARK_EXECUTOR_CORES--app-id $SPARK_APPLICATION_ID--hostname $SPARK_EXECUTOR_POD_IP--resourceProfileId $SPARK_RESOURCE_PROFILE_ID);;*)echo "Non-spark-on-k8s command provided, proceeding in pass-through mode..."CMD=("$@");;
esac# modify hosts
cat /tmp/hosts >> /etc/hosts# Execute the container CMD under tini for better hygiene
exec /usr/bin/tini -s -- "${CMD[@]}"
[root@localhost spark-3.1.1-bin-hadoop2.7]#
Spark3.1.1 Docker镜像中修改/etc/hosts相关推荐
- 在docker镜像中加入环境变量
原文链接 前言 reference:https://vsupalov.com/docker-build-time-env-values/ 很多时候,我们需要在docker镜像中加入环境变量,本人了解的 ...
- docker中linux用户名密码,linuxea:尽可能不在docker镜像中嵌入配置或者密码
切勿将配置或机密信息嵌入Docker镜像中.相反,在构建Docker镜像时,期望使用业务流程运行时将配置和机密的信息提供给容器,这些包含:Kubernetes Secrets,Docker Secre ...
- 查看docker镜像中的文件
1. 进入镜像内部 docker run -it --entrypoint sh <镜像名称> 2.执行上面命令后,会进入镜像内部.然后使用ls -al会将镜像中的文件列出来
- docker镜像中进入容器命令行并查看运行中的容器
之前遇到过一个项目直接给了一个docker环境,但是我对于docker的命令行并不熟,于是查了相关资料,这里以容器MySQL为例,给大家举个例子 启动镜像中的MySQL 启动MySQL分为两步: 查看 ...
- docker容器中修改时间
我们在docker容器中直接通过命令行修改时间会有下面这个报错: date: can't set date: Operation not permitted 这是由于docker容器是基于Linux的 ...
- 【Docker】从 Docker 镜像中下载内容到本地
使用 docker run 命令启动镜像并进入容器. docker run -it --name my-container my-image:tag /bin/bash 其中 my-container ...
- 在一个已经EXIT的docker容器中,修改配置文件
文章目录 背景 一.方法一 二.方法二 1.通过运行docker inspect [容器id]来获取到底层的容器对象信息 2.修改配置文件 总结 背景 当我们在修改容器里的配置文件后(例如mysql) ...
- 一个docker镜像中的目录删除不了问题
在一个容器中,删除一个目录,失败: bash-4.2# pwd /home/zxcdn/ottcache/tomcat bash-4.2# uname -a Linux 3516b6c97679 3. ...
- 如何下载python安装包的所有依赖_如何将包含所有依赖项的python包安装到Docker镜像中？...
我在Ubuntu 15.10中使用Docker容器为Pyspark jupyter / pyspark-notebook工作.我需要安装具有所有依赖项的folium并在容器中运行Pyspark脚本.我 ...
最新文章
- themleaf返回可以带目录结构
- python画三维温度散点图-python 绘制三维图形、三维数据散点图
- freertos之任务
- 数学之路(2)-数据分析-R基础(5)
- 多级嵌套json格式
- .NET Core多平台项目模板eShopOnContainers编译手记
- mysql 查外键关联关系 （指定被引用表，查哪些表对其有外键引用）
- windows网络版象棋的实现
- mysql 表结构反转_一道经典的MySQL面试题，答案出现三次反转
- Flink watermark
- Python基础笔记(四)切片、列表生成式、迭代
- 21SkypeForBusiness2015进阶篇--SFB后端Mirror切换到AllwaysOn--标准版准备篇
- lotus Domino调用webservice
- Window10总是自动打开网络代理的解决方案法
- 在线五子棋对战 --- 人机对战的实现
- 应急响应 - Windows启动项分析，Windows计划任务分析，Windows服务分析
- Arduino 无源蜂鸣器实验
- Salt Pepper—The Art of Illustrating Texture
- 【51Nod1585】买邮票送邮票
- Python教程（十）