Kubernetes环境下如何运行Coherence缓存集群
2024-06-03 09:31:12
Oracle官方出了一个如何在Docker环境下运行Coherence的技术文档，大家可以参考：
https://github.com/oracle/docker-images/tree/master/OracleCoherence
但是对于一个熟悉Coherence的老司机来说，简单搭建起来只是个初步方案，在客户的环境总是各种特性和定制化配置，所以本文研究的也是如何将已经客户化的Coherence架构构建在Kubernetes开源框架上。
背景架构说明
话不多说，找一个客户的典型的Coherence架构
架构说明：
- Coherence的节点有不同的角色，分成了存储节点，代理节点，管理节点和客户端
- 不同的角色需要配置不同的Cacheconfig文件
- 整个架构是属于Coherence Extend的方式，客户端通过tcp长连接的方式连入Coherence集群
- 存储节点存储数据，代理节点，管理节点和客户端都不存储数据，代理节点主要负责调度客户端请求到Coherence集群中。
Kubernetes环境下架构
区别在于:
- 客户端WebLogic Server基本都是基于Replication Controller部署成一个个的Pod
- 后端的CacheServer角色的节点以及负责代理的节点都是部署成两组不同的Replication Controller
- cacheserver中每个pod都通过flanneld绑定不同的ip,proxy server节点也是绑定不同的ip以及相同的端口9099
现在问题是proxy server的每个ip都是不固定的，这样在weblogic端就需要指定后端动态变化的ip连入集群。初步想法是让proxy pod绑定服务，然后通过servicename接入。
所以首先需要配置的是dns,作用是每个weblogic pod都能通过dns解析servicename,并转换到proxy server的具体地址。
构建Coherence Proxy Images
其实coherence cacheserver和proxyserver可以定制一个image,然后通过不同的参数来切换不同的配置文件，但这里采用最简单的办法，针对不同角色构建不同的images.
在 /home/weblogic/docker/OracleCoherence/dockerfiles/12.2.1.0.0 目录下加入一个文件proxy-cache-config.xml作为proxyserver启动的配置参数。
[root@k8s-node-1 12.2.1.0.0]# cat proxy-cache-config.xml <?xml version="1.0"?> <!DOCTYPE cache-config SYSTEM "cache-config.dtd"> <cache-config><caching-scheme-mapping><cache-mapping><cache-name>*</cache-name><scheme-name>distributed-scheme</scheme-name></cache-mapping></caching-scheme-mapping><caching-schemes><!-- Distributed caching scheme. --><distributed-scheme><scheme-name>distributed-scheme</scheme-name><service-name>DistributedCache</service-name><thread-count>50</thread-count><backup-count>1</backup-count><backing-map-scheme><local-scheme><scheme-name>LocalSizeLimited</scheme-name></local-scheme> </backing-map-scheme><autostart>true</autostart><local-storage>false</local-storage></distributed-scheme><local-scheme><scheme-name>LocalSizeLimited</scheme-name><eviction-policy>LRU</eviction-policy><high-units>500</high-units><unit-calculator>BINARY</unit-calculator><unit-factor>1048576</unit-factor><expiry-delay>48h</expiry-delay></local-scheme><proxy-scheme> <service-name>ExtendTcpProxyService</service-name> <thread-count>5</thread-count> <acceptor-config> <tcp-acceptor> <local-address> <address>0.0.0.0</address> <port>9099</port> </local-address> </tcp-acceptor> </acceptor-config> <autostart>true</autostart> </proxy-scheme> </caching-schemes> </cache-config>
注意这里的address,0.0.0.0意味着可以绑定任何生成的ip.
修改Dockerfile,最好建立一个新的Dockerfile.proxy
[root@k8s-node-1 12.2.1.0.0]# cat Dockerfile.proxy # LICENSE CDDL 1.0 + GPL 2.0 # # ORACLE DOCKERFILES PROJECT # -------------------------- # This is the Dockerfile for Coherence 12.2.1 Standalone Distribution # # REQUIRED BASE IMAGE TO BUILD THIS IMAGE # --------------------------------------- # This Dockerfile requires the base image oracle/serverjre:8 # (see https://github.com/oracle/docker-images/tree/master/OracleJava) # # REQUIRED FILES TO BUILD THIS IMAGE # ---------------------------------- # (1) fmw_12.2.1.0.0_coherence_Disk1_1of1.zip # # Download the Standalone installer from http://www.oracle.com/technetwork/middleware/coherence/downloads/index.html # # HOW TO BUILD THIS IMAGE # ----------------------- # Put all downloaded files in the same directory as this Dockerfile # Run: # $ sh buildDockerImage.sh -s # # or if your Docker client requires root access you can run: # $ sudo sh buildDockerImage.sh -s ## Pull base image # --------------- FROM oracle/serverjre:8# Maintainer # ---------- MAINTAINER Jonathan Knight# Environment variables required for this build (do NOT change) ENV FMW_PKG=fmw_12.2.1.0.0_coherence_Disk1_1of1.zip \FMW_JAR=fmw_12.2.1.0.0_coherence.jar \ORACLE_HOME=/u01/oracle/oracle_home \PATH=$PATH:/usr/java/default/bin:/u01/oracle/oracle_home/oracle_common/common/bin \CONFIG_JVM_ARGS="-Djava.security.egd=file:/dev/./urandom"ENV COHERENCE_HOME=$ORACLE_HOME/coherence# Copy files required to build this image COPY $FMW_PKG install.file oraInst.loc /u01/ COPY start.sh /start.sh COPY proxy-cache-config.xml $COHERENCE_HOME/conf/proxy-cache-config.xmlRUN useradd -b /u01 -m -s /bin/bash oracle && \echo oracle:oracle | chpasswd && \chmod +x /start.sh && \chmod a+xr /u01 && \chown -R oracle:oracle /u01USER oracle# Install and configure Oracle JDK # Setup required packages (unzip), filesystem, and oracle user # ------------------------------------------------------------ RUN cd /u01 && $JAVA_HOME/bin/jar xf /u01/$FMW_PKG && cd - && \$JAVA_HOME/bin/java -jar /u01/$FMW_JAR -silent -responseFile /u01/install.file -invPtrLoc /u01/oraInst.loc -jreLoc $JAVA_HOME -ignoreSysPrereqs -force -novalidation ORACLE_HOME=$ORACLE_HOME && \rm /u01/$FMW_JAR /u01/$FMW_PKG /u01/oraInst.loc /u01/install.fileENTRYPOINT ["/start.sh"]
区别在于需要把刚才的文件copy到容器中去
然后修改start.sh
[root@k8s-node-1 12.2.1.0.0]# cat start.sh
#!/usr/bin/env sh#!/bin/sh -e -x -utrap "echo TRAPed signal" HUP INT QUIT KILL TERMmain() {COMMAND=serverSCRIPT_NAME=$(basename "${0}")MAIN_CLASS="com.tangosol.net.DefaultCacheServer"case "${1}" inserver) COMMAND=${1}; shift ;;console) COMMAND=${1}; shift ;;queryplus) COMMAND=queryPlus; shift ;;help) COMMAND=${1}; shift ;;esaccase ${COMMAND} inserver) server ;;console) console ;;queryPlus) queryPlus ;;help) usage; exit ;;*) server ;;esac
}# ---------------------------------------------------------------------------
# Display the help text for this script
# ---------------------------------------------------------------------------
usage() {echo "Usage: ${SCRIPT_NAME} [type] [args]"echo ""echo "type: - the type of process to run, must be one of:"echo " server - runs a storage enabled DefaultCacheServer"echo " (server is the default if type is omitted)"echo " console - runs a storage disabled Coherence console"echo " query - runs a storage disabled QueryPlus session"echo " help - displays this usage text"echo ""echo "args: - any subsequent arguments are passed as program args to the main class"echo ""echo "Environment Variables: The following environment variables affect the script operation"echo ""echo "JAVA_OPTS - this environment variable adds Java options to the start command,"echo " for example memory and other system properties"echo ""echo "COH_WKA - Sets the WKA address to use to discover a Coherence cluster."echo ""echo "COH_EXTEND_PORT - If set the Extend Proxy Service will listen on this port instead"echo " of the default ephemeral port."echo ""echo "Any jar files added to the /lib folder will be pre-pended to the classpath."echo "The /conf folder is on the classpath so any files in this folder can be loaded by the process."echo ""
}server() {PROPS=""CLASSPATH=""MAIN_CLASS="com.tangosol.net.DefaultCacheServer"start
}console() {PROPS="-Dcoherence.localstorage=false"CLASSPATH=""MAIN_CLASS="com.tangosol.net.CacheFactory"start
}queryPlus() {PROPS="-Dcoherence.localstorage=false"CLASSPATH="${COHERENCE_HOME}/lib/jline.jar"MAIN_CLASS="com.tangosol.coherence.dslquery.QueryPlus"start
}start() {if [ "${COH_WKA}" != "" ]thenPROPS="${PROPS} -Dcoherence.wka=${COH_WKA}"fiif [ "${COH_EXTEND_PORT}" != "" ]thenPROPS="${PROPS} -Dcoherence.cacheconfig=extend-cache-config.xml -Dcoherence.extend.port=${COH_EXTEND_PORT}"fiCLASSPATH="/conf:/lib/*:${CLASSPATH}:${COHERENCE_HOME}/conf:${COHERENCE_HOME}/lib/coherence.jar"CMD="${JAVA_HOME}/bin/java -cp ${CLASSPATH} ${PROPS} -Dtangosol.coherence.distributed.localstorage=false -Dtangosol.coherence.cacheconfig=proxy-cache-config.xml ${JAVA_OPTS} ${MAIN_CLASS} ${COH_MAIN_ARGS}"echo "Starting Coherence ${COMMAND} using ${CMD}"exec ${CMD}
}main "$@"主要是在最后的java -cp中的修改。修改相应的build文件
最后build image(-s是指用standardalong版本)
sh buildProxyServer.sh -v 12.2.1.0.0 -s
构建Coherence CacheServer Images
按照类似方法，先新建一个storage-cache-server.xml
[root@k8s-node-1 12.2.1.0.0]# cat storage-cache-config.xml <?xml version="1.0"?> <!DOCTYPE cache-config SYSTEM "cache-config.dtd"> <cache-config><caching-scheme-mapping><cache-mapping><cache-name>*</cache-name><scheme-name>distributed-pof</scheme-name></cache-mapping></caching-scheme-mapping><caching-schemes><distributed-scheme><scheme-name>distributed-pof</scheme-name><service-name>DistributedCache</service-name><backing-map-scheme> <local-scheme/> </backing-map-scheme> <listener/><autostart>true</autostart><local-storage>true</local-storage></distributed-scheme></caching-schemes> </cache-config>
然后建立一个Dockerfile.cacheserver
[root@k8s-node-1 12.2.1.0.0]# cat Dockerfile.cacheserver # LICENSE CDDL 1.0 + GPL 2.0 # # ORACLE DOCKERFILES PROJECT # -------------------------- # This is the Dockerfile for Coherence 12.2.1 Standalone Distribution # # REQUIRED BASE IMAGE TO BUILD THIS IMAGE # --------------------------------------- # This Dockerfile requires the base image oracle/serverjre:8 # (see https://github.com/oracle/docker-images/tree/master/OracleJava) # # REQUIRED FILES TO BUILD THIS IMAGE # ---------------------------------- # (1) fmw_12.2.1.0.0_coherence_Disk1_1of1.zip # # Download the Standalone installer from http://www.oracle.com/technetwork/middleware/coherence/downloads/index.html # # HOW TO BUILD THIS IMAGE # ----------------------- # Put all downloaded files in the same directory as this Dockerfile # Run: # $ sh buildDockerImage.sh -s # # or if your Docker client requires root access you can run: # $ sudo sh buildDockerImage.sh -s ## Pull base image # --------------- FROM oracle/serverjre:8# Maintainer # ---------- MAINTAINER Jonathan Knight# Environment variables required for this build (do NOT change) ENV FMW_PKG=fmw_12.2.1.0.0_coherence_Disk1_1of1.zip \FMW_JAR=fmw_12.2.1.0.0_coherence.jar \ORACLE_HOME=/u01/oracle/oracle_home \PATH=$PATH:/usr/java/default/bin:/u01/oracle/oracle_home/oracle_common/common/bin \CONFIG_JVM_ARGS="-Djava.security.egd=file:/dev/./urandom"ENV COHERENCE_HOME=$ORACLE_HOME/coherence# Copy files required to build this image COPY $FMW_PKG install.file oraInst.loc /u01/ COPY start.sh /start.sh COPY storage-cache-config.xml $COHERENCE_HOME/conf/storage-cache-config.xmlRUN useradd -b /u01 -m -s /bin/bash oracle && \echo oracle:oracle | chpasswd && \chmod +x /start.sh && \chmod a+xr /u01 && \chown -R oracle:oracle /u01USER oracle# Install and configure Oracle JDK # Setup required packages (unzip), filesystem, and oracle user # ------------------------------------------------------------ RUN cd /u01 && $JAVA_HOME/bin/jar xf /u01/$FMW_PKG && cd - && \$JAVA_HOME/bin/java -jar /u01/$FMW_JAR -silent -responseFile /u01/install.file -invPtrLoc /u01/oraInst.loc -jreLoc $JAVA_HOME -ignoreSysPrereqs -force -novalidation ORACLE_HOME=$ORACLE_HOME && \rm /u01/$FMW_JAR /u01/$FMW_PKG /u01/oraInst.loc /u01/install.fileENTRYPOINT ["/start.sh"]
最后修改start.sh,关键语句是
CMD="${JAVA_HOME}/bin/java -cp ${CLASSPATH} ${PROPS} -Dtangosol.coherence.distributed.localstorage=true -Dtangosol.coherence.cacheconfig=storage-cache-config.xml ${JAVA_OPTS} ${MAIN_CLASS} ${COH_MAIN_ARGS}"最后build image
sh buildCacheServer.sh -v 12.2.1.0.0 -s
通过docker images看到
[root@k8s-node-1 12.2.1.0.0]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE 1213-domain v2 326bf14bb29f About an hour ago 2.055 GB oracle/coherence 12.2.1.0.0-cacheserver 57a90e86e1d2 20 hours ago 625 MB oracle/coherence 12.2.1.0.0-proxy 238c85d61468 23 hours ago 625 MB
在master节点创建一系列ReplicationController
coherence-proxy.yaml
[root@k8s-master ~]# cat coherence-proxy.yaml apiVersion: v1 kind: ReplicationController metadata:name: coherence-proxy spec:replicas: 2template:metadata:labels:coherencecluster: "proxy"version: "0.1"spec:containers:- name: coherenceproxyimage: oracle/coherence:12.2.1.0.0-proxyports:- containerPort: 9099 --- apiVersion: v1 kind: Service metadata:name: coherenceproxysvclabels:coherencecluster: proxy spec:type: NodePortports:- port: 9099protocol: TCPtargetPort: 9099nodePort: 30033selector:coherencecluster: proxy
coherence-cacheserver.yaml
[root@k8s-master ~]# cat coherence-cacheserver.yaml apiVersion: v1 kind: ReplicationController metadata:name: coherence-cacheserver spec:replicas: 2template:metadata:labels:coherencecluster: "mycluster"version: "0.1"spec:containers:- name: coherencecacheserverimage: oracle/coherence:12.2.1.0.0-cacheserver
kubectl create -f coherence-proxy.yamlkubectl create -f coherence-cacheserver.yaml
然后看看pod是否启动成功
[root@k8s-master ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE coherence-cacheserver-96kz7 1/1 Running 0 1h 192.168.33.4 k8s-node-1 coherence-cacheserver-z67ht 1/1 Running 0 1h 192.168.33.3 k8s-node-1 coherence-proxy-j7r0w 1/1 Running 0 1h 192.168.33.5 k8s-node-1 coherence-proxy-tg8n8 1/1 Running 0 1h 192.168.33.6 k8s-node-1
登录进去后确定coherence集群成员状态，看member成员的个数基本确定已经都加入集群了。
MasterMemberSet(ThisMember=Member(Id=8, Timestamp=2017-06-16 10:45:04.244, Address=192.168.33.6:38184, MachineId=42359, Location=machine:coherence-proxy-tg8n8,process:1, Role=CoherenceServer)OldestMember=Member(Id=1, Timestamp=2017-06-16 10:24:16.941, Address=192.168.33.4:46114, MachineId=10698, Location=machine:coherence-cacheserver-96kz7,process:1, Role=CoherenceServer)ActualMemberSet=MemberSet(Size=4Member(Id=1, Timestamp=2017-06-16 10:24:16.941, Address=192.168.33.4:46114, MachineId=10698, Location=machine:coherence-cacheserver-96kz7,process:1, Role=CoherenceServer)Member(Id=2, Timestamp=2017-06-16 10:24:20.836, Address=192.168.33.3:44182, MachineId=23654, Location=machine:coherence-cacheserver-z67ht,process:1, Role=CoherenceServer)Member(Id=7, Timestamp=2017-06-16 10:45:02.144, Address=192.168.33.5:39932, MachineId=47892, Location=machine:coherence-proxy-j7r0w,process:1, Role=CoherenceServer)Member(Id=8, Timestamp=2017-06-16 10:45:04.244, Address=192.168.33.6:38184, MachineId=42359, Location=machine:coherence-proxy-tg8n8,process:1, Role=CoherenceServer))MemberId|ServiceJoined|MemberState1|2017-06-16 10:24:16.941|JOINED,2|2017-06-16 10:24:20.836|JOINED,7|2017-06-16 10:45:02.144|JOINED,8|2017-06-16 10:45:04.244|JOINEDRecycleMillis=1200000RecycleSet=MemberSet(Size=4Member(Id=3, Timestamp=2017-06-16 10:25:25.741, Address=192.168.33.6:37346, MachineId=50576)Member(Id=4, Timestamp=2017-06-16 10:25:25.74, Address=192.168.33.5:46743, MachineId=65007)Member(Id=5, Timestamp=2017-06-16 10:43:45.413, Address=192.168.33.5:37568, MachineId=44312)Member(Id=6, Timestamp=2017-06-16 10:43:45.378, Address=192.168.33.6:33362, MachineId=42635)))TcpRing{Connections=[7]}
IpMonitor{Addresses=3, Timeout=15s}确定service状态
[root@k8s-master ~]# kubectl get services NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE coherenceproxysvc 10.254.22.102 <nodes> 9099:30033/TCP 1h kubernetes 10.254.0.1 <none> 443/TCP 26d
coherenceproxysvc已经启动。
开始Coherence客户端WebLogic Pod的配置
因为需要修改setDomainEnv.sh文件，将客户端的coherence配置文件写入，所以转入weblogic目录
[root@k8s-node-1 1213-domain]# pwd /home/weblogic/docker/OracleWebLogic/samples/1213-domain
新建一个proxy-client.xml
[root@k8s-node-1 1213-domain]# cat proxy-client.xml <?xml version="1.0"?> <!DOCTYPE cache-config SYSTEM "cache-config.dtd"> <cache-config><caching-scheme-mapping><cache-mapping><cache-name>*</cache-name><scheme-name>extend-dist</scheme-name></cache-mapping></caching-scheme-mapping><caching-schemes> <remote-cache-scheme><scheme-name>extend-dist</scheme-name><service-name>ExtendTcpCacheService</service-name><initiator-config><tcp-initiator><remote-addresses><socket-address><address>coherenceproxysvc</address><port>9099</port> </socket-address> </remote-addresses> <connect-timeout>10s</connect-timeout> </tcp-initiator> <outgoing-message-handler> <request-timeout>5s</request-timeout> </outgoing-message-handler> </initiator-config> </remote-cache-scheme> </caching-schemes> </cache-config>
需要注意的是address要指到service的名称，依靠dns去解析。
修改Dockerfile,核心是加入JAVA_OPTIONS和CLASSPATH.
FROM oracle/weblogic:12.1.3-generic# Maintainer
# ----------
MAINTAINER Bruno Borges <bruno.borges@oracle.com># WLS Configuration
# -------------------------------
ARG ADMIN_PASSWORD
ARG PRODUCTION_MODEENV DOMAIN_NAME="base_domain" \DOMAIN_HOME="/u01/oracle/user_projects/domains/base_domain" \ADMIN_PORT="7001" \ADMIN_HOST="wlsadmin" \NM_PORT="5556" \MS_PORT="7002" \PRODUCTION_MODE="${PRODUCTION_MODE:-prod}" \JAVA_OPTIONS="-Dweblogic.security.SSL.ignoreHostnameVerification=true -Dtangosol.coherence.distributed.localstorage=false -Dtangosol.coherence.cacheconfig=/u01/oracle/proxy-client.xml" \CLASSPATH="/u01/oracle/coherence.jar" \PATH=$PATH:/u01/oracle/oracle_common/common/bin:/u01/oracle/wlserver/common/bin:/u01/oracle/user_projects/domains/base_domain/bin:/u01/oracle# Add files required to build this image
USER oracle
COPY container-scripts/* /u01/oracle/
COPY coherence.jar /u01/oracle/
COPY proxy-client.xml /u01/oracle/# Configuration of WLS Domain
WORKDIR /u01/oracle
RUN /u01/oracle/wlst /u01/oracle/create-wls-domain.py && \mkdir -p /u01/oracle/user_projects/domains/base_domain/servers/AdminServer/security && \echo "username=weblogic" > /u01/oracle/user_projects/domains/base_domain/servers/AdminServer/security/boot.properties && \ echo "password=$ADMIN_PASSWORD" >> /u01/oracle/user_projects/domains/base_domain/servers/AdminServer/security/boot.properties && \echo ". /u01/oracle/user_projects/domains/base_domain/bin/setDomainEnv.sh" >> /u01/oracle/.bashrc && \ echo "export PATH=$PATH:/u01/oracle/wlserver/common/bin:/u01/oracle/user_projects/domains/base_domain/bin" >> /u01/oracle/.bashrc && \cp /u01/oracle/commEnv.sh /u01/oracle/wlserver/common/bin/commEnv.sh && \rm /u01/oracle/create-wls-domain.py /u01/oracle/jaxrs2-template.jar # Expose Node Manager default port, and also default http/https ports for admin console
EXPOSE $NM_PORT $ADMIN_PORT $MS_PORTWORKDIR $DOMAIN_HOME # Define default command to start bash.
CMD ["startWebLogic.sh"]然后build image
docker build -t 1213-domain:v2 --build-arg ADMIN_PASSWORD=welcome1 .
构建weblogic pod作为一个客户端
[root@k8s-master ~]# cat weblogic-pod.yaml apiVersion: v1 kind: Pod metadata:name: weblogic spec:containers:- name: weblogicimage: 1213-domain:v2ports:- containerPort: 7001
启动以后，通过log确定weblogic启动时确实把我们客户化的参数加入
[root@k8s-master ~]# kubectl logs weblogic . . JAVA Memory arguments: -Djava.security.egd=file:/dev/./urandom . CLASSPATH=/u01/oracle/wlserver/../oracle_common/modules/javax.persistence_2.1.jar:/u01/oracle/wlserver/../wlserver/modules/com.oracle.weblogic.jpa21support_1.0.0.0_2-1.jar:/usr/java/jdk1.8.0_101/lib/tools.jar:/u01/oracle/wlserver/server/lib/weblogic_sp.jar:/u01/oracle/wlserver/server/lib/weblogic.jar:/u01/oracle/wlserver/../oracle_common/modules/net.sf.antcontrib_1.1.0.0_1-0b3/lib/ant-contrib.jar:/u01/oracle/wlserver/modules/features/oracle.wls.common.nodemanager_2.0.0.0.jar:/u01/oracle/wlserver/../oracle_common/modules/com.oracle.cie.config-wls-online_8.1.0.0.jar:/u01/oracle/wlserver/common/derby/lib/derbyclient.jar:/u01/oracle/wlserver/common/derby/lib/derby.jar:/u01/oracle/wlserver/server/lib/xqrl.jar:/u01/oracle/coherence.jar . PATH=/u01/oracle/wlserver/server/bin:/u01/oracle/wlserver/../oracle_common/modules/org.apache.ant_1.9.2/bin:/usr/java/jdk1.8.0_101/jre/bin:/usr/java/jdk1.8.0_101/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/java/default/bin:/u01/oracle/oracle_common/common/bin:/u01/oracle/oracle_common/common/bin:/u01/oracle/wlserver/common/bin:/u01/oracle/user_projects/domains/base_domain/bin:/u01/oracle . *************************************************** * To start WebLogic Server, use a username and * * password assigned to an admin-level user. For * * server administration, use the WebLogic Server * * console at http://hostname:port/console * *************************************************** starting weblogic with Java version: java version "1.8.0_101" Java(TM) SE Runtime Environment (build 1.8.0_101-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode) Starting WLS with line: /usr/java/jdk1.8.0_101/bin/java -server -Djava.security.egd=file:/dev/./urandom -Dweblogic.Name=AdminServer -Djava.security.policy=/u01/oracle/wlserver/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.ignoreHostnameVerification=true -Dtangosol.coherence.distributed.localstorage=false -Dtangosol.coherence.cacheconfig=/u01/oracle/proxy-client.xml -Djava.endorsed.dirs=/usr/java/jdk1.8.0_101/jre/lib/endorsed:/u01/oracle/wlserver/../oracle_common/modules/endorsed -da -Dwls.home=/u01/oracle/wlserver/server -Dweblogic.home=/u01/oracle/wlserver/server -Dweblogic.utils.cmm.lowertier.ServiceDisabled=true weblogic.Server
部署一个HelloWorld.war文件，核心的index.jsp代码是:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<%@page import="java.util.*"%>
<%@page import="com.tangosol.net.*"%>
<%@ page contentType="text/html;charset=windows-1252"%><html><body> This is a Helloworld test</body>
<h3><%String mysession;NamedCache cache;cache = CacheFactory.getCache("demoCache");cache.put("eric","eric.nie@oracle.com");%>Get Eric Email:<%=cache.get("eric").toString()%></h3></html>[root@k8s-master ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE coherence-cacheserver-96kz7 1/1 Running 0 1h 192.168.33.4 k8s-node-1 coherence-cacheserver-z67ht 1/1 Running 0 1h 192.168.33.3 k8s-node-1 coherence-proxy-j7r0w 1/1 Running 0 1h 192.168.33.5 k8s-node-1 coherence-proxy-tg8n8 1/1 Running 0 1h 192.168.33.6 k8s-node-1 weblogic 1/1 Running 0 1h 192.168.33.7 k8s-node-1
部署后访问
查看后面的weblogic日志
<Jun 16, 2017 10:32:42 AM GMT> <Notice> <WebLogicServer> <BEA-000329> <Started the WebLogic Server Administration Server "AdminServer" for domain "base_domain" running in production mode.> <Jun 16, 2017 10:32:42 AM GMT> <Notice> <WebLogicServer> <BEA-000360> <The server started in RUNNING mode.> <Jun 16, 2017 10:32:42 AM GMT> <Warning> <Server> <BEA-002611> <The hostname "localhost", maps to multiple IP addresses: 127.0.0.1, 0:0:0:0:0:0:0:1.> <Jun 16, 2017 10:32:42 AM GMT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING.> 2017-06-16 10:36:29.193/280.892 Oracle Coherence 12.1.3.0.0 <Info> (thread=[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Loaded operational configuration from "jar:file:/u01/oracle/coherence/lib/coherence.jar!/tangosol-coherence.xml" 2017-06-16 10:36:29.451/281.086 Oracle Coherence 12.1.3.0.0 <Info> (thread=[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Loaded operational overrides from "jar:file:/u01/oracle/coherence/lib/coherence.jar!/tangosol-coherence-override-dev.xml" 2017-06-16 10:36:29.479/281.114 Oracle Coherence 12.1.3.0.0 <D5> (thread=[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Optional configuration override "/tangosol-coherence-override.xml" is not specified 2017-06-16 10:36:29.511/281.146 Oracle Coherence 12.1.3.0.0 <D5> (thread=[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Optional configuration override "cache-factory-config.xml" is not specified 2017-06-16 10:36:29.525/281.159 Oracle Coherence 12.1.3.0.0 <D5> (thread=[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Optional configuration override "cache-factory-builder-config.xml" is not specified 2017-06-16 10:36:29.526/281.161 Oracle Coherence 12.1.3.0.0 <D5> (thread=[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Optional configuration override "/custom-mbeans.xml" is not specifiedOracle Coherence Version 12.1.3.0.0 Build 52031Grid Edition: Development mode Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.2017-06-16 10:36:29.672/281.306 Oracle Coherence GE 12.1.3.0.0 <Info> (thread=[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Loaded cache configuration from "file:/u01/oracle/proxy-client.xml"; this document does not refer to any schema definition and has not been validated. 2017-06-16 10:36:30.225/281.860 Oracle Coherence GE 12.1.3.0.0 <Info> (thread=[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Created cache factory com.tangosol.net.ExtensibleConfigurableCacheFactory 2017-06-16 10:36:30.507/282.142 Oracle Coherence GE 12.1.3.0.0 <D5> (thread=[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Connecting Socket to 10.254.203.94:9099 2017-06-16 10:36:30.534/282.169 Oracle Coherence GE 12.1.3.0.0 <Info> (thread=[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Error connecting Socket to 10.254.203.94:9099: java.net.ConnectException: Connection refused 2017-06-16 10:46:11.056/862.693 Oracle Coherence GE 12.1.3.0.0 <Info> (thread=[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Restarting Service: ExtendTcpCacheService 2017-06-16 10:46:11.156/862.791 Oracle Coherence GE 12.1.3.0.0 <D5> (thread=[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Connecting Socket to 10.254.22.102:9099 2017-06-16 10:46:11.188/862.837 Oracle Coherence GE 12.1.3.0.0 <Info> (thread=[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)', member=n/a): Connected Socket to 10.254.22.102:9099
问题和定位
关于DNS解析是否正确，可以通过下面命令
iptables -L -v -n -t nat
查看路由是否正确，主要是看coherenceproxysvc的路由是否到正确的pod和端口.
[root@k8s-node-1 1213-domain]# iptables -L -v -n -t nat Chain PREROUTING (policy ACCEPT 5 packets, 309 bytes)pkts bytes target prot opt in out source destination 30708 1898K KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */3 172 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCALChain INPUT (policy ACCEPT 0 packets, 0 bytes)pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)pkts bytes target prot opt in out source destination 2057 126K KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */0 0 DOCKER all -- * * 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCALChain POSTROUTING (policy ACCEPT 4 packets, 240 bytes)pkts bytes target prot opt in out source destination 4100 283K MASQUERADE all -- * !docker0 192.168.33.0/24 0.0.0.0/0 28494 1716K KUBE-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes postrouting rules */2 277 RETURN all -- * * 192.168.122.0/24 224.0.0.0/24 0 0 RETURN all -- * * 192.168.122.0/24 255.255.255.255 0 0 MASQUERADE tcp -- * * 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-655350 0 MASQUERADE udp -- * * 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-655350 0 MASQUERADE all -- * * 192.168.122.0/24 !192.168.122.0/24 Chain DOCKER (2 references)pkts bytes target prot opt in out source destination 0 0 RETURN all -- docker0 * 0.0.0.0/0 0.0.0.0/0 Chain KUBE-MARK-DROP (0 references)pkts bytes target prot opt in out source destination 0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK or 0x8000Chain KUBE-MARK-MASQ (6 references)pkts bytes target prot opt in out source destination 0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK or 0x4000Chain KUBE-NODEPORTS (1 references)pkts bytes target prot opt in out source destination 0 0 KUBE-MARK-MASQ tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/coherenceproxysvc: */ tcp dpt:300330 0 KUBE-SVC-BQXHRGVXFCEH2BHH tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/coherenceproxysvc: */ tcp dpt:30033Chain KUBE-POSTROUTING (1 references)pkts bytes target prot opt in out source destination 0 0 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000Chain KUBE-SEP-67FRRWLKQK2OD4HZ (1 references)pkts bytes target prot opt in out source destination 0 0 KUBE-MARK-MASQ all -- * * 192.168.33.2 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */ tcp to:192.168.33.2:53Chain KUBE-SEP-GIM2MHZZZBZJL55J (2 references)pkts bytes target prot opt in out source destination 0 0 KUBE-MARK-MASQ all -- * * 192.168.0.105 0.0.0.0/0 /* default/kubernetes:https */0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes:https */ recent: SET name: KUBE-SEP-GIM2MHZZZBZJL55J side: source mask: 255.255.255.255 tcp to:192.168.0.105:443Chain KUBE-SEP-IM4M52WKVEC4AZF3 (1 references)pkts bytes target prot opt in out source destination 0 0 KUBE-MARK-MASQ all -- * * 192.168.33.6 0.0.0.0/0 /* default/coherenceproxysvc: */0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/coherenceproxysvc: */ tcp to:192.168.33.6:9099Chain KUBE-SEP-LUF3R3GRCSK6KKRS (1 references)pkts bytes target prot opt in out source destination 0 0 KUBE-MARK-MASQ all -- * * 192.168.33.2 0.0.0.0/0 /* kube-system/kube-dns:dns */0 0 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */ udp to:192.168.33.2:53Chain KUBE-SEP-ZZECWQBQCJPODCBC (1 references)pkts bytes target prot opt in out source destination 0 0 KUBE-MARK-MASQ all -- * * 192.168.33.5 0.0.0.0/0 /* default/coherenceproxysvc: */0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/coherenceproxysvc: */ tcp to:192.168.33.5:9099Chain KUBE-SERVICES (2 references)pkts bytes target prot opt in out source destination 0 0 KUBE-SVC-NPX46M4PTMTKRN6Y tcp -- * * 0.0.0.0/0 10.254.0.1 /* default/kubernetes:https cluster IP */ tcp dpt:4430 0 KUBE-SVC-TCOU7JCQXEZGVUNU udp -- * * 0.0.0.0/0 10.254.254.254 /* kube-system/kube-dns:dns cluster IP */ udp dpt:530 0 KUBE-SVC-ERIFXISQEP7F7OF4 tcp -- * * 0.0.0.0/0 10.254.254.254 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:530 0 KUBE-SVC-BQXHRGVXFCEH2BHH tcp -- * * 0.0.0.0/0 10.254.22.102 /* default/coherenceproxysvc: cluster IP */ tcp dpt:90990 0 KUBE-NODEPORTS all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCALChain KUBE-SVC-BQXHRGVXFCEH2BHH (2 references)pkts bytes target prot opt in out source destination 0 0 KUBE-SEP-ZZECWQBQCJPODCBC all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/coherenceproxysvc: */ statistic mode random probability 0.500000000000 0 KUBE-SEP-IM4M52WKVEC4AZF3 all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/coherenceproxysvc: */Chain KUBE-SVC-ERIFXISQEP7F7OF4 (1 references)pkts bytes target prot opt in out source destination 0 0 KUBE-SEP-67FRRWLKQK2OD4HZ all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */Chain KUBE-SVC-NPX46M4PTMTKRN6Y (1 references)pkts bytes target prot opt in out source destination 0 0 KUBE-SEP-GIM2MHZZZBZJL55J all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes:https */ recent: CHECK seconds: 10800 reap name: KUBE-SEP-GIM2MHZZZBZJL55J side: source mask: 255.255.255.2550 0 KUBE-SEP-GIM2MHZZZBZJL55J all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes:https */Chain KUBE-SVC-TCOU7JCQXEZGVUNU (1 references)pkts bytes target prot opt in out source destination 0 0 KUBE-SEP-LUF3R3GRCSK6KKRS all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */
可优化之处：
- 构建image脚本沿用官方修改，很多不必要的地方可以删除
- cacheserver和proxy的image可以采用同一个，通过不同的参数调用不同的命令启动。
转载于:https://www.cnblogs.com/ericnie/p/7029083.html
Kubernetes环境下如何运行Coherence缓存集群相关推荐
- linux环境下通过nginx实现tomcat集群
2019独角兽企业重金招聘Python工程师标准>>> linux环境下通过nginx实现tomcat集群 安装nginx之前需要pcre依赖和jvm-remote补丁 一.准备如下 ...
- Windows环境下安装HBase(Hadoop3.1.3、HBase2.2.5）与Linux环境下搭建HBase高可用集群
Windows环境下安装HBase 下载HBase 官网: https://hbase.apache.org/downloads.html 不同版本集合:https://archive.apache. ...
- nginx集群_windows环境下搭建简单Nginx+Tomcat集群
通俗点将,负载均衡就是因为访问流量太大,导致项目访问不流畅.甚至宕掉,所以通过一种分流的方式来缓解这种情况. 一. 工具 nginx-1.8.0 apache-tomcat-6.0.33 二. 目标 ...
- linux下mysql 有双机_在Linux环境下配置mysql cluster双机集群
一.介绍 如何安装配置基于2台服务器的MySQL集群.并且实现任意一台服务器出现问题或宕机时MySql集群依然能够继续运行.加下后续的(keepalived+lvs+mysql cluster文档), ...
- [Elasticsearch](五)Docker环境下搭建Elasticsearch，Elasticsearch集群，Elasticsearch-Head以及IK分词插件和拼音分词插件
目录: https://github.com/dolyw/ProjectStudy/tree/master/Elasticsearch DockerStudy dolyw:https://note.d ...
- Linux环境快速搭建elasticsearch6.5.4集群和Head插件
今天来一起实战Linux环境下搭建elasticsearch6.5.4集群和Head插件: 环境信息 操作系统:Ubuntu 16.04.5 LTS JDK:1.8.0_191 elasticsear ...
- 检查是否禁止asp.net服务扩展_在 Kubernetes 环境下部署 OpenWhisk 服务
本文使用 Zhihu On VSCode 创作并发布 1. 总体目标 终于开始准备毕业设计了.在和导师们的讨论之后,我们确定之后的研究方向将会专注于 Serverless Computing. 先长舒 ...
- 云原生之在kubernetes环境下部署wordpress
云原生之在kubernetes环境下部署wordpress 一.wordpress介绍 1.wordpress简介 2.wordpress特点 二.kubernetes集群介绍 1.k8s简介 2.k ...
- 项目总结10：通过反射解决springboot环境下从redis取缓存进行转换时出现ClassCastException异常问题...
通过反射解决springboot环境下从redis取缓存进行转换时出现ClassCastException异常问题 关键字 springboot热部署 ClassCastException异常 反射 ...
- Ubuntu 环境下交叉编译运行正点原子的程序
Ubuntu 环境下交叉编译运行正点原子的程序 在校学生,求职过程中对方要求在Ubuntu环境下交叉编译stm32的程序,我觉得很有意思,因为之前就听说过可以在Linux环境下运行stm32的程序,因 ...
最新文章
- KDD 2021 | 一种使用真负样本的在线延迟反馈建模
- React-Router ---withRouter
- 2019牛客多校第一场
- c++中new的总结（动态管理，malloc存在的问题，malloc与new的区别）
- py语法错误与异常处理
- arcgis选出点规定范围的面
- Linux C基础笔记（3）
- 实现二叉树的基本操作(Java版)
- 重定向与请求转发的区别
- webservice的wsdl文件怎么生成
- Matlab常用的滤波函数集合（会陆续更新详解）
- python处理word文档格式_python处理word文档（docxtpl 中文说明文档）
- 手机端自适应表格table样式如何写
- 排查并解决磁盘满了问题
- SQL数据库查询语句
- 工业虚拟现实解决方案
- 冬季你最需要什么水果？
- unity rtsp 视频渲染(一)
- android毛玻璃效果，模糊视图工具类！类 高斯模糊
- 基于java旅游管理系统计算机毕业设计源码+系统+lw文档+mysql数据库+调试部署