文章目录

基本情况
环境配置
- 所需环境 CESM2 Software/Operating System Prerequisites
- 使用source脚本配置模块和环境变量
- MPI测试
CESM安装及移植
- CESM源码下载
- 在CIME中配置天河
- - config_machines.xml
  - config_compilers.xml
  - config_batch.xml
CESM简单运行流程
- 在无法连接外网下使用chksum
验证CESM是否移植成功
附录
- 相关文档
- 天河MPI编译环境

基本情况

超算中心：国家超级计算天津中心
软件名称：CESM
软件版本：2.2.0
集群：TH-1A
编译节点：TH-1A-LN9
编译器版本：Intel_compiler/17.0.4
MPI版本：MPI/mpich/intel2017

环境配置

所需环境 CESM2 Software/Operating System Prerequisites

参考：https://escomp.github.io/CESM/versions/cesm2.2/html/introduction.html#cesm2-software-operating-system-prerequisites

UNIX style operating system such as CNL, AIX or Linux
python >= 2.7
perl 5
subversion client (version 1.8 or greater but less than v1.11) for downloading CAM, POP, and WW3
git client (1.8 or greater)
Fortran compiler with support for Fortran 2003
C compiler
MPI (although CESM does not absolutely require it for running on one processor)
NetCDF 4.3 or newer.
ESMF 5.2.0 or newer (optional).
pnetcdf 1.7.0 is required and 1.8.1 is optional but recommended
Trilinos may be required for certain configurations
LAPACK and BLAS
CMake 2.8.6 or newer

若perl>=5.6
需要把<CESM源码目录>/components/clm/bld/CLMBuildNamelist.pm里面的bsd_glob改成glob
"Since v5.6.0, Perl’s CORE::glob() is implemented in terms of bsd_glob()."libxml2问题
libreadline.so.5
'/vol-th/software/libxml2-291/bin/xmllint: error while loading shared libraries: libreadline.so.5: cannot open shared object file: No such file or directory'
ERROR: Command: '/vol-th/software/libxml2-291/bin/xmllint --xinclude --noout --schema /vol-th/home/laiyq/cesm/cesm220/cime/config/xml_schemas/env_entry_id.xsd /vol-th/home/laiyq/cesm/scratch/F2000climo/env_case.xml' failed with error '/vol-th/software/libxml2-291/bin/xmllint: error while loading shared libraries: libreadline.so.5: cannot open shared object file: No such file or directory' from dir '/vol-th/home/laiyq/cesm/scratch/F2000climo
在TH-1A里，不导入perl和libxml2，用原本自带的反而正常运行了！可能和版本有关？ld: cannot find -lmkl_cdft_core
在TH-1A使用Intel_complier2018会出现该问题，因为mkl版本不是cluster版本的
Interl_complier2017是对的！Interl_complier2017使用impi在yhrun/yhbatch/sbatch时会出现错误！用mpich

使用source脚本配置模块和环境变量

source cesm_env_init

echo ""
echo "CESM Env Init"
echo ""# This is Module load part (DO NOT OVERWRITE IT)
source ~/../../software/modules/3.2.10/Modules/3.2.10/init/bashmodule purgemodule add Intel_compiler/17.0.4
module add MPI/mpich/intel2017
#import IMPI/17.0.4 manualy
#export PATH=/vol-th/software/intel2017.4/compilers_and_libraries_2017.4.196/linux/mpi/intel64/bin:$PATH
#export LIBRARY_PATH=/vol-th/software/intel2017.4/compilers_and_libraries_2017.4.196/linux/mpi/intel64/lib:$LIBRARY_PATH
#export LD_LIBRARY_PATH=/vol-th/software/intel2017.4/compilers_and_libraries_2017.4.196/linux/mpi/intel64/lib:$LD_LIBRARY_PATHmodule add cmake/3.20.3
module add netcdf/4.4
module add pnetcdf/1.6.1
module add python/3.7_anacondamodule add lapack/3.8.0
module add blas/200703module add mkl/17.0.4
module add loginnode/ln9#module add hdf5/1.8.11export PERL5LIB=/vol-th/software/loginnode/ln9/usr/lib64/perl5/vendor_perl:$PERL5LIB#export LD_LIBRARY_PATH=/vol-th/software/io_tools/hdf5/mpi/1.8.11/lib:/vol-th/software/io_tools/netcdf/mpi/4.4/lib:/vol-th/software/io_tools/pnetcdf/1.6.1/lib:$LD_LIBRARY_PATHexport CESMDATAROOT=/vol-th/home/laiyq/cesm
export CESMROOT=/vol-th/home/laiyq/cesm
export CIME_MACHINE=tianhe-1A
export CIMEROOT=/vol-th/home/laiyq/cesm/cesm220/cime

MPI测试

It is usually very helpful to assure that you can run a basic mpi parallel program on your machine prior to attempting a CIME port. Understanding how to compile and run the program fhello_world_mpi.F90 shown here could potentially save many hours of frustration.

program fhello_world_mpiuse mpiimplicit noneinteger ( kind = 4 ) errorinteger ( kind = 4 ) idinteger pcharacter(len=MPI_MAX_PROCESSOR_NAME) :: nameinteger cleninteger, allocatable :: mype(:)real ( kind = 8 ) wtimecall MPI_Init ( error )call MPI_Comm_size ( MPI_COMM_WORLD, p, error )call MPI_Comm_rank ( MPI_COMM_WORLD, id, error )if ( id == 0 ) thenwtime = MPI_Wtime ( )write ( *, '(a)' ) ' 'write ( *, '(a)' ) 'HELLO_MPI - Master process:'write ( *, '(a)' ) '  FORTRAN90/MPI version'write ( *, '(a)' ) ' 'write ( *, '(a)' ) '  An MPI test program.'write ( *, '(a)' ) ' 'write ( *, '(a,i8)' ) '  The number of processes is ', pwrite ( *, '(a)' ) ' 'end ifcall MPI_GET_PROCESSOR_NAME(NAME, CLEN, ERROR)write ( *, '(a)' ) ' 'write ( *, '(a,i8,a,a)' ) '  Process ', id, ' says "Hello, world!" ',name(1:clen)call MPI_Finalize ( error )
end program

在登录节点上进行测试：

mpif90 fhello_world_mpi.F90 -o hello_world
yhrun -n 24 --ntasks-per-node 12 -p debug ./hello_world

CESM安装及移植

CESM源码下载

由于天河环境无法连接外网，需在本地下载好代码（并且完成“./manage_externals/checkout_externals”）后上传至天河。

源码地址及下载教程：https://github.com/ESCOMP/CESM

inputdata同理。

/vol6/software/CESM2里面有inputdata！还有LUH2部分数据。
但是case.submit以后会有问题，经测试计算节点无法访问/vol6
/vol-th/software/inputdata里也有，但不完整！

/vol6/software/CESM2的du结果：
11T ./inputdata/atm
979G    ./inputdata/cesm2_init
387M    ./inputdata/data_licom
4.5G    ./inputdata/dx7
741M    ./inputdata/glc
44G ./inputdata/ice
4.0K    ./inputdata/import
8.2T    ./inputdata/lnd
3.7T    ./inputdata/ocn
7.8G    ./inputdata/rof
33G ./inputdata/share
18G ./inputdata/validation
19M ./inputdata/wav
232G    ./inputdata/ccsm4_init
148G    ./inputdata/cpl
216M    ./inputdata/mycesm2-inputdata
24T ./inputdata

在CIME中配置天河

参考：http://esmci.github.io/cime/versions/master/html/users_guide/machine.html

参考：https://blog.csdn.net/qq_38607066/article/details/109260154

主要修改以下几个文件：

config_machines.xml
- 定义机器
- file containing machine specifications for target model primary component
- $CIMEROOT/config/cesm/machines/config_machines.xml
config_compilers.xml
- 设置编译器路径和选项
- file containing compiler specifications for target model primary component
- $CIMEROOT/config/cesm/machines/config_compilers.xml
- PS：在最新版CIME中已弃用，但在CESM使用的CIME中仍保留。
config_batch.xml
- 定义批处理系统
- file containing batch system details for target system
- $CIMEROOT/config/cesm/machines/config_batch.xml
修改方式
- mkdir ~/.cime
- cd ~/.cime
- 创建<config_xxx>.xml文件
- 运行$CIMEROOT/scripts/tests/scripts_regression_tests.py 测试

config_machines.xml

<?xml version="1.0"?> <config_machines version="2.0"><machine MACH="tianhe-1A"><DESC>NSCC-tj Tianhe-1A, RHEL 5.3 2.6.32-358.123.4-TH, 12 pes/node, batch system is Slurm</DESC><NODENAME_REGEX>ln9.*</NODENAME_REGEX><OS>LINUX</OS><COMPILERS>intel</COMPILERS><MPILIBS>mpich</MPILIBS><PROJECT>none</PROJECT><CIME_OUTPUT_ROOT>$ENV{CESMROOT}/scratch</CIME_OUTPUT_ROOT><DIN_LOC_ROOT>$ENV{CESMDATAROOT}/inputdata</DIN_LOC_ROOT><DIN_LOC_ROOT_CLMFORC>$DIN_LOC_ROOT/atm/datm7</DIN_LOC_ROOT_CLMFORC><DOUT_S_ROOT>$ENV{CESMROOT}/archive/$CASE</DOUT_S_ROOT><BASELINE_ROOT>$ENV{CESMROOT}/baselines</BASELINE_ROOT><CCSM_CPRNC>$ENV{CIMEROOT}/tools/cprnc</CCSM_CPRNC><GMAKE>make</GMAKE><GMAKE_J>8</GMAKE_J><BATCH_SYSTEM>slurm</BATCH_SYSTEM><SUPPORTED_BY>laiyq@pku.edu.cn</SUPPORTED_BY><MAX_TASKS_PER_NODE>12</MAX_TASKS_PER_NODE><MAX_MPITASKS_PER_NODE>12</MAX_MPITASKS_PER_NODE><PROJECT_REQUIRED>FALSE</PROJECT_REQUIRED><mpirun mpilib="mpich"><executable>yhrun</executable><arguments><arg name="num_tasks" > -n {{ total_tasks }}</arg></arguments></mpirun><module_system type="none"></module_system><environment_variables><env name="OMP_STACKSIZE">256M</env></environment_variables><resource_limits><resource name="RLIMIT_STACK">-1</resource></resource_limits></machine>
</config_machines>

config_compilers.xml

<?xml version="1.0"?><config_compilers version="2.0"><compiler COMPILER="intel" MACH="tianhe-1A"><CPPDEFS><append>-DLINUX</append></CPPDEFS><LDFLAGS><append compile_threaded="TRUE"> -fopenmp </append></LDFLAGS><CFLAGS><append DEBUG="FALSE"> -O2 </append></CFLAGS><CONFIG_ARGS><base> --host=Linux </base></CONFIG_ARGS><FFLAGS><append DEBUG="FALSE"> -O2 </append></FFLAGS><MPICC> /vol-th/software/mpi/mpi-intel2017/bin/mpicc </MPICC><MPICXX> /vol-th/software/mpi/mpi-intel2017/bin/mpicxx </MPICXX><MPIFC> /vol-th/software/mpi/mpi-intel2017/bin/mpif90 </MPIFC><CXX_LINKER>FORTRAN</CXX_LINKER><SUPPORTS_CXX>TRUE</SUPPORTS_CXX><NETCDF_PATH> /vol-th/software/io_tools/netcdf/mpi/4.4</NETCDF_PATH><PNETCDF_PATH> /vol-th/software/io_tools/pnetcdf/1.6.1</PNETCDF_PATH><SLIBS><append>-L${NETCDF_PATH}/lib -L${PNETCDF_PATH}/lib -lnetcdff -lnetcdf -llapack -lblas -mkl=cluster</append></SLIBS></compiler>
</config_compilers>

config_batch.xml

注意分debug队列和TH_NET队列！

<?xml version="1.0"?><config_batch version="2.0"><batch_system MACH="tianhe-1A" type="slurm" ><batch_submit>yhbatch</batch_submit><submit_args><arg flag="--time" name="$JOB_WALLCLOCK_TIME"/><arg flag="-p" name="$JOB_QUEUE"/></submit_args><queues><queue walltimemax="48:00:00" default="true">TH_NET</queue><queue walltimemax="00:30:00">debug</queue></queues></batch_system>
</config_batch>

CESM简单运行流程

cd $CIMEROOT/scripts
./create_newcase --case $CESMROOT/scratch/<case_name> --compset <compset> --res <res> --machine tianhe-1A --run-unsupported
cd $CESMROOT/scratch/<case_name>
./xmlchange STOP_OPTION=nmonths,STOP_N=1,NTASKS=60
./xmlquery STOP_OPTION,STOP_N,NTASKS,NTHRDS,ROOTPE./xmlchange JOB_QUEUE=TH_NET
./xmlchange JOB_WALLCLOCK_TIME=48:00:00./case.setup
./preview_run
./case.build --skip-provenance-check
cp $CESMROOT/inputdata_checksum.dat ./run/
./check_input_data --chksum
./case.submit#以下部分为continueing a run
./xmlchange CONTINUE_RUN=true,STOP_N=28
./case.subm

在无法连接外网下使用chksum

很多时候模型运行错误是因为输入数据下载出错/不完整，因此最好使用./check_input_data --chksum进行数据检验，但是无法连接外网的情况下不能下载校验文件inputdata_checksum.dat。可通过以下方法解决：

获取一份format过的inputdata_checksum.dat（不能直接用UCAR服务器上的，格式不对）。
笔者在此提供一份：https://www.aliyundrive.com/s/r3x5GzV5bia

修改$CIMEROOT/scripts/lib/CIME/case/check_input_data.py中的def _download_checksum_file(rundir)

def _download_checksum_file(rundir):
....expect(False, "Unsupported inputdata protocol: {}".format(protocol))if not server:continue# 从此处开始修改rel_path = chksum_filefull_path = os.path.join(rundir, local_chksum_file)new_file = full_path + '.raw'if chksum_file or os.path.isfile(full_path):chksum_found = Trueelse:continue# 到此处修改结束success = Falseprotocol = type(server).__name__
....

复制inputdata_checksum.dat到<CASEROOT>/run中，运行检查

cp $CESMROOT/inputdata_checksum.dat ./run/
./check_input_data --chksum

验证CESM是否移植成功

参考：http://esmci.github.io/cime/versions/master/html/users_guide/porting-cime.html

基本功能测试

这个测试耗时非常久，暂时不做

Verify basic functionality of your port by performing the cheyenne “prealpha” tests on your machine. This can be done by issuing the following command:

./create_test --xml-category prealpha --xml-machine cheyenne --xml-compiler intel --machine <your_machine_name> --compiler <your_compiler_name>

This command will run the prealpha tests defined for cheyenne with the intel compiler, but will run them on your machine with your compiler. These tests will be run in the $CIME_OUTPUT_ROOT. To see the results of tests, you need to do the following:

> $CIME_OUTPUT_ROOT/cs.status.[testid]

where testid was indicated in the output when calling create_test

./create_test --xml-category prealpha --xml-machine cheyenne --xml-compiler intel --machine tianhe-1A --compiler intel

ensemble一致性测试

Carry out ensemble consistency tests:
This is described in $CIMEROOT/tools/statistical_ensemble_test/README. The CESM-ECT (CESM Ensemble Consistency Test) determines whether a new simulation set up (new machine, compiler, etc.) is statistically distinguishable from an accepted ensemble. The ECT process involves comparing several runs (3) generated with the new scenario to an ensemble built on a trusted machine (currently cheyenne). The python ECT tools are located in the pyCECT subdirectory $CIMEROOT/tools/statistical_ensemble_test/pyCECT
The verification tools in the CESM-ECT suite are:
CAM-ECT: detects issues in CAM and CLM (12 month runs)
UF-CAM-ECT: detects issues in CAM and CLM (9 time step runs)
POP-ECT: detects issues in POP and CICE (12 month runs)
Follow the instructions in the README file to generate three ensemble runs for any of the above tests that are most relevant to your port. Then please go to the CESM2 ensemble verification website, where you can upload your files and subsequently obtain a quick response as to the success or failure of your verification.

使用UF-CAM-ECT来进行ensemble consistency快速验证

具体参考：https://www.cesm.ucar.edu/models/cesm2/python-tools/

UF-CAM-ECT - resolution = f19_f19_mg17, compset = F2000climo (9 time step runs)

#The ensemble.sh script is located in cime/tools/statistical_ensemble_test
python ensemble.py --case /vol-th/home/laiyq/cesm/scratch/case.cesm220.uf.000 --ect cam --uf --mach tianhe-1A --compset F2000climo --res f19_f19_mg17 --compiler intel

注意由于不能连接外网，需要修改cime/tools/statistical_ensemble_test/single_run.py中的"./case.build“为”./case.build --skip-provenance-check"

Once these runs are finished, you will need to run:

cime/tools/statistical_ensemble_test/addmetadata.sh

To add metadata to the files to be uploaded. The addmetadata.sh script requires that the nco tool ncks be available in your path, on most hpc systems this is available via: module load nco.

./addmetadata.sh --caseroot /vol-th/home/laiyq/cesm/scratch/case.cesm220.uf.000 --histfile /vol-th/home/laiyq/cesm/scratch/case.cesm220.uf.000/run/case.cesm220.uf.000.cam.h0.0001-01-01-00000.nc
./addmetadata.sh --caseroot /vol-th/home/laiyq/cesm/scratch/case.cesm220.uf.001 --histfile /vol-th/home/laiyq/cesm/scratch/case.cesm220.uf.001/run/case.cesm220.uf.001.cam.h0.0001-01-01-00000.nc
./addmetadata.sh --caseroot /vol-th/home/laiyq/cesm/scratch/case.cesm220.uf.002 --histfile /vol-th/home/laiyq/cesm/scratch/case.cesm220.uf.002/run/case.cesm220.uf.002.cam.h0.0001-01-01-00000.nc

之后把上述三个文件提交至https://www.cesm.ucar.edu/models/cesm2/verification，即可与UCAR提供的数据集进行比较，进行ensemble consistency验证。

验证结果：

附录

天河MPI编译环境

参考：https://doc.nscc-tj.cn/detail/?l=71&d=163

若使用后端调用Intel编译器的MPI实现，需确保使用mpiicc/mpiifort等命令；若使用后端调用GCC编译器的MPI实现，需确保使用mpicc/mpif90等命令。

注意事项：

由于系统计算节点采用Intel芯片，结合内部应用测试，建议使用Intel编译器及基于Intel编译器生成的Intel自带的MPI版本（IMPI），这样计算和通信效率一般会较高。使用如mvapich/openmpi等的效率比IMPI低约10%。
如果用户使用makefile或autoconf编译MPI并行程序:
- 若使用Intel编译器，应将makefile中的CC、CXX、F77、F90等变量设置成mpiicc、mpiicpc、mpiifort和mpiifort；
- 若使用GCC编译器，应将makefile中的CC、CXX、F77、F90等变量设置成mpicc、mpicxx、mpif77和mpif90等。
TH-1A系统（而非TH-HPC系统）的编译环境与上述文档中不同！具体配置应根据module av结果决定。
TH-1A系统只有Interl2017.4有impi。但使用时yhrun和yhbatch会有问题，故仍使用mpich。
/vol-th/software/intel2017.4/compilers_and_libraries_2017.4.196/linux/mpi/intel64/bin/mpiicc

在TH上运行MPI时使用yhrun命令，而非mpirun命令。

   -n number of tasks to run--ntasks-per-node=n number of tasks to invoke on each node-p partition requested--time time limit(minutes)

【教程】在天河上安装、移植并验证CESM2.2.0相关推荐

EOS中JAVA从Linux下载文件,教程 - 在Linux上安装EOS
你想学习如何在Linux上安装EOS? 在本教程中,我们将向您展示如何在运行Ubuntu Linux的计算机上安装EOS. EOS相关教程: 在此页面上,我们提供了与EOS.IO相关的教程列表的快速访 ...
linux上的壁纸软件下载,Ubuntu 17.10上安装开源壁纸工具Wallch 4.0
在Ubuntu 17.04上安装Wallch Wallch是一款免费的开源通用壁纸更换器,用户可以在设置的时间更改桌面背景图片.它会在桌面上更改并显示随机图片或维基百科的当天图片.它还具有一个实时地球 ...
如何在一台没有网的电脑上安装anaconda_简述验证Anaconda是否安装成功的两种方式...
/1 前言/ 小编给大家介绍验证Anaconda是否安装成功的两种方式和Anaconda环境变量配置过程.这里仍然是以win10系统为例进行演示. /2 验证是否安装成功方式一/ 1.依次在电脑上点击 ...
保姆级教程:在Linux上安装配置Resilio Sync
原文链接:https://zhaokaifeng.com/?p=6055 Resilio Sync 是一款基于 P2P 网络的去中心化文件同步工具,用途十分广泛.本文将从零开始演示如何在 Linux ...
python3.8.2安装教程-在服务器上安装python3.8.2环境的教程详解
在服务器上安装python3.8.2环境官网3.8.2下载地址下载安装下载好了之后上传到服务器上或者在服务器上通过命令下载 wget https://www.python.org/ftp/py ...
python3.8.2安装教程-在服务器上安装python3.8.2环境
在服务器上安装python3.8.2环境博客说明文章所涉及的资料来自互联网整理和个人总结,意在于个人学习和经验汇总,如有什么地方侵权,请联系本人删除,谢谢! 官网3.8.2下载地址下载安装下载 ...
服务器上安装安卓系统教程,云服务器上安装安卓系统
云服务器上安装安卓系统内容精选换一换有,该NTP服务器仅限于在华为云控制台上购买的弹性云服务器安装使用.如需在购买的弹性云服务器上安装NTP服务器,您可以选择使用华为云控制台自己提供的NTP服务 ...
在Linux和Windows上安装kafka(版本:2.12-2.8.0)
记录:379 场景:在CentOS 7.9操作系统上,安装kafka_2.12-2.8.0.在Windows上操作系统上,安装kafka_2.12-2.8.0. 版本: JDK 1.8 CentOS ...
在Debian 4.0 rc3上安装postfix+sasl验证及防垃圾SPAM机制
作者:何祖彬[RobinHe] Mail:zubin.he@gmail.com 始于2008年8月4日 PM 版本号:Debian-Postfix-SASL-SPAM-V1.0-20080804, 2 ...

【教程】在天河上安装、移植并验证CESM2.2.0