Oracle Cloud Infrastructure(OCI)上制作RHEL7U4定制镜像

参考

本实验完全参照了白皮书：White Paper:Journey to Oracle Cloud Infrastructure with Oracle VM VirtualBox
。这篇老的白皮书作用不大了，留存。

在这篇Journey to Oracle Cloud Infrastructure with Oracle VM VirtualBox中，提到了以上白皮书。同时也提到了VirtualBox 6.0及以后，可以直接与OCI交互，前提是你的网速够快。

这篇To OCI and back with Virtualbox非常好，其中提供了PV模式和Emulated模式的性能比较。以及在RHEL中如何将网卡名改为eth0，实际上所有的OS网卡名都需要改为eth0。还有就是VirtualBox和OCI可以相互导入导出Image。这篇文章还延伸出一篇如何压缩VM Image的Blog。

这篇文档Bring Your Own Image (BYOI)
提到了RHEL 7及以后只支持Paravirtualized模式。

这篇Bring Your Own Custom Image in Paravirtualized Mode for Improved Performance提到了Custom Image支持PV模式。

https://medium.com/oracledevs/building-oci-custom-images-with-packer-7e12b10077ad

在VirtualBox中安装RHEL

新建虚机，输入参数如下：

Name: RHEL7U4
Type: Linux
Version: Red Hat(64-bit)
Create a virtual hard disk now: Yes
Hard Disk File Type：VMDK
dynamically allocated: Yes
Hard Disk Size：8G(Default)
RAM: 4G

这里指定的是VMDK模式，因为Custom Image目前仅接受VMDK和QCOW模式

如果使用VirtualBox自带的vdi模式，好处是可以压缩，但只能通过VirtualBox直接传到OCI中。

挂载安装DVD rhel-server-7.4-x86_64-dvd.iso，开始安装:

Network & Hostname: 其中滑动网卡按钮到On位置，使其状态变为Connected. 此时注意到网卡设备名为enp0s3
Software Selection选择: Server With GUI
Time Zone: Asia/Shanghai
root password: Oracle4U1234
Create User: opc, 勾选make the user Administrator

8:36开始安装，8:54提示Reboot，安装耗时18分钟。

重启（安装光盘会自动卸载），接受License，进入欢迎界面，虚机可以使用。

使用putty连接VM

安装完成后，参考文章如何使用SSH登陆配置了NAT的VirtualBox虚拟机配置虚机的port forwarding，然后配置putty可以登录虚机。这样就方便拷贝粘贴了。

VM设置

此VM是可以访问互联网的：

[root@localhost ~]# ping www.oracle.com
PING e2581.dscx.akamaiedge.net (72.247.119.62) 56(84) bytes of data.
64 bytes from a72-247-119-62.deploy.static.akamaitechnologies.com (72.247.119.62): icmp_seq=1 ttl=51 time=48.7 ms
64 bytes from a72-247-119-62.deploy.static.akamaitechnologies.com (72.247.119.62): icmp_seq=2 ttl=51 time=49.3 ms
64 bytes from a72-247-119-62.deploy.static.akamaitechnologies.com (72.247.119.62): icmp_seq=3 ttl=51 time=47.2 ms
64 bytes from a72-247-119-62.deploy.static.akamaitechnologies.com (72.247.119.62): icmp_seq=4 ttl=51 time=43.8 ms

默认网卡是enp0s3，我们需要将其改为eth0。

# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000link/ether 08:00:27:7b:10:59 brd ff:ff:ff:ff:ff:ff
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT qlen 1000link/ether 52:54:00:33:a1:46 brd ff:ff:ff:ff:ff:ff
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN mode DEFAULT qlen 1000link/ether 52:54:00:33:a1:46 brd ff:ff:ff:ff:ff:ff

在白皮书中提到了/etc/sysconfig/grub和/etc/default/grub两个文件，实际他们是同一个文件，为避免混淆，我们均使用/etc/default/grub：

# ls -l /etc/sysconfig/grub
lrwxrwxrwx. 1 root root 17 May 27 20:43 /etc/sysconfig/grub -> /etc/default/grub

最初的/etc/sysconfig/grub文件如下：

# cat /etc/sysconfig/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet"
GRUB_DISABLE_RECOVERY="true"

将网卡设备名改为eth0

最初的网卡配置文件如下：

# cat /etc/sysconfig/network-scripts/ifcfg-enp0s3
TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="dhcp"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="enp0s3"
UUID="c7ba8837-fb9c-4efa-912f-a0c576ff4d71"
DEVICE="enp0s3"
ONBOOT="yes"

编辑文件/etc/sysconfig/grub,在包含GRUB_CMDLINE_LINUX的行尾部添加net.ifnames=0 biosdevname=0:

GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet net.ifnames=0 biosdevname=0"

这个修改会是网卡设备名从enp0s3变为eth0，这是OCI上Custom Image所需要的。

使其生效：

# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.10.0-693.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-693.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-8d225db9803f481285be1ae21f573fd9
Found initrd image: /boot/initramfs-0-rescue-8d225db9803f481285be1ae21f573fd9.img
done

立即备份并重命名文件为ifcfg-eth0:

# cd /etc/sysconfig/network-scripts/
# cp ifcfg-enp0s3 ifcfg-enp0s3.orig
# mv ifcfg-enp0s3 ifcfg-eth0

编辑ifcfg-eth0文件，改为如下：

# cat ifcfg-eth0
TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="dhcp"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="no"
NAME="eth0"
UUID="c7ba8837-fb9c-4efa-912f-a0c576ff4d71"
DEVICE="eth0"
ONBOOT="yes"
USERCTL="no"
PEERDNS="yes"

然后重启VM，你会发现网卡设备名变为eth0:

# ip link show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000link/ether 08:00:27:7b:10:59 brd ff:ff:ff:ff:ff:ff

以上备份包括修改文件的操作也可以重启后再做，但重启后你会发现网络不通了。你可以登录到控制台修改，而且还需要再重启一次，比较麻烦。

验证虚机仍可以上网：

# ping www.oracle.com
PING e2581.dscx.akamaiedge.net (184.87.133.227) 56(84) bytes of data.
64 bytes from a184-87-133-227.deploy.static.akamaitechnologies.com (184.87.133.227): icmp_seq=1 ttl=50 time=47.1 ms
64 bytes from a184-87-133-227.deploy.static.akamaitechnologies.com (184.87.133.227): icmp_seq=2 ttl=50 time=49.2 ms

启用Serial Console

编辑/etc/default/grub文件，修改前内容如下：

# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet net.ifnames=0 biosdevname=0"
GRUB_DISABLE_RECOVERY="true"

修改后如下，修改了一行，增加了两行(注意其中的注释需要去掉)：

# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
# modify below line by append console=...
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet net.ifnames=0 biosdevname=0 console=tty0 console=ttyS0,115200"
GRUB_DISABLE_RECOVERY="true"
# add below 2 lines
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200"

应用改变：

# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.10.0-693.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-693.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-8d225db9803f481285be1ae21f573fd9
Found initrd image: /boot/initramfs-0-rescue-8d225db9803f481285be1ae21f573fd9.img
done

重启后，确认生效：

# dmesg |grep console=ttyS0
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.10.0-693.el7.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet net.ifnames=0 biosdevname=0 console=tty0 console=ttyS0,115200
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-693.el7.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet net.ifnames=0 biosdevname=0 console=tty0 console=ttyS0,115200

启用paravirtualized device support

OCI中定制镜像支持PV和emulated模式，PV模式性能更好。
首先确认kernel version在3.4以上，RHEL7U4是3.10，满足要求：

# uname -r
3.10.0-693.el7.x86_64

添加qemu模块并确认：

# dracut --logfile /var/log/Dracut.log --force --add qemu# lsinitrd |grep virtio
-rw-r--r--   1 root     root         7632 Jul  6  2017 usr/lib/modules/3.10.0-693.el7.x86_64/kernel/drivers/block/virtio_blk.ko.xz
-rw-r--r--   1 root     root        12812 Jul  6  2017 usr/lib/modules/3.10.0-693.el7.x86_64/kernel/drivers/char/virtio_console.ko.xz
-rw-r--r--   1 root     root         7976 Jul  6  2017 usr/lib/modules/3.10.0-693.el7.x86_64/kernel/drivers/scsi/virtio_scsi.ko.xz
drwxr-xr-x   2 root     root            0 May 27 22:38 usr/lib/modules/3.10.0-693.el7.x86_64/kernel/drivers/virtio
-rw-r--r--   1 root     root         4344 Jul  6  2017 usr/lib/modules/3.10.0-693.el7.x86_64/kernel/drivers/virtio/virtio.ko.xz
-rw-r--r--   1 root     root         9488 Jul  6  2017 usr/lib/modules/3.10.0-693.el7.x86_64/kernel/drivers/virtio/virtio_pci.ko.xz
-rw-r--r--   1 root     root         8140 Jul  6  2017 usr/lib/modules/3.10.0-693.el7.x86_64/kernel/drivers/virtio/virtio_ring.ko.xz

然后就可以关机了，VM准备工作完成。

导入并制作Custom Image

在虚机目录下，可以看到vmdk文件：

$ ll *.vmdk
-rw-r--r-- 1 yyxiao 197121 3760193536 5月  28 10:40 RHEL7U4.vmdk

拷贝一份再压缩，压缩是为了节省点带宽：

$ cp RHEL7U4.vmdk RHEL7U4-image.vmdk
$ 压缩RHEL7U4-image.vmdk成zip文件
$ $ ll *zip
-rw-r--r-- 1 yyxiao 197121 1463448701 5月  28 10:59 RHEL7U4-image.zip

利用FileZilla(WinSCP也行)上传到OCI中某中转实例，运气不错，家里平均速度3.4MiB/s（最快达到近6M）。

登录OCI中转实例，然后解压：

$ ll -h RHEL7U4-image.zip
-rw-rw-r--. 1 opc opc 1.4G May 28 03:11 RHEL7U4-image.zip$ unzip RHEL7U4-image.zip
Archive:  RHEL7U4-image.zipinflating: RHEL7U4-image.vmdk$ ll -h RHEL7U4-image.vmdk
-rw-rw-r--. 1 opc opc 3.6G May 28  2020 RHEL7U4-image.vmdk

在此实例上通过OCI CLI上传到bucket。

$ oci os object put --bucket-name bucket001 --file RHEL7U4-image.vmdk
Upload ID: 10e31097-2ed0-0abb-4dba-9263bc9a16d6
Split file into 29 parts for upload.
Uploading object  [####################################]  100%
{"etag": "ec344743-56e4-4d73-b898-75fe13f071ae","last-modified": "Thu, 28 May 2020 03:15:51 GMT","opc-multipart-md5": "eOaIvnKXwkn2MEuHCsEugw==-29"
}

在OCI Console中进入Compute>Custom Images菜单，选择Import Image：

OPERATING SYSTEM: Linux
IMAGE TYPE: VMDK
LAUNCH MODE: PARAVIRTUALIZED MODE
IMPORT FROM AN OBJECT STORAGE BUCKET: Yes

日志如下，耗时5分钟：

Ingesting image into internal data store.    Thu, May 28, 2020, 03:25:20 UTC 2020-05-28T03:25:20.143Z
Converting image.   Thu, May 28, 2020, 03:23:12 UTC 2020-05-28T03:23:12.428Z
Downloading image from Object Storage.  Thu, May 28, 2020, 03:22:25 UTC 2020-05-28T03:22:25.746Z
Preparing environment for image conversion. Thu, May 28, 2020, 03:20:10 UTC

也可以用命令行导入：

$ oci -d compute image import from-object  -bn software --name RHEL7U4-disk001.vmdk --launch-mode PARAVIRTUALIZED --operating-system LINUX --display-name rhel7u4-customimage
{"data": {"agent-features": null,"base-image-id": null,"compartment-id": "ocid1.compartment.oc1..aaaaaaaahaic...","create-image-allowed": true,"defined-tags": {"owner": {"owner": "xxxxxxx"}},"display-name": "rhel7u4-customimage","freeform-tags": {},"id": "ocid1.image.oc1.iad.aaaaaaaa2...","launch-mode": "PARAVIRTUALIZED","launch-options": {"boot-volume-type": "PARAVIRTUALIZED","firmware": "BIOS","is-consistent-volume-naming-enabled": false,"is-pv-encryption-in-transit-enabled": false,"network-type": "PARAVIRTUALIZED","remote-data-volume-type": "PARAVIRTUALIZED"},"lifecycle-state": "IMPORTING","operating-system": "LINUX","operating-system-version": "Custom","size-in-mbs": null,"time-created": "2020-09-10T02:58:06.437000+00:00"},"etag": "6fc8fd47711de318f30bd96daaf38320d720cec934f69dd119be8b4cd78ca2e6","opc-work-request-id": "ocid1.coreservicesworkrequest.oc1.iad.abuwcljtrouaiq7enbjbfdp4ez6ahi6h755qgvojjjb6rc2ap47cmfw6aiyq"
}

这个过程是异步的, 可以查询进度：

$ oci work-requests work-request get --work-request-id "ocid1.coreservicesworkrequest.oc1.iad.abuwcljtrouaiq7enbjbfdp4ez6ahi6h755qgvojjjb6rc2ap47cmfw6aiyq"
{"data": {"compartment-id": "ocid1.compartment.oc1..aaaaaaaahaicvr4wo2lods4x3bcsphhs4b5im6t7znnm3es4c4vgddkobjoa","id": "ocid1.coreservicesworkrequest.oc1.iad.abuwcljtrouaiq7enbjbfdp4ez6ahi6h755qgvojjjb6rc2ap47cmfw6aiyq","operation-type": "CreateImage","percent-complete": 10.0,"resources": [{"action-type": "IN_PROGRESS","entity-type": "image","entity-uri": "/20160918/images/ocid1.image.oc1.iad.aaaaaaaa23g2c45bqij757yim2jubvtbibbxu5vhh2ys7mierl4d6yn6hkaa","identifier": "ocid1.image.oc1.iad.aaaaaaaa23g2c45bqij757yim2jubvtbibbxu5vhh2ys7mierl4d6yn6hkaa"}],"status": "IN_PROGRESS","time-accepted": "2020-09-10T02:58:06.015000+00:00","time-finished": null,"time-started": "2020-09-10T02:58:14.726000+00:00"}
}

然后选择这个Image，选择Create Instance即可。创建实例时无需指定SSH Public Key，因为指定也没有用，后续我们通过口令登录。

通过Putty使用口令登录实例成功，使用口令是因为之前没有设SSH：

# cat /etc/os-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.4 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.4"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.4 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.4:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.4
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.4"# ip link show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast state UP mode DEFAULT qlen 1000link/ether 02:00:17:07:dc:23 brd ff:ff:ff:ff:ff:ff$ cat /etc/sysconfig/network-scripts/ifcfg-eth0
TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="dhcp"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="no"
NAME="eth0"
UUID="c7ba8837-fb9c-4efa-912f-a0c576ff4d71"
DEVICE="eth0"
ONBOOT="yes"
USERCTL="no"
PEERDNS="yes"$ cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet net.ifnames=0 biosdevname=0 console=tty0 console=ttyS0,115200"
GRUB_DISABLE_RECOVERY="true"
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200"

验证Serial Console登录成功, ~.退出控制台：

[vagrant@ol7-vagrant ~]$ ssh -o ProxyCommand='ssh -W %h:%p -p 443 ocid1.instanceconsoleconnection.oc1.eu-frankfurt-1.antheljsaneeifycqy6jalrdzpli26y2ipz326fshz2cnyu2p552q7gay37a@instance-console.eu-frankfurt-1.oraclecloud.com' ocid1.instance.oc1.eu-frankfurt-1.antheljsaneeifyc3emyftkpokexvaxayn7bkrp3lccgt2dfufjvawcpd4ta
key_load_public: invalid format
key_load_public: invalid format
The authenticity of host 'ocid1.instance.oc1.eu-frankfurt-1.antheljsaneeifyc3emyftkpokexvaxayn7bkrp3lccgt2dfufjvawcpd4ta (<no hostip for proxy command>)' can't be established.
RSA key fingerprint is SHA256:xX8WtkVtp9AOvG+gOLQ8TQZLjBg8shVTsDpcBMJxItI.
RSA key fingerprint is MD5:72:71:e9:2c:c1:be:b7:32:02:76:96:0e:f9:d3:e8:5a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ocid1.instance.oc1.eu-frankfurt-1.antheljsaneeifyc3emyftkpokexvaxayn7bkrp3lccgt2dfufjvawcpd4ta' (RSA) to the list of known hosts.Red Hat Enterprise Linux Server 7.4 (Maipo)
Kernel 3.10.0-693.el7.x86_64 on an x86_64instance-20200528-1152 login: root
Password:
Last failed login: Thu May 28 00:47:52 EDT 2020 from 180.76.179.67 on ssh:notty
There were 20 failed login attempts since the last successful login.
Last login: Thu May 28 00:05:23 from 223.72.114.250

通过VirtualBox直接上传或下载Image

是可以的，但考验网速。此不赘述。

数据盘如何迁移呢

Windows Image和Linux都是不支持的，只能有一个启动盘。参见这里

注意

RHEL镜像上云，必须满足Redhat的许可要求。

后记

这个做好的image可以从E:\Lab\Oracle OVM\RHEL\7U4中的ova导入后得到，导入时不要选转换为vdi格式。
或者直接使用文件：RHEL7U4-disk001.zip