《OpenShift / RHEL / DevSecOps 汇总目录》
说明：本文已经在OpenShift 4.9环境中验证

文章目录

File Integrity Operator 功能介绍
安装 File Integrity Operator
创建 FileIntegrity 对象
修改节点配置文件
确认文件完整性发生变化
接受对节点文件的修改
通过Prometheus监控
定制 File Integrity 配置
参考

File Integrity Operator 功能介绍

OpenShift 的 File Integrity Operator 可以在集群节点上持续地运行文件完整性检查。它部署了一个 DaemonSet，在每个节点上初始化并运行有特权的AIDE（Advanced Intrusion Detection Environment - 高级入侵检测环境）容器，提供自 DaemonSet 初始运行以后被修改的文件记录。

File Integrity Operator 根据从配置文件中找到的正则表达式规则创建一个数据库。一旦数据库被初始化，它就可以用来验证文件的完整性。它有几种消息摘要算法用于检查文件的完整性，另外文件属性也可以被检查出不一致的地方。

安装 File Integrity Operator

在 OpenShift 上使用缺省配置安装 File Integrity Operator ，缺省会将资源安装在 openshift-file-integrity 项目下。

安装后可以看到 File Integrity Operator 实例：

创建 FileIntegrity 对象

在 OpenShift 中根据以下内容创建FileIntegrity对象。其中 spec.config.gracePeriod 是两次 AIDE 完整性检查之间暂停的秒数。由于在节点中频繁进行 AIDE 检查需要大量资源，因此可以指定较长的时间间隔（默认为 900 秒）。

apiVersion: fileintegrity.openshift.io/v1alpha1
kind: FileIntegrity
metadata:name: example-fileintegritynamespace: openshift-file-integrity
spec:config:gracePeriod: 900debug: falsetolerations:- effect: NoSchedulekey: node-role.kubernetes.io/masteroperator: Exists

如果没有指定，FileIntegrity 会在所有节点运行一个 Daemonset Pod 以进行完整性检查。执行命令可以看到 Daemonset 运行在集群的 5 个节点中。

$ oc get daemonset -n openshift-file-integrity
NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
aide-example-fileintegrity   5         5         5       5            5           <none>          7h41m$ oc get pod -n openshift-file-integrity -o wide -l app=aide-example-fileintegrity
NAME                               READY   STATUS    RESTARTS   AGE     IP            NODE                                         NOMINATED NODE   READINESS GATES
aide-example-fileintegrity-2vpbq   1/1     Running   1          7h45m   10.128.2.15   ip-10-0-241-123.us-east-2.compute.internal   <none>           <none>
aide-example-fileintegrity-czf4w   1/1     Running   1          7h45m   10.131.0.11   ip-10-0-134-47.us-east-2.compute.internal    <none>           <none>
aide-example-fileintegrity-vx5q7   1/1     Running   1          7h45m   10.128.0.6    ip-10-0-173-213.us-east-2.compute.internal   <none>           <none>
aide-example-fileintegrity-wd4pm   1/1     Running   1          7h45m   10.129.0.10   ip-10-0-173-152.us-east-2.compute.internal   <none>           <none>
aide-example-fileintegrity-wx42d   1/1     Running   1          7h45m   10.130.0.8    ip-10-0-135-39.us-east-2.compute.internal    <none>           <none>

查看内容为 “FileIntegrityStatus” 的事件，可以看到当前已经是 “Active”。

$ oc get event -n openshift-file-integrity --field-selector reason=FileIntegrityStatus
LAST SEEN   TYPE     REASON                OBJECT                                MESSAGE
2m28s       Normal   FileIntegrityStatus   fileintegrity/example-fileintegrity   Pending
2m28s       Normal   FileIntegrityStatus   fileintegrity/example-fileintegrity   Initializing
2m13s       Normal   FileIntegrityStatus   fileintegrity/example-fileintegrity   Active

在example-fileintegrity完成首次扫描后，可以执行以下命令确认所有节点的 fileintegritynodestatus 的状态都是 “Succeeded”，这说明被监控节点的目标文件没有变化。

$ oc get fileintegritynodestatus -n openshift-file-integrity
NAME                                                               NODE                                         STATUS
example-fileintegrity-ip-10-0-134-47.us-east-2.compute.internal    ip-10-0-134-47.us-east-2.compute.internal    Succeeded
example-fileintegrity-ip-10-0-135-39.us-east-2.compute.internal    ip-10-0-135-39.us-east-2.compute.internal    Succeeded
example-fileintegrity-ip-10-0-173-152.us-east-2.compute.internal   ip-10-0-173-152.us-east-2.compute.internal   Succeeded
example-fileintegrity-ip-10-0-173-213.us-east-2.compute.internal   ip-10-0-173-213.us-east-2.compute.internal   Succeeded
example-fileintegrity-ip-10-0-241-123.us-east-2.compute.internal   ip-10-0-241-123.us-east-2.compute.internal   Succeeded

另外查看事件也能看到此时所有目标都 “no changes to node”。

$ oc get events -n openshift-file-integrity --field-selector reason=NodeIntegrityStatus
LAST SEEN   TYPE     REASON                OBJECT                                MESSAGE
7m4s        Normal   NodeIntegrityStatus   fileintegrity/example-fileintegrity   no changes to node ip-10-0-134-47.us-east-2.compute.internal
7m3s        Normal   NodeIntegrityStatus   fileintegrity/example-fileintegrity   no changes to node ip-10-0-241-123.us-east-2.compute.internal
7m2s        Normal   NodeIntegrityStatus   fileintegrity/example-fileintegrity   no changes to node ip-10-0-173-152.us-east-2.compute.internal
7m          Normal   NodeIntegrityStatus   fileintegrity/example-fileintegrity   no changes to node ip-10-0-135-39.us-east-2.compute.internal
6m59s       Normal   NodeIntegrityStatus   fileintegrity/example-fileintegrity   no changes to node ip-10-0-173-213.us-east-2.compute.internal

修改节点配置文件

进入到一个Node的RHCOS内部。

$ NODE=ip-10-0-135-39.us-east-2.compute.internal
$ NODESTATUS=example-fileintegrity-${NODE}
$ oc debug node/${NODE}

执行命令，更新 “** /host/etc/resolv.conf**” 文件。

sh-4.2# echo "# integrity test" >> /host/etc/resolv.conf
sh-4.2# exit

确认文件完整性发生变化

执行以下命令，或在 OpenShift 控制台上查看 example-fileintegrity 的事件。确认提示有一个节点的完整性检查失败，其中完整性变化是：“a:0,c:1,r:0”，即：0个新增（a）；1个变化（c）；0个删除（r）。

$ oc get events -n openshift-file-integrity --field-selector reason=NodeIntegrityStatus,type=Warning
LAST SEEN   TYPE      REASON                OBJECT                                MESSAGE
17m         Warning   NodeIntegrityStatus   fileintegrity/example-fileintegrity   node ip-10-0-135-39.us-east-2.compute.internal has changed! a:0,c:1,r:0 log:openshift-file-integrity/aide-example-fileintegrity-ip-10-0-135-39.us-east-2.compute.internal-failed

2. 还可执行以下命令，确认其中一个的 STATUS 已经变为 “Failed”。

$ oc get fileintegritynodestatus -n openshift-file-integrity
NAME                                                               NODE                                         STATUS
example-fileintegrity-ip-10-0-134-47.us-east-2.compute.internal    ip-10-0-134-47.us-east-2.compute.internal    Succeeded
example-fileintegrity-ip-10-0-135-39.us-east-2.compute.internal    ip-10-0-135-39.us-east-2.compute.internal    Failed
example-fileintegrity-ip-10-0-173-152.us-east-2.compute.internal   ip-10-0-173-152.us-east-2.compute.internal   Succeeded
example-fileintegrity-ip-10-0-173-213.us-east-2.compute.internal   ip-10-0-173-213.us-east-2.compute.internal   Succeeded
example-fileintegrity-ip-10-0-241-123.us-east-2.compute.internal   ip-10-0-241-123.us-east-2.compute.internal   Succeeded

执行以下命令，可以在该节点的 fileintegritynodestatus 中查到记录的完整性检查结果。可以看到完整性检查开始是 “Secceeded”，然后变成了“Failed”，这是由 1 个 filesChanged 引起的。File Integrity Operator 将完整性检查详细结果存放在 “openshift-file-integrity” 项目中名为 “aide-example-fileintegrity-ip-10-0-135-39.us-east-2.compute.internal-failed” 的 ConfigMap 对象中了。

$ oc get fileintegritynodestatus/${NODESTATUS} -n openshift-file-integrity -ojsonpath='{.results}' | jq -r
[{"condition": "Succeeded","lastProbeTime": "2022-03-05T03:19:42Z"},{"condition": "Failed","filesChanged": 1,"lastProbeTime": "2022-03-05T03:35:27Z","resultConfigMapName": "aide-example-fileintegrity-ip-10-0-135-39.us-east-2.compute.internal-failed","resultConfigMapNamespace": "openshift-file-integrity"}
]

从 ConfigMap 对象中查看完整性检查结果失败说明，其中提示 “/hostroot/etc/resolv.conf” 发生了变化。

$ oc describe cm aide-${NODESTATUS}-failedName:         aide-example-fileintegrity-ip-10-0-135-39.us-east-2.compute.internal-failed
Namespace:    openshift-file-integrity
Labels:       file-integrity.openshift.io/node=ip-10-0-135-39.us-east-2.compute.internalfile-integrity.openshift.io/owner=example-fileintegrityfile-integrity.openshift.io/result-log=
Annotations:  file-integrity.openshift.io/files-added: 0file-integrity.openshift.io/files-changed: 1file-integrity.openshift.io/files-removed: 0Data
====
integritylog:
----
Start timestamp: 2022-03-05 03:50:27 +0000 (AIDE 0.16)
AIDE found differences between database and filesystem!!Summary:Total number of entries:  35211Added entries:              0Removed entries:            0Changed entries:            1---------------------------------------------------
Changed entries:
---------------------------------------------------f   ...    .C... : /hostroot/etc/resolv.conf---------------------------------------------------
Detailed information about changes:
---------------------------------------------------File: /hostroot/etc/resolv.confSHA512   : Xl2pzxjmRPtW8bl6Kj49SkKOSBVJgsCI | 4M9rtSPrVj1Q34k0r/j3sZlreZT6uieCFw3VVMAbuGgHwINR5Fi5CYGuLhNmmDNh | 850ZVM9KcJ5DYB5Gf0IGbuojvIF1exdIdC7kQIKcAjFwB3Ib+UaOzw==         | vXZwgl8BvjmcezDPzmTGPA==---------------------------------------------------
The attributes of the (uncompressed) database(s):
---------------------------------------------------/hostroot/etc/kubernetes/aide.db.gzMD5      : OeZ96l1VZFyigc/hL7vv5w==SHA1     : a8EFf5Ylgi1h86rq549XFFea6CM=RMD160   : SV7z0d1kjNxzSRn0Pu+6Hps02Ew=TIGER    : HHLY/HeLmpTjF3/bNyTjcZ3+/AlDfYrKSHA256   : DsDlSlIxzysKry78/mYualTEt6BqtjJJnIoRhHwy1jo=SHA512   : iIIJZZKx36suzJG2BsdG78jAonAm1EpIZKmz40Gw5h6eWIu88+VyFlStpkOLCeFn3j8uZda/6Qgste6zBQV/Sg==End timestamp: 2022-03-05 03:51:10 +0000 (run time: 0m 43s)BinaryData
====Events:  <none>

接受对节点文件的修改

如果这个修改是必要的，则可以接受这一修改。

执行命令重置完整性对比数据库。

$ oc annotate fileintegrity/example-fileintegrity file-integrity.openshift.io/re-init=

然后查看由 FileIntegrityStatus 引发的事件。

$ oc get events -n openshift-file-integrity --field-selector reason=FileIntegrityStatus
LAST SEEN   TYPE     REASON                OBJECT                                MESSAGE
54s         Normal   FileIntegrityStatus   fileintegrity/example-fileintegrity   Initializing
47s         Normal   FileIntegrityStatus   fileintegrity/example-fileintegrity   Active

可以再次进入前面操作的 RHCOS，确认已经在 “/host/etc/kubernetes/” 目录中对 “aide.db.gz”和“aide.log”进行了备份。

sh-4.4# chroot /host
sh-4.4# ls -lR /etc/kubernetes/aide.*
-rw-------. 1 root root 1953565 Mar  5 10:47 /etc/kubernetes/aide.db.gz
-rw-------. 1 root root 1953538 Mar  5 10:46 /etc/kubernetes/aide.db.gz.backup-20220305T10_46_29
-rw-------. 1 root root 1953565 Mar  5 10:47 /etc/kubernetes/aide.db.gz.new
-rw-------. 1 root root    1012 Mar  5 10:31 /etc/kubernetes/aide.log
-rw-------. 1 root root    1012 Mar  5 10:46 /etc/kubernetes/aide.log.backup-20220305T10_46_29
-rw-------. 1 root root     778 Mar  5 10:47 /etc/kubernetes/aide.log.new

过一段时间后每个节点的 DaemonSet 会完成一次新的完整性对比。再此过程会中会基于修改后的文件重新建完整性对比数据库，这样对比状态又恢复成 “Succeeded” 了。

$ oc get fileintegritynodestatus -n openshift-file-integrity
NAME                                                               NODE                                         STATUS
example-fileintegrity-ip-10-0-134-47.us-east-2.compute.internal    ip-10-0-134-47.us-east-2.compute.internal    Succeeded
example-fileintegrity-ip-10-0-135-39.us-east-2.compute.internal    ip-10-0-135-39.us-east-2.compute.internal    Succeeded
example-fileintegrity-ip-10-0-173-152.us-east-2.compute.internal   ip-10-0-173-152.us-east-2.compute.internal   Succeeded
example-fileintegrity-ip-10-0-173-213.us-east-2.compute.internal   ip-10-0-173-213.us-east-2.compute.internal   Succeeded
example-fileintegrity-ip-10-0-241-123.us-east-2.compute.internal   ip-10-0-241-123.us-east-2.compute.internal   Succeeded

通过Prometheus监控

File Integrity 提供了 Prometheus 接口，运行以下命令可以从 Prometheus 接口访问到当前集群文件完整性的状态。其中：
“file_integrity_operator_node_failed gauge” 是以往完整性失败的情况。
“file_integrity_operator_node_status_total counter” 是当前集群节点完整性的统计。

$ oc run --rm -i --restart=Never --image=registry.fedoraproject.org/fedora-minimal:latest -n openshift-file-integrity metrics-test -- bash -c 'curl -ks -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://metrics.openshift-file-integrity.svc:8585/metrics-fio'If you don't see a command prompt, try pressing enter.
# HELP file_integrity_operator_node_failed A gauge that is set to 1 when a node has unresolved integrity failures, and 0 when it is healthy
# TYPE file_integrity_operator_node_failed gauge
file_integrity_operator_node_failed{node="ip-10-0-134-47.us-east-2.compute.internal"} 1
file_integrity_operator_node_failed{node="ip-10-0-135-39.us-east-2.compute.internal"} 0
# HELP file_integrity_operator_node_status_total The total number of FileIntegrityNodeStatus transitions, per condition and node
# TYPE file_integrity_operator_node_status_total counter
file_integrity_operator_node_status_total{condition="Failed",node="ip-10-0-134-47.us-east-2.compute.internal"} 1
file_integrity_operator_node_status_total{condition="Succeeded",node="ip-10-0-135-39.us-east-2.compute.internal"} 1
。。。

定制 File Integrity 配置

当创建一个 FileIntegrity，它的详细配置保存在对应名称的 ConfigMap 对象中。可以执行以下命令查看该 ConfigMap 的内容，其中使用正则表达式列出了所有受 FileIntegrity 保护的目标，默认配置包含以下目录中的文件 “/root”、“/boot”、“/usr”、“/etc”，而不包含的目录文件有 “/var”、“/opt”、“/etc/”。

$ oc describe cm/example-fileintegrity -n openshift-file-integrity
Name:         example-fileintegrity
Namespace:    openshift-file-integrity
Labels:       file-integrity.openshift.io/aide-conf=file-integrity.openshift.io/owner=example-fileintegrity
Annotations:  <none>Data
====
aide.conf:
----
@@define DBDIR /hostroot/etc/kubernetes
@@define LOGDIR /hostroot/etc/kubernetes
database=file:@@{DBDIR}/aide.db.gz
database_out=file:@@{DBDIR}/aide.db.gz.new
gzip_dbout=yes
verbose=5
report_url=file:@@{LOGDIR}/aide.log.new
report_url=stdout
PERMS = p+u+g+acl+selinux+xattrs
CONTENT_EX = sha512+ftype+p+u+g+n+acl+selinux+xattrs/hostroot/boot/        CONTENT_EX
/hostroot/root/\..* PERMS
/hostroot/root/   CONTENT_EX
!/hostroot/root/\.kube
!/hostroot/usr/src/
!/hostroot/usr/tmp//hostroot/usr/    CONTENT_EX# OpenShift specific excludes
!/hostroot/opt/
!/hostroot/var
!/hostroot/etc/NetworkManager/system-connections/
!/hostroot/etc/mtab$
!/hostroot/etc/.*~
!/hostroot/etc/kubernetes/static-pod-resources
!/hostroot/etc/kubernetes/aide.*
!/hostroot/etc/kubernetes/manifests
!/hostroot/etc/docker/certs.d
!/hostroot/etc/selinux/targeted
!/hostroot/etc/openvswitch/conf.db
!/hostroot/etc/kubernetes/cni/net.d/*
!/hostroot/etc/machine-config-daemon/currentconfig$
!/hostroot/etc/pki/ca-trust/extracted/java/cacerts$
!/hostroot/etc/cvo/updatepayloads# Catch everything else in /etc
/hostroot/etc/    CONTENT_EXBinaryData
====Events:  <none>

可以将该 ConfigMap 内容导出成文件。

$ oc extract cm/example-fileintegrity -n openshift-file-integrity --keys=aide.conf

修改后再根据修改后的文件生成一个新的 ConfigMap 对象。

$ oc create cm master-aide-conf -n openshift-file-integrity --from-file=aide.conf

最后可以根据以下YAML只在 “master” 节点上根据建的 “master-aide-conf” 配置创建 FileIntegrity 对象。

node-role.kubernetes.io/master: “” 在所有 master 节点上调度 AIDE；name 和 namespace 属性指向配置映射

apiVersion: fileintegrity.openshift.io/v1alpha1
kind: FileIntegrity
metadata:name: master-fileintegritynamespace: openshift-file-integrity
spec:nodeSelector:node-role.kubernetes.io/master: ""config:name: master-aide-confnamespace: openshift-file-integrity

参考

https://access.redhat.com/documentation/zh-cn/openshift_container_platform/4.9/html-single/security_and_compliance/index
https://github.com/openshift/file-integrity-operator
https://aide.github.io/

OpenShift 4 - 利用 File Integrity Operator 实现对集群节点进行入侵检测相关推荐

OpenShift 4 - 用KubeletConfig和ContainerRuntimeConfig分别修改集群节点的Kubelet和cri-o的配置
<OpenShift 4.x HOL教程汇总> 说明:本文已经在OpenShift 4.6环境中验证文章目录 Kubelet.KubeletConfig和KubeletConfigCon ...
OpenShift 4 - 集群节点日志和API审计日志策略
<OpenShift / RHEL / DevSecOps 汇总目录> 说明:本文已经在OpenShift 4.8 环境中验证文章目录集群节点日志集群节点日志类型收集集群节点日志 ...
利用memcached实现CAS单点登录集群部署
前言:利用memcached实现CAS单点登录集群部署负载均衡: 将接口请求的有状态性变成无状态性.是我们在实现负载均衡时必要要解决的问题.以应用接口的session状态为例,一般解决方法都是将se ...
OpenShift 4 之让Route只运行在集群中Infra节点
很多OpenShift的用户都非常喜欢其自带的Route功能,Route为外部用户提供了访问Pod的负载均衡功能,它要比Kubernetes缺省提供的Ingress功能强大很多.有关介绍可参见< ...
ZooKeeper1 利用虚拟机搭建自己的ZooKeeper集群
前言: 前段时间自己参考网上的文章,梳理了一下基于分布式环境部署的业务系统在解决数据一致性问题上的方案,其中有一个方案是使用ZooKeeper,加之在大数据处理中,ZooKeeper确实起 ...
OpenShift 4 - 在集群节点用crictl对Pod/Image/Container进行操作
<OpenShift 4.x HOL教程汇总> 说明:本文已经在OpenShift 4.6环境中验证文章目录查看crictl命令的配置文件用crictl命令操作Pod/Image/C ...
OpenShift 4 - 设置集群节点和Pod容器的时间和时区
<OpenShift 4.x HOL教程汇总> 说明:本文已经在OpenShift 4.6环境中验证文章目录 OpenShift的时间和时区节点和容器的时间节点和容器的时区节点时间 ...
利用FRP跨局域网操纵虚拟机集群
利用FRP跨局域网操纵虚拟机集群我们在日常学习和生活中常常遇到这样的痛点,自己有两台笔记本A和B,自己的一台笔记本A安装了三台Linux虚拟机集群在学校机房里面(三台虚拟机都没有公网IP,并在学校局 ...
operator部署redis集群
operator部署redis集群 1.现在operator的redis相关包 git clone https://github.com/ucloud/redis-cluster-operator.g ...

OpenShift 4 - 利用 File Integrity Operator 实现对集群节点进行入侵检测