本地编译执行 Kubernetes e2e 测试

一、Kubernetes e2e 测试简介

Kubernetes e2e(end-to-end) 测试是单元测试和集成测试的补充，其主要目标是确保 Kubernetes 行为的一致可靠，并在单元和集成测试不足时，在用户真正使用软件之前捕获难以测试的 Bug。Kubernetes e2e 测试基于 Ginkgo 和 Gomega 实现。

二、在本地运行 Kubernetes e2e 测试

要在本地运行 e2e 测试，方法有很多，官方建议基于 kubetest 启动，但相对来说环境要求较高。本文介绍一种比较简单且通用的方式：先编译出用于执行 e2e 测试的二进制可执行程序，然后放到目标集群内执行。

2.1 编译 e2e 测试可执行程序

a. 准备一个 Linux 操作系统（例如 CentOS Linux 7）

b. 安装 Go 语言

注意: 版本要与 K8s 代码库要求保持一致（可在 https://github.com/kubernetes/kubernetes/blob/master/hack/lib/golang.sh 查找 minimum_go_version 以确认当前代码库需要的最低 Golang 版本，例如 K8s 1.19 需要 Golang 1.14.4 版本及以上）。

c. 克隆代码到本地

# mkdir $GOPATH/src/k8s.io
# cd $GOPATH/src/k8s.io
# git clone https://github.com/kubernetes/kubernetes.git

d. 编译 e2e.test 二进制

# cd $GOPATH/src/k8s.io/kubernetes
# make all WHAT=test/e2e/e2e.test
+++ [0708 09:42:01] Building go targets for linux/amd64:./vendor/k8s.io/code-generator/cmd/prerelease-lifecycle-gen
+++ [0708 09:42:06] Building go targets for linux/amd64:./vendor/k8s.io/code-generator/cmd/deepcopy-gen
+++ [0708 09:42:14] Building go targets for linux/amd64:./vendor/k8s.io/code-generator/cmd/defaulter-gen
+++ [0708 09:42:24] Building go targets for linux/amd64:./vendor/k8s.io/code-generator/cmd/conversion-gen
+++ [0708 09:42:38] Building go targets for linux/amd64:./vendor/k8s.io/kube-openapi/cmd/openapi-gen
+++ [0708 09:42:52] Building go targets for linux/amd64:./vendor/github.com/go-bindata/go-bindata/go-bindata
+++ [0708 09:42:53] Building go targets for linux/amd64:test/e2e/e2e.test

编译好的可执行程序位于：$GOPATH/src/k8s.io/kubernetes/_output/bin/e2e.test

e. 编译对应版本的 ginkgo (可选)

e2e.test 自身已经可以用于执行测试，但为了更好地利用 ginkgo 的高级能力（例如指定执行哪些用例，跳过某些用例等），建议使用 ginkgo 来启动 e2e.test

# make all WHAT=vendor/github.com/onsi/ginkgo/ginkgo
+++ [0709 11:18:10] Building go targets for linux/amd64:vendor/github.com/onsi/ginkgo/ginkgo

编译好的可执行程序位于：$GOPATH/src/k8s.io/kubernetes/_output/bin/ginkgo

2.2 执行 e2e 测试

准备一个 Kubernetes 集群，作为目标测试集群，然后将 e2e.test 和 ginkgo 拷贝到能够通过 kubectl 访问集群的节点上，执行测试命令（这里以执行 NetworkPolicy 测试为例）：

# ./ginkgo --focus="NetworkPolicy" ./e2e.test -- --disable-log-dump --provider="skeleton" --kubeconfig="/root/.kube/config" > >(tee e2e.log)
I0709 12:48:04.233600   20566 test_context.go:427] Tolerating taints "node-role.kubernetes.io/master" when considering if nodes are ready
I0709 12:48:04.233768   20566 e2e.go:129] Starting e2e run "b404a0ac-6071-481a-bf1c-4f721393a76a" on Ginkgo node 1
{"msg":"Test Suite starting","total":29,"completed":0,"skipped":0,"failed":0}
Running Suite: Kubernetes e2e suite
===================================
Random Seed: 1594270083 - Will randomize all specs
Will run 29 of 5208 specsJul  9 12:48:04.253: INFO: >>> kubeConfig: /root/.kube/config
Jul  9 12:48:04.256: INFO: Waiting up to 30m0s for all (but 0) nodes to be schedulable
Jul  9 12:48:04.268: INFO: Waiting up to 10m0s for all pods (need at least 0) in namespace 'kube-system' to be running and ready
Jul  9 12:48:04.295: INFO: 10 / 10 pods in namespace 'kube-system' are running and ready (0 seconds elapsed)
Jul  9 12:48:04.295: INFO: expected 2 pod replicas in namespace 'kube-system', 2 are Running and Ready.
Jul  9 12:48:04.295: INFO: Waiting up to 5m0s for all daemonsets in namespace 'kube-system' to start
Jul  9 12:48:04.308: INFO: 2 / 2 pods ready in namespace 'kube-system' in daemonset 'fabric-node' (0 seconds elapsed)
Jul  9 12:48:04.308: INFO: 2 / 2 pods ready in namespace 'kube-system' in daemonset 'kube-proxy' (0 seconds elapsed)
Jul  9 12:48:04.308: INFO: e2e test version: v0.0.0-master+$Format:%h$
Jul  9 12:48:04.311: INFO: kube-apiserver version: v1.18.2
Jul  9 12:48:04.311: INFO: >>> kubeConfig: /root/.kube/config
Jul  9 12:48:04.321: INFO: Cluster IP family: ipv4
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should support a 'default-deny-ingress' policy [Feature:NetworkPolicy]/root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/network_policy.go:83
[BeforeEach] [sig-network] NetworkPolicy [LinuxOnly]/root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:174
STEP: Creating a kubernetes client
Jul  9 12:48:04.322: INFO: >>> kubeConfig: /root/.kube/config
STEP: Building a namespace api object, basename network-policy
Jul  9 12:48:04.346: INFO: No PodSecurityPolicies found; assuming PodSecurityPolicy is disabled.
STEP: Waiting for a default service account to be provisioned in namespace
[BeforeEach] [sig-network] NetworkPolicy [LinuxOnly]/root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/network_policy.go:55
[BeforeEach] NetworkPolicy between server and client/root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/network_policy.go:61
STEP: Creating a simple server that serves on port 80 and 81.
STEP: Creating a server pod server in namespace network-policy-4845
Jul  9 12:48:04.353: INFO: Created pod server-6hfz8
STEP: Creating a service svc-server for pod server in namespace network-policy-4845
Jul  9 12:48:04.372: INFO: Created service svc-server
STEP: Waiting for pod ready
Jul  9 12:48:04.396: INFO: The status of Pod server-6hfz8 is Pending, waiting for it to be Running (with Ready = true)
Jul  9 12:48:06.398: INFO: The status of Pod server-6hfz8 is Pending, waiting for it to be Running (with Ready = true)
Jul  9 12:48:08.398: INFO: The status of Pod server-6hfz8 is Pending, waiting for it to be Running (with Ready = true)
Jul  9 12:48:10.398: INFO: The status of Pod server-6hfz8 is Running (Ready = false)
Jul  9 12:48:12.398: INFO: The status of Pod server-6hfz8 is Running (Ready = false)
Jul  9 12:48:14.398: INFO: The status of Pod server-6hfz8 is Running (Ready = false)
......

常用命令

1. 查看 e2e 测试将会使用的镜像

以便基于私有仓库提前准备好（或预分发到集群节点）

# ./e2e.test -list-images
gcr.io/k8s-staging-csi/nfs-provisioner:v2.2.2
gcr.io/kubernetes-e2e-test-images/sample-apiserver:1.17
docker.io/library/busybox:1.29
k8s.gcr.io/etcd:3.4.7
gcr.io/kubernetes-e2e-test-images/nonewprivs:1.0
gcr.io/kubernetes-e2e-test-images/regression-issue-74839-amd64:1.0
gcr.io/kubernetes-e2e-test-images/cuda-vector-add:1.0
gcr.io/kubernetes-e2e-test-images/echoserver:2.2
gcr.io/kubernetes-e2e-test-images/ipc-utils:1.0
gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0
docker.io/library/nginx:1.14-alpine
k8s.gcr.io/pause:3.2
docker.io/library/redis:5.0.5-alpine
gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0
gcr.io/authenticated-image-pulling/windows-nanoserver:v1
us.gcr.io/k8s-artifacts-prod/build-image/debian-iptables:v12.1.0
gcr.io/kubernetes-e2e-test-images/volume/iscsi:2.0
gcr.io/kubernetes-e2e-test-images/volume/rbd:1.0.1
docker.io/library/httpd:2.4.38-alpine
k8s.gcr.io/sd-dummy-exporter:v0.2.0
gcr.io/kubernetes-e2e-test-images/volume/gluster:1.0
gcr.io/kubernetes-e2e-test-images/apparmor-loader:1.0
gcr.io/kubernetes-e2e-test-images/cuda-vector-add:2.0
docker.io/library/httpd:2.4.39-alpine
k8s.gcr.io/prometheus-dummy-exporter:v0.1.0
gcr.io/k8s-authenticated-test/agnhost:2.6
docker.io/gluster/glusterdynamic-provisioner:v1.0
gcr.io/kubernetes-e2e-test-images/nonroot:1.0
k8s.gcr.io/prometheus-to-sd:v0.5.0
invalid.com/invalid/alpine:3.1
gcr.io/kubernetes-e2e-test-images/kitten:1.0
gcr.io/kubernetes-e2e-test-images/nautilus:1.0
docker.io/library/nginx:1.15-alpine
gcr.io/kubernetes-e2e-test-images/resource-consumer:1.5
us.gcr.io/k8s-artifacts-prod/e2e-test-images/agnhost:2.20
gcr.io/authenticated-image-pulling/alpine:3.7
gcr.io/kubernetes-e2e-test-images/metadata-concealment:1.2
docker.io/library/perl:5.26

2. 查看一致性测试用例列表

# ./e2e.test -list-conformance-tests
- testname: Pod Lifecycle, post start exec hookcodename: '[k8s.io] Container Lifecycle Hook when create a pod with lifecycle hookshould execute poststart exec hook properly [NodeConformance] [Conformance]'description: When a post start handler is specified in the container lifecycle usinga 'Exec' action, then the handler MUST be invoked after the start of the container.A server pod is created that will serve http requests, create a second pod witha container lifecycle specifying a post start that invokes the server pod usingExecAction to validate that the post start is executed.release: v1.9file: test/e2e/common/lifecycle_hook.go
.....