【Docker】clair镜像扫描的实现

clair镜像扫描的实现

一、前言

clair扫描的相关基础请先移步我的另外一篇文章镜像安全扫描工具clair与clairctl

这次我们采用clair api方式的扫描，基本思路是

打包镜像
解压tar包至tomcat的ROOT目录，得到每一个镜像的分层文件及描述文件
调用每个镜像的分层描述文件
使用clair api按照描述文件中提供的层序提交分层至clair中
clair依照漏洞-特征数据库进行比对
使用clair api并携带最后一层的id，查询漏洞数据
删除解压目录

二、启动容器

我们需要三个容器，分别是

Tomcat：给镜像的分层提供下载地址

docker run -d --name tomcat8 -p 6062:8080 -v /opt/deploy/clair/scanImage:/usr/local/tomcat/webapps/ROOT tomcat:jdk8-corretto

Postgres：存放漏洞-特征匹配数据

docker run -d -e POSTGRES_PASSWORD="password" -p 5432:5432 -v /opt/deploy/clair/postgres/data:/var/lib/postgresql/data --name postgres vmware/postgresql-photon:v1.5.0

Clair：扫描工具

新建目录/opt/deploy/clair/config,在该目录下新建文件config.yml

  database:type: pgsqloptions:source: postgresql://postgres:password@127.0.0.1:5432/postgres?sslmode=disablecachesize: 16384api:port: 6060healthport: 6061timeout: 900supdater:interval: 2hnotifier:attempts: 3renotifyinterval: 2h

docker run --net=host -d -p 6060-6061:6060-6061 -v /opt/deploy/clair/config:/config quay.io/coreos/clair:v2.0.1 -config=/config/config.yml

其中6060为扫描端口，6061为健康检查端口

注意，Postgres必须在clair之前启动，否则clair会连不上数据库，直接停止容器。

clair与postgres正常启动后，clair会自动去拉取漏洞数据插入数据库中，这一过程往往持续几个小时，且由于网络原因，大量漏洞数据无法拉取，此时需要另辟蹊径。

三、全部步骤

（1）检查工作

判断该镜像所在的仓库是否正常，否则无法拉取该镜像
判断tomcat、clair、postgres容器是否正常运行中

（2）调用脚本

    private String executeRemoteShell() {String result = null;SSH ssh = new SSH(主机ip, 用户名, 密码);try {boolean isConnect = ssh.connect();if (isConnect) {result = ssh.executeWithResult(脚本全路径，传入参数);}} catch (IOException e) {e.printStackTrace();}return result;}

#!/bin/bashbaseDir=/opt/deploy/clairif [ $? -eq 0 ];thendocker pull ${1} > /dev/null 2>&1imageName=`echo ${1} | sed 's!\/!_!g'`  > /dev/null 2>&1imageDir="${imageName}"  > /dev/null 2>&1mkdir ${baseDir}/scanImage/${imageDir}  > /dev/null 2>&1docker save -o ${baseDir}/scanImage/${imageDir}/${imageDir}.tar ${1}  > /dev/null 2>&1tar -xvf ${baseDir}/scanImage/${imageDir}/${imageDir}.tar -C ${baseDir}/scanImage/${imageDir}/ > /dev/null 2>&1cat ${baseDir}/scanImage/${imageDir}/manifest.jsonexit 0
elseecho 'scan image failed!'exit 6fi

（3）使用clair api提交镜像分层数据，postLayers为false时，此时已经提交过分层数据，可以利用最后一层的id直接查询漏洞。（postgres会将已经查询过的镜像漏洞存储，提交完分层数据后，可以使用最后一层的id获取漏洞数据）

//将镜像的分层文件提交至clair，进行分析并返回结果private ImageScanResult getScanResultFromClair(ImageScan imageScan, boolean postLayers) throws Exception {String imageName = imageScan.getImageName();String imageTag = imageScan.getImageTag();//1.获取manifest层序描述文件内容String result = RestTemplateUtils.get(getManifestPath(imageName, imageTag));JSONArray jsonArray = JSONArray.parseArray(result);if (jsonArray == null || jsonArray.size() == 0) {throw new Exception("无法获取或解析镜像层序的描述文件");}JSONObject jsonObject = (JSONObject) jsonArray.get(0);@SuppressWarnings("unchecked")List<String> layers = (List<String>) jsonObject.get("Layers");//提交的apiString clairApi = "http://" + systemConfig.getRelayServerIp() + ":" + systemConfig.getClairPort() + "/v1/layers";if (postLayers) {//上一层idString beforePath = "";//2.向clair逐层提交for (String layer : layers) {String name = layer.substring(0, layer.indexOf("/"));String path = getAbsoluteLayerPath(layer, imageScan.getImageName(), imageScan.getImageTag());String parentName = beforePath;String format = "Docker";beforePath = name;Map<String, Map<String, String>> postParameter = new HashMap<>();Map<String, String> layerParameter = new HashMap<>();layerParameter.put("Name", name);layerParameter.put("Path", path);layerParameter.put("ParentName", parentName);layerParameter.put("Format", format);postParameter.put("Layer", layerParameter);JSONObject layerJSONObject = JSONObject.parseObject(JSONObject.toJSONString(postParameter));RestTemplateUtils.post(clairApi, layerJSONObject, null);}}//3.所有层提交完毕后，使用最后一层进行查询漏洞int last = layers.size() - 1;String lastName = layers.get(last).substring(0, layers.get(last).indexOf("/"));String leak = RestTemplateUtils.get(clairApi + "/" + lastName + "?features&vulnerabilities");JSONObject leakJSONObject = JSONObject.parseObject(leak);JSONObject leakLayerJSONObject = leakJSONObject.getJSONObject("Layer");JSONArray featuresJSONArray = leakLayerJSONObject.getJSONArray("Features");List<Feature> featureList = new ArrayList<>();Feature feature;for (Object f : featuresJSONArray) {JSONObject featureJSONObject = (JSONObject) f;feature = new Feature();String name = featureJSONObject.getString("Name");String namespaceName = featureJSONObject.getString("NamespaceName");String versionFormat = featureJSONObject.getString("VersionFormat");String version = featureJSONObject.getString("Version");String addedBy = featureJSONObject.getString("AddedBy");List<Vulnerability> vulnerabilities = new ArrayList<>();JSONArray vulnerabilitiesJSONArray = featureJSONObject.getJSONArray("Vulnerabilities");if (vulnerabilitiesJSONArray == null || vulnerabilitiesJSONArray.size() == 0) {vulnerabilitiesJSONArray = new JSONArray();}Vulnerability vulnerability;for (Object v : vulnerabilitiesJSONArray) {JSONObject vulnerabilitiesJSONObject = (JSONObject) v;vulnerability = new Vulnerability();String vulnerabilityName = vulnerabilitiesJSONObject.getString("Name");String vulnerabilityNamespaceName = vulnerabilitiesJSONObject.getString("NamespaceName");String vulnerabilityDescription = vulnerabilitiesJSONObject.getString("Description");String vulnerabilityLink = vulnerabilitiesJSONObject.getString("Link");String vulnerabilitySeverity = vulnerabilitiesJSONObject.getString("Severity");String vulnerabilityFixedBy = vulnerabilitiesJSONObject.getString("FixedBy");vulnerability.setName(vulnerabilityName);vulnerability.setNamespaceName(vulnerabilityNamespaceName);vulnerability.setDescription(vulnerabilityDescription);vulnerability.setLink(vulnerabilityLink);vulnerability.setSeverity(vulnerabilitySeverity);vulnerability.setFixedBy(vulnerabilityFixedBy);vulnerabilities.add(vulnerability);}feature.setName(name);feature.setNamespaceName(namespaceName);feature.setVersionFormat(versionFormat);feature.setVersion(version);feature.setAddedBy(addedBy);feature.setVulnerabilities(vulnerabilities);featureList.add(feature);}//4.封装镜像扫描结果ImageScanResult imageScanResult = new ImageScanResult();int featureNum = featureList.size();long vulnerabilityNum = 0;long high = 0;long medium = 0;long low = 0;for (Feature temp : featureList) {List<Vulnerability> vulnerabilityList = temp.getVulnerabilities();vulnerabilityNum += vulnerabilityList.size();high += vulnerabilityList.stream().filter(v -> v.getSeverity().equals("High")).count();medium += vulnerabilityList.stream().filter(v -> v.getSeverity().equals("Medium")).count();low += vulnerabilityList.stream().filter(v -> v.getSeverity().equals("Low")).count();}imageScanResult.setImageName(imageScan.getImageName());imageScanResult.setImageTag(imageScan.getImageTag());imageScanResult.setFeatureNum(featureNum);imageScanResult.setVulnerabilityNum(vulnerabilityNum);imageScanResult.setHigh(high);imageScanResult.setMedium(medium);imageScanResult.setLow(low);imageScanResult.setFeatureList(featureList);return imageScanResult;}

（4）善后工作

清理扫描成功后的文件，这时我们打算清理解压出来的所有的tar包文件，否则解压目录将会异常庞大。博主在清理完一个24G的目录后，仅剩下几M的额外文件。

为什么我们不直接删除所有解压目录？这是因为我们需要留下镜像的分层描述文件（manifest.json），这个文件存放着该镜像的分层信息，包括分层id，分层tar包的地址等。后期我们可以依据此文件拿到最后一层的id，并调用clair api直接获取到该镜像的扫描结果。如果我们将扫描结果单独入库的话，那可以在扫描成功后删除整个解压目录。

    //扫描完成后删除镜像解压目录private void delImageDirAfterScan(ImageScan imageScan) {String imageIdentification = formatImageNameAndTag(imageScan.getImageName(), imageScan.getImageTag());String imageDir = systemConfig.getImageScanShellPath() + "scanImage/" + imageIdentification + "/";SSH ssh = new SSH(systemConfig.getRelayServerIp(), systemConfig.getRelayServerUser(), systemConfig.getRelayServerPwd());try {boolean isConnect = ssh.connect();if (isConnect) {ssh.execute("find " + imageDir + " -name '*.tar' -type f -print -exec rm -rf {} \\;");}} catch (IOException e) {e.printStackTrace();}}