女主宣言

在对象存储迭代研发、测试过程中，为了方便、准确验证 S3 接口协议兼容性，本文作者对Ceph官方采用的兼容性测试工具s3-tests进行了调研，并对其配置以及使用做出了详细介绍，相信对于s3的使用者，会起到实质性的帮助，下来就跟随作者一探究竟吧。

PS：丰富的一线技术、多元化的表现形式，尽在“360云计算”，点关注哦！

1

简介

在对象存储迭代研发、测试过程中，为了方便、准确验证 S3 接口协议兼容性，我们内部配置了 Ceph 官方采用的兼容性测试工具 s3-tests ，该工具基于 Python Nose 测试框架和官方的 Boto 库（同时支持 boto 2 / boto 3），并通过结合 HTML 扩展可将测试结果输出到Web界面，通过页面查询相关测试统计、分析结果。

s3-tests

S3 compatibility tests - This is a set of unofficial Amazon AWS S3 compatibility tests, that can be useful to people implementing software that exposes an S3-like API. The tests use the Boto2 and Boto3 libraries.

仓库地址：https://github.com/ceph/s3-tests

Nose 框架

Nose 是一种流行的 Python 单元测试框架扩展，它可以方便地自动运行一批测试和插件，比如度量代码覆盖率。Nose 能支持基于函数的测试，也能支持与 unittest 相类似的基于测试类的测试方式。

nose-html-reporting 扩展

Nose plugin that generates a nice html test report with ability of using template based on jinja2 templates from any folder.

源自: https://github.com/ionelmc/nose-htmloutput ，增加了对 jinja2 模板支持。

2

配置 s3-tests 实践

1. 下载源码

   # git clone https://github.com/ceph/s3-tests.git
   

2. 安装、部署

   # sudo apt-get install python-virtualenv
   # cd s3-tests
   

   启动测试运行环境，安装依赖软件：

   # ./bootstrap
   

3. 创建配置文件

   首先，需要通过 S3 管理接口提前创建好配置文件中的用户、access_key，secret_key 等必要配置信息。

   创建 config.yaml 增加如下内容：

   [DEFAULT]
   ## this p is just used as default for all the "s3 *"
   ## ps, you can place these variables also directly there
   ## replace with e.g. "localhost" to run against local software
   host = test-shyc2.s3.360.local  # 也可用本机 S3 Gateway IP 地址
   ## uncomment the port to use something other than 80
   port = 80
   ## say "no" to disable TLS
   is_secure = no
   [fixtures]
   ## all the buckets created will start with this prefix;
   ## {random} will be filled with random characters to pad
   ## the prefix to 30 characters long, and avoid collisions
   bucket prefix = s3test-{random}-
   [s3 main]
   ## the tests assume two accounts are defined, "main" and "alt".
   ## user_id is a 64-character hexstring
   user_id = infra-s3
   ## display name typically looks more like a unix login, "jdoe" etc
   display_name = infra-s3
   ## replace these with your access keys
   access_key = infra-ak
   secret_key = infra-sk
   [s3 alt]
   ## another user account, used for ACL-related tests
   user_id = infra-alt
   display_name = infra-alt
   ## the "alt" user needs to have email set, too
   email = infra-alt@qihoo.net
   access_key = infra-alt-ak
   secret_key = infra-alt-sk
   

4. 查看测试项

   ~/s3-tests# S3TEST_CONF=./config.yaml ./virtualenv/bin/nosetests -v --collect-only
   s3tests.functional.test_headers.test_object_create_bad_md5_invalid_short ... ok
   s3tests.functional.test_headers.test_object_create_bad_md5_bad ... ok
   s3tests.functional.test_headers.test_object_create_bad_md5_empty ... ok
   s3tests.functional.test_headers.test_object_create_bad_md5_unreadable ... ok
   s3tests.functional.test_headers.test_object_create_bad_md5_none ... ok
   s3tests.functional.test_headers.test_object_create_bad_expect_mismatch ... ok
   s3tests.functional.test_headers.test_object_create_bad_expect_empty ... ok
   s3tests.functional.test_headers.test_object_create_bad_expect_none ... ok
   s3tests.functional.test_headers.test_object_create_bad_expect_unreadable ... ok
   s3tests.functional.test_headers.test_object_create_bad_contentlength_empty ... ok
   s3tests.functional.test_headers.test_object_create_bad_contentlength_negative ... ok
   s3tests.functional.test_headers.test_object_create_bad_contentlength_none ... ok
   s3tests.functional.test_headers.test_object_create_bad_contentlength_unreadable ... ok
   s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_above ... ok
   s3tests.functional.test_headers.test_object_create_bad_contenttype_invalid ... ok
   s3tests.functional.test_headers.test_object_create_bad_contenttype_empty ... ok
   s3tests.functional.test_headers.test_object_create_bad_contenttype_none ... ok
   s3tests.functional.test_headers.test_object_create_bad_contenttype_unreadable ... ok
   s3tests.functional.test_headers.test_object_create_bad_authorization_unreadable ... ok
   s3tests.functional.test_headers.test_object_create_bad_authorization_empty ... ok
   

3

Nose Html 扩展

配置静态资源服务器

搭建一个 HTTP 资源服务器用于展示 nose 输出。

在线安装 nginx / httpd 即可，下文中直接将输出文件保存到默认路径 /var/www/html/ 下。

安装 nose-html-reporting

官方地址 : https://pypi.org/project/nose-html-reporting/#description。

安装 nose-html-reporting。

# ./virtualenv/bin/pip install nose-html-reporting

验证，有如下输出，代表扩展已安装，支持输出 html 文件。

~/s3-tests# ./virtualenv/bin/nosetests -h | grep html--with-html           Enable plugin HtmlReport:  Output test results aspretty html.  [NOSE_WITH_HTML]--html-report=FILE    Path to html file to store the report in. Default isnosetests.html in the working directory--html-report-template=FILEPath to html template file in with jinja2format.Default is report.html in the lib--cover-html          Produce HTML coverage information--cover-html-dir=DIR  Produce HTML coverage information in dir
~/s3-tests#

执行单个用例

单独运行 test_bucket_list_empty 用例，指定 html 模板文件为 report2.jinja2。

执行如下命令：

~/s3-tests# S3TEST_CONF=./config.yaml ./virtualenv/bin/nosetests s3tests.functional.test_s3:test_bucket_list_empty -v --with-html --html-report=/var/www/html/index.html --html-report-template=virtualenv/lib/python2.7/site-packages/nose_html_reporting/templates/report2.jinja2。

示例：

~/s3-tests# S3TEST_CONF=./config.yaml ./virtualenv/bin/nosetests s3tests.functional.test_s3:test_bucket_list_empty  -v --with-html --html-report=/var/www/html/index.html --html-report-template=virtualenv/lib/python2.7/site-packages/nose_html_reporting/templates/report2.jinja2
s3tests.functional.test_s3.test_bucket_list_empty ... ok
----------------------------------------------------------------------
HTML: /var/www/html/index.html
----------------------------------------------------------------------
Ran 1 test in 0.322s
OK
~/s3-tests#

打开 web 端，查看测试结果：

部分测试

运行 atomic read/write 单元测试用例:

#!/bin/bash
cases="\s3tests.functional.test_s3:test_atomic_read_1mb \s3tests.functional.test_s3:test_atomic_read_4mb \s3tests.functional.test_s3:test_atomic_read_8mb \s3tests.functional.test_s3:test_atomic_write_1mb \s3tests.functional.test_s3:test_atomic_write_4mb \s3tests.functional.test_s3:test_atomic_write_8mb \s3tests.functional.test_s3:test_atomic_dual_write_1mb \s3tests.functional.test_s3:test_atomic_dual_write_4mb \s3tests.functional.test_s3:test_atomic_dual_write_8mb \s3tests.functional.test_s3:test_atomic_conditional_write_1mb"
S3_USE_SIGV4=1 S3TEST_CONF=config.yaml virtualenv/bin/nosetests $cases -a \
auth_aws4 -x --debug-log=./tests.log --cover-html --process-timeout=60 \
--processes=10

如上通过指定特定的 case 进行测试，配置 ENV 中 S3_USE_SIGV4=1 和参数 -a auth_aws4 使能 AWS V4 鉴权（默认为 V2）, --debug-log 用于记录测试过程中详细日志。

执行全量测试

执行如下命令：

S3_USE_SIGV4=1 S3TEST_CONF=./config.yaml ./virtualenv/bin/nosetests -a auth_aws4 -v --with-html --html-report=/var/www/html/index.html

示例输出：

s3tests.functional.test_headers.test_object_create_bad_md5_invalid_short ... ok
s3tests.functional.test_headers.test_object_create_bad_md5_bad ... ok
s3tests.functional.test_headers.test_object_create_bad_md5_empty ... ok
s3tests.functional.test_headers.test_object_create_bad_md5_unreadable ... ok
s3tests.functional.test_headers.test_object_create_bad_md5_none ... ok
s3tests.functional.test_headers.test_object_create_bad_expect_mismatch ... ok
s3tests.functional.test_headers.test_object_create_bad_expect_empty ... ok
s3tests.functional.test_headers.test_object_create_bad_expect_none ... ok
s3tests.functional.test_headers.test_object_create_bad_expect_unreadable ... ok
s3tests.functional.test_headers.test_object_create_bad_contentlength_empty ... ok
s3tests.functional.test_headers.test_object_create_bad_contentlength_negative ... ok
s3tests.functional.test_headers.test_object_create_bad_contentlength_none ... ok
s3tests.functional.test_headers.test_object_create_bad_contentlength_unreadable ... ok
s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_above ... FAIL
s3tests.functional.test_headers.test_object_create_bad_contenttype_invalid ... ok
s3tests.functional.test_headers.test_object_create_bad_contenttype_empty ... ok
s3tests.functional.test_headers.test_object_create_bad_contenttype_none ... ok
s3tests.functional.test_headers.test_object_create_bad_contenttype_unreadable ... FAIL
s3tests.functional.test_headers.test_object_create_bad_authorization_unreadable ... FAIL
s3tests.functional.test_headers.test_object_create_bad_authorization_empty ... ok
s3tests.functional.test_headers.test_object_create_date_and_amz_date ... ok
s3tests.functional.test_headers.test_object_create_amz_date_and_no_date ... ok

最后见到如下统计输出，并且可以看到 HTML 输出到 /var/www/html/index.html 中了，代表任务成功运行，现在可以通过界面查看 HTML 端统计结果。

----------------------------------------------------------------------
HTML: /var/www/html/index.html
----------------------------------------------------------------------
Ran 570 tests in 542.311s
FAILED (SKIP=85, errors=26, failures=22)

4

统计分析

概览统计

查询测试结果，点击 Detail 展开单元测试项运行状态。

详细信息

通过点击失败用例对应的 Fail 按钮，查询具体错误信息。

通常我们根据断言信息及请求、响应 header 就可以定位到具体问题了。

5

自定义测试项

对于我们自定义的类S3接口（如自定义Meta信息、私有管理接口等），也可以很方便的通过扩展现有单元测试函数来实现，下面是 HTTP Range 请求下载功能的测试函数，函数位置：s3-tests/s3tests/functional/test_s3.py 中 Http Range 函数。

示例：

@attr(resource='object')
@attr(method='get')
@attr(operation='range')
@attr(assertion='returns correct data, 206')
def test_ranged_big_request_response_code():bucket = get_new_bucket()key = bucket.new_key('testobj')string = os.urandom(8 * 1024 * 1024)# 上传对象key.set_contents_from_string(string)# 设置 header，指定 Range 数据段key.open('r', headers={'Range': 'bytes=3145728-5242880'})status = key.resp.statuscontent_range = key.resp.getheader('Content-Range')fetched_content = ''for data in key:fetched_content += data;key.close()# 验证 Gateway 响应状态码，数据长度等eq(fetched_content, string[3145728:5242881])eq(status, 206)eq(content_range, 'bytes 3145728-5242880/8388608')

可以此为例，进行自定义用例扩展。

360云计算

由360云平台团队打造的技术分享公众号，内容涉及数据库、大数据、微服务、容器、AIOps、IoT等众多技术领域，通过夯实的技术积累和丰富的一线实战经验，为你带来最有料的技术分享