locust压测工具：http测试过程与定时停止

locust压测环境描述

本文环境python3.5.2
locust版本0.9.0

locust示例的执行过程

上文大概描述了locust的启动了流程，本文主要是接着上文继续分析，示例代码中的http的测试的执行过程，是如何去访问远端的http接口的流程。接着就分析如何通过传入的运行时间参数来停止locust的运行。

http测试用例的执行

示例代码中访问url的代码如下：

@task(2)
def index(self):self.client.get("/")@task(1)
def profile(self):self.client.get("/profile")

其中的client就是继承自HttpLocust初始化过程中创建的，由于示例代码里面的WebsiteUser继承自HttpLocust，在初始化的过程的时候；

class HttpLocust(Locust):"""Represents an HTTP "user" which is to be hatched and attack the system that is to be load tested.The behaviour of this user is defined by the task_set attribute, which should point to a :py:class:`TaskSet <locust.core.TaskSet>` class.This class creates a *client* attribute on instantiation which is an HTTP client with support for keeping a user session between requests."""client = None"""Instance of HttpSession that is created upon instantiation of Locust. The client support cookies, and therefore keeps the session between HTTP requests."""def __init__(self):super(HttpLocust, self).__init__()if self.host is None:raise LocustError("You must specify the base host. Either in the host attribute in the Locust class, or on the command line using the --host option.")self.client = HttpSession(base_url=self.host)               # 初始化一个client实例

此时的client就是一个HttpSession的实例，分析该类：

class HttpSession(requests.Session):                                    # 继承自requests.Session"""Class for performing web requests and holding (session-) cookies between requests (in orderto be able to log in and out of websites). Each request is logged so that locust can display statistics.This is a slightly extended version of `python-request <http://python-requests.org>`_'s:py:class:`requests.Session` class and mostly this class works exactly the same. However the methods for making requests (get, post, delete, put, head, options, patch, request) can now take a *url* argument that's only the path part of the URL, in which case the host part of the URL will be prepended with the HttpSession.base_url which is normally inheritedfrom a Locust class' host property.Each of the methods for making requests also takes two additional optional arguments which are Locust specific and doesn't exist in python-requests. These are::param name: (optional) An argument that can be specified to use as label in Locust's statistics instead of the URL path. This can be used to group different URL's that are requested into a single entry in Locust's statistics.:param catch_response: (optional) Boolean argument that, if set, can be used to make a request return a context manager to work as argument to a with statement. This will allow the request to be marked as a fail based on the content of the response, even if the response code is ok (2xx). The opposite also works, one can use catch_response to catch a requestand then mark it as successful even if the response code was not (i.e 500 or 404)."""def __init__(self, base_url, *args, **kwargs):super(HttpSession, self).__init__(*args, **kwargs)                                  # 调用父类的初始化方法self.base_url = base_url                                                            # 设置Host# Check for basic authenticationparsed_url = urlparse(self.base_url)                                                # 解析urlif parsed_url.username and parsed_url.password:                                     # 检查是否配置了用户名与密码netloc = parsed_url.hostnameif parsed_url.port:netloc += ":%d" % parsed_url.port# remove username and password from the base_urlself.base_url = urlunparse((parsed_url.scheme, netloc, parsed_url.path, parsed_url.params, parsed_url.query, parsed_url.fragment))# configure requests to use basic authself.auth = HTTPBasicAuth(parsed_url.username, parsed_url.password)def _build_url(self, path):""" prepend url with hostname unless it's already an absolute URL """if absolute_http_url_regexp.match(path):                                            # 检查是否是绝对路径return path                                                                     # 如果是绝对路径则直接返回else:return "%s%s" % (self.base_url, path)                                           # 返回完整的urldef request(self, method, url, name=None, catch_response=False, **kwargs):"""Constructs and sends a :py:class:`requests.Request`.Returns :py:class:`requests.Response` object.:param method: method for the new :class:`Request` object.:param url: URL for the new :class:`Request` object.:param name: (optional) An argument that can be specified to use as label in Locust's statistics instead of the URL path. This can be used to group different URL's that are requested into a single entry in Locust's statistics.:param catch_response: (optional) Boolean argument that, if set, can be used to make a request return a context manager to work as argument to a with statement. This will allow the request to be marked as a fail based on the content of the response, even if the response code is ok (2xx). The opposite also works, one can use catch_response to catch a requestand then mark it as successful even if the response code was not (i.e 500 or 404).:param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`.:param data: (optional) Dictionary or bytes to send in the body of the :class:`Request`.:param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`.:param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`.:param files: (optional) Dictionary of ``'filename': file-like-objects`` for multipart encoding upload.:param auth: (optional) Auth tuple or callable to enable Basic/Digest/Custom HTTP Auth.:param timeout: (optional) How long in seconds to wait for the server to send data before giving up, as a float, or a (`connect timeout, read timeout <user/advanced.html#timeouts>`_) tuple.:type timeout: float or tuple:param allow_redirects: (optional) Set to True by default.:type allow_redirects: bool:param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.:param stream: (optional) whether to immediately download the response content. Defaults to ``False``.:param verify: (optional) if ``True``, the SSL cert will be verified. A CA_BUNDLE path can also be provided.:param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair."""# prepend url with hostname unless it's already an absolute URLurl = self._build_url(url)                                                                  # 获取访问路径# store meta data that is used when reporting the request to locust's statisticsrequest_meta = {}                                                                           # 请求头部信息# set up pre_request hook for attaching meta data to the request objectrequest_meta["method"] = method                                                             # 请求的方法request_meta["start_time"] = time.time()                                                    # 请求开始的时间response = self._send_request_safe_mode(method, url, **kwargs)                              # 访问请求并获取返回值# record the consumed timerequest_meta["response_time"] = (time.time() - request_meta["start_time"]) * 1000           # 记录响应处理的时间request_meta["name"] = name or (response.history and response.history[0] or response).request.path_url# get the length of the content, but if the argument stream is set to True, we take# the size from the content-length header, in order to not trigger fetching of the bodyif kwargs.get("stream", False):request_meta["content_size"] = int(response.headers.get("content-length") or 0)         # 获取返回的文件长度else:request_meta["content_size"] = len(response.content or b"")                             # 否则获取响应的整体大小if catch_response:                                                                          # 如果要包含请求信息response.locust_request_meta = request_meta return ResponseContextManager(response)                                                 # 用ResponseContextManager包裹responseelse:try:response.raise_for_status()                                                         # 检查返回状态except RequestException as e:events.request_failure.fire(request_type=request_meta["method"], name=request_meta["name"], response_time=request_meta["response_time"], exception=e, )                                                                                   # 如果失败则通知所有的失败请求事件执行else:events.request_success.fire(request_type=request_meta["method"],name=request_meta["name"],response_time=request_meta["response_time"],response_length=request_meta["content_size"],)                                                                                   # 如果成功则通知所有的成功事件执行return responsedef _send_request_safe_mode(self, method, url, **kwargs):"""Send an HTTP request, and catch any exception that might occur due to connection problems.Safe mode has been removed from requests 1.x."""try:return requests.Session.request(self, method, url, **kwargs)                # 调用requests的Session去请求接口except (MissingSchema, InvalidSchema, InvalidURL):raiseexcept RequestException as e:r = LocustResponse()r.error = er.status_code = 0  # with this status_code, content returns Noner.request = Request(method, url).prepare() return r

由该代码可知，处理的请求都是通过requests库中的Session来进行请求的，示例代码中的client都是通过requests的代码进行请求，并且还可以使用session来保持会话，从而使接口请求的时候能够带上权限检查等额外信息。

locust定时退出

由于可以在启动locust可以指定执行的时间，可以到时间退出，我们分析一下该退出函数的执行，

    def spawn_run_time_limit_greenlet():logger.info("Run time limit set to %s seconds" % options.run_time)def timelimit_stop():logger.info("Time limit reached. Stopping Locust.")runners.locust_runner.quit()gevent.spawn_later(options.run_time, timelimit_stop)

等到了run_time之后，就会执行timelimit_stop函数，而该函数就是调用了实例化的locust类实例的quit方法；

def stop(self):# if we are currently hatching locusts we need to kill the hatching greenlet firstif self.hatching_greenlet and not self.hatching_greenlet.ready():       # 检查是否有hatch_greenlet并且没有准备好 就杀掉该协程self.hatching_greenlet.kill(block=True)self.locusts.kill(block=True)                                           # 杀死所有的协程组self.state = STATE_STOPPED                                              # 更改状态为停止态events.locust_stop_hatching.fire()                                      # 通知所有locust_stop_hatching的函数执行def quit(self):self.stop()                                                             # 停止执行所有的greenletself.greenlet.kill(block=True)                                          # 主greenlet杀死

此时就是停止所有的协程执行，终止该测试用例的执行。其中events.locust_stop_hatching使用了典型的观察者设计模式：

locust_stop_hatching = EventHook()class EventHook(object):"""Simple event class used to provide hooks for different types of events in Locust.Here's how to use the EventHook class::my_event = EventHook()def on_my_event(a, b, **kw):print "Event was fired with arguments: %s, %s" % (a, b)my_event += on_my_eventmy_event.fire(a="foo", b="bar")If reverse is True, then the handlers will run in the reverse orderthat they were inserted"""def __init__(self):self._handlers = []                                         # 所有待处理的handlersdef __iadd__(self, handler):self._handlers.append(handler)                              # 添加到处理的Handler列表中return selfdef __isub__(self, handler):self._handlers.remove(handler)                              # 移除handlerreturn selfdef fire(self, reverse=False, **kwargs):if reverse:self._handlers.reverse()                                # 是否排序for handler in self._handlers:                              # 依次遍历handler并执行handler(**kwargs)

总结

本文主要是继续分析了locust启动之后，http的请求的处理与定时退出的功能，其中http的请求都是基于requests.Session来实现的，定时退出的功能主要还是依赖于gevent中的Group来控制所有已经运行的协程，通过停止所有运行的协程来达到关闭停止运行的目的，其中还有些许细节并没有详细说明，大家有兴趣可自行查阅相关源码。鉴于本人才疏学浅，如有疏漏请批评指正