When scrolling in elasticsearch it is important to provide at each scroll the latest scroll_id:

The initial search request and each subsequent scroll request returns

a new scroll_id — only the most recent scroll_id should be used.

The following example (taken from here) puzzle me. First, the srolling initialization:

rs = es.search(index=['tweets-2014-04-12','tweets-2014-04-13'],

scroll='10s',

search_type='scan',

size=100,

preference='_primary_first',

body={

"fields" : ["created_at", "entities.urls.expanded_url", "user.id_str"],

"query" : {

"wildcard" : { "entities.urls.expanded_url" : "*.ru" }

}

}

)

sid = rs['_scroll_id']

and then the looping:

tweets = [] while (1):

try:

rs = es.scroll(scroll_id=sid, scroll='10s')

tweets += rs['hits']['hits']

except:

break

It works, but I don't see where sid is updated... I believe that it happens internally, in the python client; but I don't understand how it works...

解决方案

In fact the code has a bug in it - in order to use the scroll feature correctly you are supposed to use the new scroll_id returned with each new call in the next call to scroll(), not reuse the first one:

Important

The initial search request and each subsequent scroll request returns

a new scroll_id — only the most recent scroll_id should be used.

It's working because Elasticsearch does not always change the scroll_id in between calls and can for smaller result sets return the same scroll_id as was originally returned for some time. This discussion from last year is between two other users seeing the same issue, the same scroll_id being returned for awhile:

So while your code is working for a smaller result set it's not correct - you need to capture the scroll_id returned in each new call to scroll() and use that for the next call.