ChaosSearch Scroll API - Python

This section shows an example scroll API request

Scroll API Request

A search request through the ChaosSearch API returns a single page of results. The scroll API can be used to retrieve large numbers of results (or even all results) from a single search request.

📘

Scrolling is not intended for real-time user requests, but rather for processing large amounts of data.

Example Request

Fields to update:

  • Input your access-key-id and secret-access-key from Settings (line 5)
  • Update the Index view name with the one you created in the Refinery (line 31)
  • Scroll- how often you want the scroll API to move through results (i.e. 1 minute = 1m) (line 33)
  • Size - Number of results that should return each time the scroll API makes a request (line 34)
from elasticsearch import Elasticsearch, helpers, exceptions, RequestsHttpConnection
import json
from requests_aws4auth import AWS4Auth

awsauth = AWS4Auth("Access-Key-ID", "Secret-Access-Key", "us-east-1", 's3')
es = Elasticsearch(
  hosts = [{'host': 'poc-trial.chaossearch.io', 'port': 443, 'url_prefix': '/elastic', 'use_ssl': True}],
  http_auth=awsauth,
  connection_class=RequestsHttpConnection,
  verify_certs=True
)

client = es

try:
    client_info = Elasticsearch.info(client)
    print ('Elasticsearch client info:', json.dumps(client_info, indent=4))
except exceptions.ConnectionError as err:
    print ('Elasticsearch client error:', err)
    client = None
if client != None:
    search_body = {
        "size": 500,
        "query": {
            "match_all": {}
        }
    }

    resp = helpers.scan(
        client,
        index = 'Index-View-Name', # <---- SPECIFY HERE
        query = search_body,
        scroll = '1m',
        size = 10,
    )

    print (type(resp))

    for num, doc in enumerate(resp):
        print ('\n', num, '', doc)

Did this page help you?