AWS S3 API

CHAOSSEARCH introduces several new concepts for refining and querying data from your S3 buckets. Think of each of these concepts as phases in the data analysis flow, to ultimately gain insights and value.

CHAOSSEARCH currently supports Amazon S3 REST API calls for:

GET Service (ListAllMyBuckets)
GET Bucket (List Objects) Version 2
GET Object

GET Service (Specific Bucket)

https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html

Returns some or all (up to 1,000) of the objects in a bucket

GET /V1/chaos-example-docs-bucket?list-type=2&delimiter=%2F&max-keys=100 HTTP/1.1
Host: subaccount.chaossearch.io
x-amz-date: Fri, 28 Feb 2019 21:32:02 GMT
Authorization: AWS ITKEVINDAVISEXAMPLEWKF:woKfdm36sGS+r3gF8KEM7IJ70=
x-amz-chaossumo-route-token: 5a31958b-fe3d-3m30-f00d-1da790bceee6

BucketName: "chaos-example-docs-bucket"
DiscoverData: {Complete: true, TrendingPrefixes: ["ip-127-0-0-1/"], Throttled: false,…}
Complete: true
DuplicateStats: []
FileTypes: [{Type: "application/appd-stream", TotalFiles: 4, TotalSize: 327916, TotalTime: 6204714853000}]
0: {Type: "application/appd-stream", TotalFiles: 4, TotalSize: 327916, TotalTime: 6204714853000}
SecurityStats: [{Type: "Public Objects", Stats: 0,…}, {Type: "Policies Allowed", Stats: 0,…},…]
0: {Type: "Public Objects", Stats: 0,…}
DetailsURL: "s3://cs-5a31958b-fe3d-3m30-f00d-1da790bceee6/cs-data/stats/2019-03-12T14-26-29.114Z.chaos-example-docs-bucket.policy.objects.txt:52"
Stats: 0
Type: "Public Objects"
1: {Type: "Policies Allowed", Stats: 0,…}
DetailsURL: "s3://cs-5a31958b-fe3d-3m30-f00d-1da790bceee6/cs-data/stats/2019-03-12T14-26-29.114Z.chaos-example-docs-bucket.policies.allowed.txt:55"
Stats: 0
Type: "Policies Allowed"
2: {Type: "Bucket Policies", Stats: 0,…}
DetailsURL: "s3://cs-5a31958b-fe3d-3m30-f00d-1da790bceee6/cs-data/stats/2019-03-12T14-26-29.114Z.chaos-example-docs-bucket.bucket.policies.txt:54"
Stats: 0
Type: "Bucket Policies"
StartTime: 1552400789114
Throttled: false
TotalFiles: 4
TotalSize: 327916
TrendingPrefixes: ["ip-127-0-0-1/"]
0: "ip-127-0-0-1/"
id: "4478e5c9-hi42-ru81-or73-882f3c81f00d"

GET Service (List Buckets)

GET / HTTP/1.1
Host: subaccount.chaossearch.io
x-amz-date: Fri, 28 Feb 2019 21:32:02 GMT
Authorization: AWS ITKEVINDAVISEXAMPLEWKF:woKfdm36sGS+r3gF8KEM7IJ70=
x-amz-chaossumo-route-token: 5a31958b-fe3d-3m30-f00d-1da790bceee6

GET Bucket (List Objects) Version 2

https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html

Operation returns some or all (up to 1,000) of the objects in a bucket

GET / HTTP/1.1
Host: subaccount.chaossearch.io
x-amz-date: Fri, 28 Feb 2019 21:32:02 GMT
Authorization: AWS ITKEVINDAVISEXAMPLEWKF:woKfdm36sGS+r3gF8KEM7IJ70=
x-amz-chaossumo-route-token: 5a31958b-fe3d-3m30-f00d-1da790bceee6

GET Object

http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectGET.html

To list metadata about all of the versions of objects in a bucket

GET /my-file.gz HTTP/1.1
Host: subaccount.chaossearch.io
x-amz-date: Fri, 28 Feb 2019 21:32:02 GMT
Authorization: AWS ITKEVINDAVISEXAMPLEWKF:woKfdm36sGS+r3gF8KEM7IJ70=

{
  "Bucket": "Cloudwatch Logs",
  "ModelStatus": {
    "parsed": {
      "rowCount": 60438,
      "analyzedSizeInBytes": 8990412
    },
    "objects": {
      "total": {
        "count": 316,
        "sizeInBytes": 911886,
        "first": {
          "etag": "\"362efbc187223423lkjslgkja34523\"",
          "size": 4220,
          "lastModified": "2019-02-28T22:20:19.000Z",
          "storageClass": "STANDARD",
          "key": "cs_query_alerting/2019022517201551392417/24234284-d09a-44d2-b0aa-1739f023erf3/2019-02-28-[$LATEST]f02986b6bfb34298b6aa89ad2dcdf8ee/000000.gz"
        },
    "pending": {
      "tasks": 0,
      "register": 0
    }
  },
  "Schema": {
    "fields": [
      {
        "id": "23342kl23-bacd-45a4-8b3f-7396c93651b1",
        "dataset": "79b86f69-10ff-4b13-85e4-658864891682",
        "pos": 0,
        "_type": "hidden"
      },
      {
        "_type": "direct",
        "pos": 1,
        "label": [
          "timestamp"
        ],
        "ref": 0,
        "id": "23342kl23-bacd-45a4-8b3f-7396c93651b1",
        "dataset": "79b86f69-10ff-4b13-85e4-658864891682",
        "loc": 1,
        "type": "TIMEVAL"
      },
      {
        "_type": "direct",
        "pos": 2,
        "label": [
          "message"
        ],
        "ref": 1,
        "id": "23342kl23-bacd-45a4-8b3f-7396c93651b1",
        "dataset": "79b86f69-10ff-4b13-85e4-658864891682",
        "loc": 2,
        "type": "STRING"
      }
    ],
    "orderings": [
      {
        "dID": "03651870233049146189",
        "first": {
          "field": {
            "id": "023342kl23-bacd-45a4-8b3f-7396c93651b1",
            "dataset": "23342kl23-bacd-45a4-8b3f-7396c93651b1",
            "pos": 0,
            "_type": "hidden"
          },
          "order": "asc"
        },
        "rest": [],
        "_type": "spec"
      }
    ]
  }
}

Common Request Headers

🚧
Common Request Headers