Integrate with Datadog

Introduction

This section describes the Sqreen security events and threats payloads, sent to Datadog as logs.

It serves as a reference documentation to help you customise your Datadog setup, for instance by building custom Dashoard widgets, security rules, etc.

Event types

All event types share a common sq.dd0 namespace.

The integration relies on the following event types:

  • sq.dd0.threat.attack
  • sq.dd0.threat.vulnerability_exploit
  • sq.dd0.incident
  • sq.dd0.playbook_trigger
  • sq.dd0.user_event.login
  • sq.dd0.user_event.signup
  • sq.dd0.custom_event

Those types are found in the event_name field.

Compatibility with Datadog built-in fields and distributed traces

The integration fills most of the standard Datadog attributes, as documented in the Datadog Docs.

The trace and span IDs are collected by the Sqreen microagent when deployed on the same web applications than the Datadog Tracer library (APM).

Payload versioning

The payload contains a version, following Semantic Versioning (SemVer). The version is contained in the event_version fields.

We recommend to specify the version of the payload used in your asset definitions (dashboards, security rules, etc), to ensure backward-compatibility as new version roll out. The current payload version is 1.0.0.

Base payload structure

All event types follow this JSON schema:

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "context": {
        "type": "object",
        "properties": {
            "actor": {
                "$ref": "actor.schema.json"
            },
            "http": {
                "$ref": "request_response.schema.json"
            },
            "service": {
                "$ref": "service.schema.json"
            }
        },
        "required": [
            "service"
        ]
    },
    "detected_at": {
        "type": "string",
        "format": "date-time"
    },
    "event_name": {
        "type": "string"
    },
    "event_version": {
        "type": "string"
    },
    "payload": {
        "type": "object"
    },
    "required": [
        "context",
        "detected_at",
        "event_name",
        "event_version",
        "payload"
    ]
}

Actor

This section contains all the information related to the IP and the authenticated user who originated the security event or the threat detected by Sqreen. It follows this JSON schema:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "properties": {
    "identifiers": {
      "type": ["object", "null"]
    },
    "ip": {
      "type": "object",
      "properties": {
        "address": {
          "type": "string",
          "format": "ipv4",
        },
        "geo": {
          "type": "object",
          "properties": {
            "city": {
              "type": "string"
            },
            "country_code": {
              "type": "string"
            },
            "latitude": {
              "type": "number"
            },
            "longitude": {
              "type": "number"
            }
          },
          "required": [
            "city",
            "country_code",
            "latitude",
            "longitude"
          ]
        },
        "metadata": {
          "type": "object",
          "properties": {
            "global": {
              "type": "boolean"
            },
            "loopback": {
              "type": "boolean"
            },
            "multicast": {
              "type": "boolean"
            },
            "network_type": {
              "type": "string"
            },
            "private": {
              "type": "boolean"
            },
            "reserved": {
              "type": "boolean"
            },
            "unspecified": {
              "type": "boolean"
            },
            "version": {
              "type": "integer"
            },
            "proxy": {
              "type": "boolean"
            },
            "tor": {
              "type": "boolean"
            },
            "vpn": {
              "type": "boolean"
            },
            "datacenter": {
              "type": ["object", "null"],
              "properties": {
                "name": {
                  "type": "string"
                },
                "url": {
                  "type": "string"
                }
              }
            }
          },
          "required": [
            "global",
            "loopback",
            "multicast",
            "network_type",
            "private",
            "reserved",
            "unspecified",
            "version",
            "proxy",
            "tor",
            "vpn",
            "datacenter"
          ]
        },

      },
      "required": [
        "address",
        "geo",
        "metadata"
      ]
    },
    "sqreen_identifier": {
      "type": ["string", "null"],
      "contentEncoding": "base64"
    }
  },
  "required": [
    "identifiers",
    "ip",
    "sqreen_identifier"
  ]
}

HTTP

This section contains all the information about the related HTTP request/response where Sqreen detected the security event/threat.

Each event type includes a specific payload definition, including specific information about the related event - for instance, the In-App WAF ruleset which detected an attack.

It follows this JSON schema:

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "properties": {
        "request": {
            "type": "object",
            "properties": {
                "host": {
                    "type": "string"
                },
                "path": {
                    "type": "string"
                },
                "port": {
                    "type": "integer"
                },
                "referer": {
                    "type": "null"
                },
                "remote_ip": {
                    "type": "string",
                    "format": "ipv4"
                },
                "remote_port": {
                    "type": "string"
                },
                "scheme": {
                    "type": "string",
                    "enum": ["http", "https"]
                },
                "user_agent": {
                    "type": "string"
                },
                "verb": {
                    "type": "string",
                    "enum": ["GET", "POST", "PUT", "PATCH", "DELETE", "OPTIONS"]
                }
            },
            "required": [
                "host",
                "path",
                "port",
                "referer",
                "remote_ip",
                "remote_port",
                "scheme",
                "user_agent",
                "verb"
            ]
        },
        "response": {
            "type": "object",
            "properties": {
                "blocked": {
                    "type": "boolean"
                },
                "status": {
                  "type": "integer"
                }
            },
            "required": [
                "blocked"
            ]
        }
    },
    "required": [
        "request",
        "response"
    ]
}

Service

This section describes the web application where Sqreen detected the security event/threat. It follows this JSON schema:

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "properties": {
        "env": {
            "type": "string"
        },
        "name": {
            "type": "string"
        },
        "sqreen_id": {
            "type": "string"
        }
    },
    "required": [
        "env",
        "name",
        "sqreen_id"
    ]
}

Specific payload definitions

Attack (sq.dd0.threat.attack)

This event type represents an attack detected by the Sqreen In-App WAF.

The payload definition follows this JSON schema:

{
    "type": "object",
    "properties": {
        "ruleset": {
            "type": "string",
            "enum": ["java_injection", "security_scanner", "nosql_injection", "js_injection", "sql_injection", "php_eval", "rfi", "shell_injection", "crs", "lfi", "xss", "protocol"]
        }
    },
    "required": [
        "ruleset"
    ]
}

Example log payload:

{
  "context": {
    "actor": {
      "identifiers": null,
      "ip": {
        "address": "172.58.56.226",
        "geo": {
          "city": "Albuquerque",
          "country_code": "USA",
          "latitude": 35.0813,
          "longitude": -106.6212
        },
        "metadata": {
          "global": true,
          "loopback": false,
          "multicast": false,
          "network_type": "public",
          "private": false,
          "reserved": false,
          "unspecified": false,
          "version": 4,
          "datacenter": {
            "name": "AWS",
            "url": "https://aws.amazon.com/"
          },
          "proxy": false,
          "tor": false,
          "vpn": false
        },
      },
      "sqreen_identifier": null
    },
    "http": {
      "request": {
        "host": "www.sqreen.com",
        "path": "/api/viewer/setup/",
        "port": 45031,
        "referer": null,
        "remote_ip": "10.13.235.87",
        "remote_port": 27205,
        "scheme": "http",
        "user_agent": "Amazon CloudFront",
        "verb": "GET"
      },
      "response": {
        "blocked": false
      }
    },
    "service": {
      "env": "production",
      "name": "sqreen bff",
      "sqreen_id": "5a26d99d23c2ce001be276c1"
    }
  },
  "detected_at": "2021-04-12T04:04:36.548Z",
  "event_name": "sq.dd0.threat.attack",
  "event_version": "1.0.0",
  "payload": {
    "ruleset": "crs"
  }
}

Vulnerability exploit (sq.dd0.threats.vulnerability_exploit)

This event type represents a vulnerability exploit detected by the Sqreen RASP.

The payload definition follows this JSON schema:

{
    "type": "object",
    "properties": {
        "protection": {
            "type": "string"
        }
    },
    "required": [
        "protection"
    ]
}

Example log payload:

{
  "context": {
    "actor": {
      "identifiers": null,
      "ip": {
        "address": "10.20.1.221",

        "geo": {},
        "metadata": {
          "global": false,
          "loopback": false,
          "multicast": false,
          "network_type": "private",
          "private": true,
          "reserved": false,
          "unspecified": false,
          "version": 4,
          "datacenter": {
            "name": "AWS",
            "url": "https://aws.amazon.com/"
          },
          "proxy": false,
          "tor": false,
          "vpn": false
        },
      },
      "sqreen_identifier": null
    },
    "http": {
      "request": {
        "host": "www.example.com",
        "parameters": {
          "form": [],
          "json": {},
          "other": {},
          "query": {
            "indexfile": "/var/www/exampleApi/frontend/deploy/index.html"
          }
        },
        "path": "/info/about-us/contact",
        "port": 8005,
        "referer": null,
        "remote_ip": "127.0.0.1",
        "remote_port": 45154,
        "scheme": "http",
        "user_agent": "ELB-HealthChecker/1.0",
        "verb": "GET"
      },
      "response": {
        "blocked": false
      }
    },
    "service": {
      "env": "development",
      "name": "deploy-api",
      "sqreen_id": "5db84b012df9f4003f92fed7"
    }
  },
  "detected_at": "2021-04-13T00:03:34.664Z",
  "event_name": "sq.dd0.threat.vulnerability_exploit.lfi",
  "event_version": "1.0.0",
  "payload": {
    "protection": "lfi"
  }
}

Security Incident trigger/update (sq.dd0.incident)

This event type represents a security incident detected by the Sqreen platform. This event is emitted either when the incident is first detected by Sqreen or when new security events are aggregated in the incident report.

The payload definition follows this JSON schema:

{
    "type": "object",
    "properties": {
        "event_type": {
            "type": "string",
            "enum": ["create", "update"]
        },
        "incident_type": {
            "type": "string",
            "enum": ["massive_sec_scan" ,"sql_injection", "nosql_injection", "shellshock", "ssrf", "shell_injection", "xxe", "xss", "lfi", "targeted_attack", "eval_injection", "insecure_deserialization", "failed_auth_peak", "user_risk_increase", "unusual_volume_of_signin", "account_creation_peak"]
        },
        "incident_id": {
            "type": "string"
        },
        "name": {
            "type": "string"
        },
        "severity": {
            "type": "string",
            "enum": ["minor", "major", "critical"]
        }
    },
    "required": [
        "event_type",
        "incident_type",
        "incident_id",
        "name",
        "severity"
    ]
}

Example log payload:

{
  "context": {
    "service": {
      "env": "production",
      "name": "gb-core",
      "sqreen_id": "5f62b63ec15112004ef2abbf"
    }
  },
  "detected_at": null,
  "event_name": "sq.dd0.incident",
  "event_version": "1.0.0",
  "payload": {
    "event_type": "create",
    "incident_type": "massive_sec_scan",
    "incident_id": "track_collection_abb776b736ca6654d2219aba0e874749",
    "name": "Massive security scan",
    "severity": "minor"
  }
}

Playbook trigger (sq.dd0.playbook_trigger)

This event type represents a security automation playbook triggers detected by the Sqreen platform.

The payload definition follows this JSON schema:

{
    "type": "object",
    "properties": {
        "name": {
            "type": "string"
        },
        "playbook_slug": {
            "type": "string"
        }
    },
    "required": [
        "name",
        "playbook_slug"
    ]
}

The playbook_slug field contains a slug of the playbook name. For example, given a playbook named "Bruteforce auth.signin", the slug would be: bruteforce_auth.signin.

Example log payload:

{
  "context": {
    "actor": {
      "identifiers": {
        "uid": ""
      },
      "ip": {
        "address": "10.109.129.87",

        "geo": {},
        "metadata": {
          "global": false,
          "loopback": false,
          "multicast": false,
          "network_type": "private",
          "private": true,
          "reserved": false,
          "unspecified": false,
          "version": 4,
          "datacenter": {
            "name": "AWS",
            "url": "https://aws.amazon.com/"
          },
          "proxy": false,
          "tor": false,
          "vpn": false
        },
      },
      "sqreen_identifier": "W1sidWlkIiwgIiJdXQ=="
    },
    "service": {
      "env": "production",
      "name": "lotr-api",
      "sqreen_id": "2c5782b9c4d92e001c0e38ec"
    }
  },
  "detected_at": "2021-04-12T04:04:36.548Z",
  "event_name": "sq.dd0.playbook_trigger",
  "event_version": "1.0.0",
  "payload": {
    "name": "Account takeover",
    "playbook_slug": "failed_auth_peak"
  }
}

User login/signup (sq.dd0.user_event.login/sq.dd0.user_event.signup)

Those event types represents user login and signup. Those user activities are detected either through the Sqreen SDK implementation or through native integration with authentication middleware (Ruby Devise, Django, Passport.js).

The payload definition follows this JSON schema:

{
    "type": "object",
    "properties": {
        "success": {
            "type": "boolean"
        }
    },
    "required": [
        "success"
    ]
}

Example log payload:

{
  "context": {
    "actor": {
      "identifiers": {
        "email": "example@yahoo.fr",
        "id": "2846803"
      },
      "ip": {
        "address": "91.184.105.203",
        "geo": {
          "city": "Paris",
          "country_code": "FRA",
          "latitude": 48.8323,
          "longitude": 2.4075
        },
        "metadata": {
          "global": true,
          "loopback": false,
          "multicast": false,
          "network_type": "public",
          "private": false,
          "reserved": false,
          "unspecified": false,
          "version": 4,
          "datacenter": {
            "name": "AWS",
            "url": "https://aws.amazon.com/"
          },
          "proxy": false,
          "tor": false,
          "vpn": false          
        },
      },
      "sqreen_identifier": "11siZW1haWwiLCAib2hheW9uLnNhcmFoQHlhaG9vLmZyIl0sIFsiaWQiLCAiMjg0NjgwMyJdXQ=="
    },
    "http": {
      "request": {},
      "response": {
        "blocked": false
      }
    },
    "service": {
      "env": "production",
      "name": "Example Web",
      "sqreen_id": "12bcd73d5a66460018cfb32b"
    }
  },
  "detected_at": "2021-04-12T04:04:36.548Z",
  "event_name": "sq.dd0.user_event.login",
  "event_version": "1.0.0",
  "payload": {
    "success": true
  }
}

Custom events (sq.dd0.custom_event)

This event type represents custom event occurrence, as tracked using the Sqreen Event SDK.

The event property keys/values are contained in the options field.

The payload definition follows this JSON schema:

{
    "type": "object",
    "properties": {
        "sdk_event_name": {
            "type": "string"
        },
        "options": {
            "type": "object"
        }
    },
    "required": [
        "event_name",
        "options"
    ]
}

Example log payload:

{
  "context": {
    "actor": {
      "identifiers": {
        "email": "user@example.com"
      },
      "ip": {
        "address": "220.235.53.246",
        "geo": {
          "city": "Kalamunda",
          "country_code": "AUS",
          "latitude": -31.9775,
          "longitude": 116.0551
        },
        "metadata": {
          "global": true,
          "loopback": false,
          "multicast": false,
          "network_type": "public",
          "private": false,
          "reserved": false,
          "unspecified": false,
          "version": 4,
          "datacenter": {
            "name": "AWS",
            "url": "https://aws.amazon.com/"
          },
          "proxy": false,
          "tor": false,
          "vpn": false
        },
      },
      "sqreen_identifier": "W1siZW1haWwiLCAiYW5kcmV3Lndhb123cHJrcy5jb20uYXUiXV0="
    },
    "http": {
      "request": {
        "host": "run.outfieldapp.com",
        "parameters": {
          "form": {},
          "other": {
            "format": "json",
            "page": [
              "1",
              "1"
            ],
            "updated_after_date": [
              "0001-01-01 00:00:00 +0000",
              "0001-01-01 00:00:00 +0000"
            ]
          },
          "query": {
            "page": "1",
            "updated_after_date": "0001-01-01 00:00:00 +0000"
          }
        },
        "path": "/api/v2/products",
        "port": 443,
        "remote_ip": "10.123.249.13",
        "remote_port": 0,
        "scheme": "https",
        "user_agent": "outfield-swift/200 CFNetwork/1220.1 Darwin/20.3.0",
        "verb": "GET"
      },
      "response": {
        "blocked": false,
        "status": 304
      }
    },
    "service": {
      "env": "production",
      "name": "hapyak",
      "sqreen_id": "5a26d92223c2ce001be276c1"
    }
  },
  "detected_at": "2021-04-13T00:03:34.664Z",
  "event_name": "sq.dd0.custom_event",
  "event_version": "1.0.0",
  "payload": {
    "event_name": "custom_event",
    "options": {
      "authenticateduserrole": "something",
      "isauthenticatedadmin": false,
      "isauthenticatedltistudent": false,
      "isauthenticatedopenclassroomuser": false,
      "isauthenticateduserusingthirdparty": false,
      "size": "17945091",
      "type": "video/mp4"
    }
  }
}

Distributed tracing context collection - known limits

The distributed context - i.e trace and span IDs - is collected by the Sqreen microagents when deployed next to the Datadog library - also referred to as the tracing library.

Trace ID are always provided, enabling to correlate threats detected by Sqreen to Datadog traces. However, the Sqreen microagents may not be able to link the attack or the vulnerability exploit to the exact span where it has been detected. In most cases, the span ID collected will point to the parent span (N-1). In rare occasions, it'll point to the HTTP request.

In Ruby, in order to block requests, the Sqreen microagent throws an exception. This exception is caught by the Datadog library and reported in the context of the trace.