How Sqreen works: In depth

Reading time: ~15 minutes

Sqreen architecture

Sqreen has three parts that work together to monitor and protect your application: the Sqreen Microagent, the Security Engine inside the microagent, and the Sqreen Platform.

Sqreen Microagent

The microagent is a library that you install in the web application, microservice, or API that you want to secure. The microagent is a design-time dependency (for example, a Java JAR file or a Ruby Gem) which executes at runtime. You install the language-specific microagent to match the language your app uses: Node.js, Ruby, PHP, Java, Go, or Python.

The following diagram illustrates where the microagent is located in the context of app development and deployment.

The microagent performs several critical functions from inside your application.

  • It communicates with the cloud-based Sqreen Platform.
  • It applies dynamic instrumentation points to the app’s code.
  • It monitors your application at each instrumentation point.
  • It performs security responses as directed by the Security Engine that resides inside it.

The microagent contains the Security Engine inside a self-contained sandbox (a V8 Javascript engine) inside your application. From inside this sandbox, the Security Engine performs all its critical checks and executes all callbacks on instrumented functions but remains insulated from the app.

When the application starts, the microagent launches a task to open a secure HTTPS communication with the cloud-based platform on port 443. The microagent examines the app’s execution logic, then sends the application’s stack information (databases, frameworks, languages, etc.) to the platform. The platform uses the full context of the app to determine which security rules the app needs, which it then sends to the Security Engine inside the microagent.

The microagent collects information using multiple dynamic instrumentation points in the application with marginal impact on the performance of the application itself. These dynamic instrumentation points are the hooks through which the microagent monitors and collects security metadata in the app. They are also the means through which the Security Engine inside the microagent executes its security logic at runtime.

Instrumentation: static vs. dynamic

To access an application’s internal data, an agent, such as a library, needs to instrument the code of the application it executes in.

Static instrumentation involves inserting instrumentation code into the application before runtime, using bespoke code crafted specifically for a dedicated place in the application dependencies. It is easier for a developer to write static instrumentation code based on what they know at build/deployment time about the content of the app’s underlying code. However, problems may stem from overspecializing the instrumentation to the underlying code: when the app's instrumented code changes, the overspecialized instrumentation may break. Application Performance Management (APM) tools like AppDynamics, New Relic, DataDog, and Dynatrace use static instrumentation.

Dynamic instrumentation, on the other hand, inserts instrumentation code into the application at runtime. It can be harder for a developer to write dynamic instrumentation code because they must build a generic way of instrumenting any kind of structure (function, public methods, private method, overloaded method). However, it is generally more robust because it is less dependent on the content of the instrumented code: changes in the app’s instrumented code do not break the instrumentation. The Sqreen Microagent uses dynamic instrumentation.

The following is a list of some of the functions and systems that the microagent dynamically instruments:

  • HTTP framework
  • SQL drivers
  • NoSQL drivers
  • template rendering engines
  • shell execution functions
  • file access functions
  • code evaluation functions
  • network libraries
  • XML libraries
  • cryptographic libraries

After it has set itself up, the microagent immediately begins to monitor the application’s execution logic at runtime.

Security Engine

The Security Engine is the term Sqreen uses to describe the security layers that function inside the microagent. The Security Engine houses the Runtime Application Self-Protection (RASP), Content Security Policy (CSP) and Security Headers, and In-App Web Application Firewall (In-App WAF) layers that observe requests and block attacks.

The Security Engine employs request-based detection, an activity that occurs synchronously inside your app. It observes and parses each request, then acts according to the rules and instructions the platform gave it about blocking or logging. Think of the engine as a bit of machinery inside the microagent that is constantly inspecting requests coming into the app and blocking requests, IP addresses, and/or user accounts that are unwelcome.

As an end-user, you have the ability to instruct the Security Engine how to take action when it detects an attack. It can either block a request, an IP address, or a user account, or log the metadata for asynchronous analysis in the platform. Log in to the Sqreen Dashboard to adjust the behavior, then apply the security rules without redeploying your app.

Runtime Application Self-Protection (RASP)

Sqreen’s RASP protects the app from real-time attacks from within the app’s own runtime environment, hence the name. To do this, RASP correlates the context and semantics of a request to look for anomalies.

For example, imagine your app uses a SQL database and you have installed Sqreen to protect your app. The microagent you installed inside the app hooked into the SQL driver methods so that it can check the queries going into the database. Before a request reaches the database, RASP uses a SQL parser to determine if the database will interpret the request parameters as executable SQL code. If the semantics of the parameters of the generated query are executable, such as SELECT * FROM Users WHERE UserId = 105 OR 1=1;, the Security Engine detects a SQL injection vulnerability and blocks or logs the information. (Whether Sqreen blocks the query or simply logs the metadata depends upon how you configure its behavior in the Sqreen Dashboard.)

Beyond SQL injection, Sqreen’s RASP detects and protects against several types of attacks, including Server-side Request Forgery (SSRF), Reflected Cross-Site Scripting (XSS), Local File Inclusion (LFI), and Shell Injection.

Learn more in the Protection documentation.

Content Security Policy (CSP) and Security Headers

A Content Security Policy (CSP) is an HTTP response header that works to prevent specific types of attacks, primarily Cross Site Scripting (XSS). Sqreen enables you to monitor the content sources with which a browser interacts so you can build a content security policy that specifies sources a browser can trust.

A browser’s HTTP response headers mainly contain metadata that instruct a website how to interact with the browser. However, you can include security rules with the response headers that a browser uses to protect itself against specific kinds of attacks. Use Sqreen to set up several HTTP security headers to protect against clickjacking, MIME sniffing, Cross-Site Scripting (XSS), and referrer leakage.

Learn more in the Protection documentation.

In-App Web Application Firewall (In-App WAF)

As its name implies, Sqreen’s In-App WAF operates as a web application firewall from within your application, microservice, or API. Like a WAF that exists outside an app, Sqreen’s In-App WAF inspects all incoming traffic and uses pattern-matching to detect and block malicious traffic. However, unlike an external WAF that applies a broad range of rules to inspect traffic, the In-App WAF uses the full context of your application – its databases, its frameworks, its language, its correctly-parsed parameters – to narrowly apply the most efficient set of inspection rules.

For example, if your app is in PHP and uses a SQL database, the In-App WAF inspects incoming traffic only for PHP-relevant and SQL-relevant patterns. This contextual efficiency not only reduces the amount of inspection work, it drastically reduces the false positive rate of an external WAF.

Also unlike an external WAF, the In-App WAF does not redirect or proxy the request traffic entering your app. This guards against single point of failure: if the In-App WAF experiences an issue, it does not affect the app.

Sqreen’s In-App WAF detects and protects against several types of attacks, such as Reflected Cross-site Scripting (XSS), Local File Inclusion (LFI), NoSQL Injection, and Shell Injection.

Learn more in the Protection documentation.

Access the Sqreen blog to learn more about how Sqreen built a WAF at the application layer.

Sqreen Platform

The platform comprises the cloud-based part of Sqreen’s hybrid Software as a Service (SaaS) model: the platform is in the cloud and the microagent is on-premise inside your app. Currently, Sqreen uses AWS EMEA to host the platform and is planning to expand its hosting geography.

The Sqreen Platform is the command center that communicates with the microagent embedded in your app. It collects request metadata from several microagents deployed in multiple instances of your app, then aggregates and normalizes the data. The platform then analyzes the aggregated data for anomalies.

When it detects an anomaly, the platform sends security responses to the Security Engine inside the microagent. Depending on how you configure Sqreen, the platform may send any of the following:

  • instructions to block an IP address or user
  • updated security rules
  • updated configurations you made on the Sqreen Dashboard (for example, changing an In-App WAF setting from monitoring to blocking)

Beyond its role as the asynchronous monitor and data aggregator, the platform also:

The Sqreen Platform also exposes the web app that you use to interact with Sqreen. Log in to your Sqreen Dashboard through to adjust configurations, track security events, and set performance limitations. You can also integrate the platform with several third-party tools and technologies to deepen your data analysis or hasten remediation. If you wish, you can extend Sqreen’s capabilities even further using Security Automation Playbooks and Sqreen Webhooks.

Sqreen operation

Imagine you have installed the microagent in your application. The microagent examined your app’s execution logic and collected information about the stack. It connected with the platform via HTTPS port 443 and sent the application’s stack information to the platform, which sent back security rules for the Security Engine inside the microagent. Then the microagent set up hundreds of its own dynamic instrumentation points in your app and immediately began monitoring the app’s execution logic. Sqreen is up and running.

Every 20 seconds, the microagent communicates with the platform with a heartbeat.

The microagent sends a bundle of request metadata it collected and any security events it detected or blocked since the last heartbeat. The following list offers examples of the metadata and security events that the microagent sends:

  • What: type of security event, anomaly, trace, metrics
  • Where: code-level stacktrace, server/container, URI/path
  • Why: scrubbed payloads, rules triggered
  • When: timestamp
  • Who: user identifier, IP address

The platform sends blocking instructions, updated security responses, and/or adjustments you made to the security configurations in the Sqreen Dashboard. If the platform has no new or updated information to convey, it sends a simple acknowledgment to indicate that it received the metadata the microagent sent. When the microagent receives acknowledgment that the platform correctly received the metadata, it can safely drop the metadata and start preparing for the next delivery.

Sensitive data and PII

The microagent does not send sensitive data, including Personally Identifiable Information (PII), to the Sqreen Platform. With each heartbeat, the microagent scrubs the metadata to remove sensitive data and replace any instances with Redacted by Sqreen.

By default, the microagent scrubs the following values from the metadata it sends:

  • Values that look like they contain credit card numbers, according to a basic regular expression: ^(?:\d[ -]*?){13,16}$
  • Values associated with any of the following keys:
    • password
    • secret
    • passwd
    • authorization
    • api_key
    • apikey
    • access_token

You can customize the sensitive data that the microagent redacts. Access your microagent's Configuration documentation to learn how.

Do not send PII! 🥧

Avoid configuring the microagent to send sensitive data, Personally Identifying Information (PII), or Personal Health Information (PHI) to the Sqreen Platform. Instead, use Universally Unique Identifiers (UUID) or hashes. Read this blog post to learn about best practices for user monitoring and PII.

Sqreen performance and reliability

The microagent operates with very limited performance impact on the app in which it is installed. For most applications, the microagent introduces a marginal CPU overhead of 4%. As a user, you can view the performance overhead and set performance thresholds using the Sqreen Dashboard.

The Security Engine runs in a self-contained sandbox inside the microagent embedded in the application. Any execution issues that the application experiences do not impact the Security Engine or microagent, and vice versa. This guards against single point of failure (SPOF) as a Security Engine or microagent that experiences difficulties does not stop or otherwise prevent the app from functioning.

The microagent does not impact network performance. Though it exchanges information with the cloud-based platform in regular 20-second intervals, the microagent does not depend on its connection to the platform and can continue to work by itself.

If connectivity with the platform breaks, the microagent caches the security metadata it collects in a fixed-size buffer. It keeps the metadata in the buffer until the platform acknowledges its receipt or the buffer becomes full. The microagent uses exponential backoff to continually retry its connection to the platform, but if it cannot reach the platform and the buffer becomes full, the microagent drops older metadata to prevent memory exhaustion. When it re-establishes a connection, the microagent sends the security metadata to the platform on a first-in-first-out basis.

If the microagent or the Security Engine encounter any operational issues and cannot continue monitoring or protecting your app, they do not interrupt or stop the app’s performance. Your app still processes queries and requests without interruption.