Troubleshoot the Ruby microagent

This is a non-exhaustive list of troubleshooting strategies. If you need help, please email us at support@sqreen.com or contact us through our web chat.

Increase log verbosity

The Sqreen microagent logs messages to a text file. These messages contain warnings about possible abnormal behavior and describe some actions the microagent is executing. Increase the verbosity of these messages to help you troubleshoot issues with the Sqreen microagent.

Use two environment variables to configure Sqreen's log verbosity:

Env variable name Role Default value Options
SQREEN_LOG_LOCATION Specifies a custom file to which the microagent writes logs. the terminal custom file path
SQREEN_LOG_LEVEL Sets the microagent logging level. WARN or CRITICAL INFO, DEBUG

Use INFO level in production environments.

Use DEBUG only for short periods of time due to the amount of data it generates.

Non-server Ruby code like rails console

Problem: Sqreen operates only in the context of HTTP requests. In addition, reporting to the backend needs an asynchronous background processing thread to be started, which only happens when a HTTP server is running. Therefore Sqreen (and notably the SDK) cannot work in rails console or other non-server Ruby code.

Solution: We encourage you to perform your tests within a controller, e.g in a running server via rails server.

Load Sqreen later

While the Sqreen gem makes a best effort to detect and handle dependencies, you may need to enforce an initialisation order.

For example in your Gemfile:

gem 'newrelic_rpm'
gem 'sqreen', require: false
Then in your Rails application's config/application.rb

module DemoApp
  class Application < Rails::Application
    # other settings redacted for brevity...
    initializer :sqreen do |app|
      require 'sqreen'
    end
  end
end

See this NewRelic issue for details.

Deprecated application tokens

Problem: There is a problem with my application token.

Solution: Sqreen microagents used to use application tokens. These tokens were unique to an application. Sqreen has deprecated application tokens.

Sqreen now uses organization tokens. These tokens are available throughout the organization to which your account belongs.

While Sqreen continues to support application tokens for backward compatibility in the short term, we encourage you to convert your applications to use organization tokens.

Expired tokens

Problem: My organization token is not working. The error message reads 401 Unauthorized.

Solution: If your Sqreen plan expires, the microagent and the organization tokens remain in your app, but the microagent cannot authenticate with the Sqreen Platform anymore.

To resolve the issue, you can uninstall the Sqreen microagent from your app, or you can contact Sqreen to renew your plan subsciption. With a renewed subscription, your microagent will be able to authenticate with the Sqreen Platform using the same organization tokens.

Failed to build gem native extension

Problem: When installing the microagent on , it fails with the error: ERROR: Failed to build gem native extension. The most common cause of this is that Alpine does not include the build system packages by default.

Solution: Install the build system packages using apk add build-base.

Unable to parse configuration file

Problem: The Sqreen microagent throws an error at startup: Unable to parse configuration file.

Solution: In the sqreen.yaml configuration file, the configurations have been input without correct indentation. Be sure to indent with two spaces.

Agent fails to visibly start

Problem: There is no log file nor visible output of the microagent attempting to boot. While the Sqreen gem makes a best effort to start automatically, some server startup commands, servers, or combination thereof have very different boot flows, and in limited cases cannot be reliably instrumented to perform automatic agent boot. Known problematic methods are unicorn_rails or rackup with some web servers.

Solution: The microagent may be unable to handle some specific combinations of Ruby web servers and startup commands. If a specific web server or command fails, try another one. Make sure to prefix commands with bundle exec (optional for rails commands). Commands known to work include rails server, puma, unicorn, thin start, passenger start, rainbows. Commands known not to work or only work partially include rackup, and rails server when combined with the convenience unicorn-rails gem. We recommend trying with another command such as unicorn, puma, or rails server, or attempt to use another web server with your preferred command.

Sqreen starts but no data appears in the Dashboard

Problem: Skylight < 5.0.0.beta instruments in a way that prevents Sqreen's instrumentation from being called.

Solution: See "Infinite recursive calls" below.

Infinite recursive calls leading to SystemStackError (stack level too deep)

Problem: Multiple gems attempt to perform instrumentation in incompatible ways.

Solution: Make sure to use gem versions that use the Module#prepend best practice (even if they don't appear in the error):

  • opentelemetry
  • scout_apm >= 2.5.2
  • ddtrace >= 0.27
  • skylight >= 5.0.0.beta
  • elastic-apm >= 4.0 (upcoming release)
  • airbrake >= 11.0.2
  • newrelic_rpm >= 6.14 (with prepend_active_record_instrumentation: true configuration)

Ruby process segfaults, the C-level stack trace references mini_racer

Problem: The Ruby VM fails to isolate internal symbols when loading binary extensions. This can produce a jump from one C function from an extension to the same-named one in the other extension when clashing symbols are met. This happens notably when both mini_racer and sq_mini_racer gems are loaded but each uses a different libv8 versions.

Solution: Use recent versions of both mini_racer and sq_mini_racer gems. Unfortunately the fix in mini_racer is not released yet (as of 0.3.1) but is present on its repo's master branch. A workaround is to specify gem 'mini_racer', github: 'rubyjs/mini_racer' in your Gemfile.

Ruby process segfaults, the C-level stack trace mentions V8 isolates or contexts

Problem: libv8 currently needs threads to create and manage isolates. These threads do not properly survive forks, and the child would end up with an inconsistent V8 instance, causing a crash.

Solution: As V8 isolates are created by Sqreen to perform request protection, refrain from forking during a request, and favor background job processors such as Sidekiq. Alternatively, disabling protections that require JS execution would prevent libv8 contexts from spawning.

A new, "single threaded mode" of operation is being developed in V8, which could only be integrated in mini_racer and sq_mini_racer gems once released.

Agent notifications contains request error messages

Problem: Error in request to ingestion.sqreen.com: Response code for /ping is 502 or similar message appears in Agent notifications.

Solution: A proxy or firewall might block the mentioned hostname. If so, you might want to passlist it for HTTPS access. It might also be a transient network issue. If the issue persist, check if you can reliably reach the mentioned hostname over HTTPS from the location where your apps run. You can also check our service status page for any known disruption.

sq_mini_racer fails to install, logs mention "unrecognized relocation (0x2a)"

Problem: The system's linker is outdated and does not support a critical relocation feature. This is often encountered with Debian 8 "Jessie" and derivative OSes or Docker images.

Solution: Update the binutils OS package, or use legacy release sq_mini_racer 0.2.4.0.2, but we recommend you update your system.