SolarWinds Application Performance Monitoring (APM)

Features

Pinpoint the root cause of performance issues

Find slow queries, bugs, inefficient code, or problems with the infrastructure.

AppOptics trace- and service-level root cause simplify pinpointing application performance issues. These simplified views work for legacy monolithic applications just as well as they work for modern apps running in Kubernetes. Remove the guesswork from your incident response and have your teams focus on working together to solve the problem.

Performance monitoring

Comprehensive performance monitoring, from custom on-premises to your highly distributed cloud applications, in one product.

AppOptics performance monitoring goes deep and wide, including applications, transactions, services, servers, hosts, containers, and serverless. It’s powerful made simple with auto-instrumentation, a one-click connection from hosts and transactions to their associated logs, and simplified root cause summaries, making APM valuable in just about anyone’s hands.

Full-stack visibility

Remove the wall between Application and Infrastructure metrics, allowing combined dashboards.

Get a bird’s-eye view across all your servers, hosts, containers, and serverless environments. Cross-reference application and infrastructure metrics side by side in the same dashboard.

Modern infrastructure monitoring

Consolidated monitoring of all your servers, hosts, containers, and serverless environments.

Infrastructure monitoring for your legacy and new IT implementations with simple-to-click drilldowns into resource utilization and performance by services, containers, and processes.

AWS monitoring

SaaS-based monitoring of your AWS services and custom applications.

Monitor over 30 AWS services with broad application language support (.Net, Java, PHP, Python, Scala, Node.js, Go, and Ruby).

Azure monitoring

SaaS-based monitoring of your Azure services and custom applications.

Monitor Azure services including Load Balancer, Cosmos DB, SQL Database, Redis, and Azure App Services with broad application language support (Java, PHP, Python, Scala, Node.js, Go, and Ruby).

Identify hotspots early

Quickly identify which hosts have exceeded thresholds and could be impacting application availability and performance.

Avoid downtime with color-coded heatmaps that provide visualizations of all your hosts, and containers with comprehensive alerting on key performance metrics across the stack.

150+ Plugins and Integrations

With over 150 out-of-the-box plugins, integrations, and custom metrics, you have everything you need to monitor your critical systems.

Measure what matters with AppOptics. It’s an APM tool offering the ability to perform full-stack monitoring of your systems, including key business metrics associated with those applications.

Traces in context of logs

Go from a specific trace to all the logs associated with that trace request.

Automated log context tracking out of the box. View the logs to get a full understanding of an issue. Enables you to click a single button to view the logs from a particular trace.

Identify and alert

Be the first to know when there’s an availability or performance issue.

Single pane of glass visualization of thresholds that have been exceeded, paired with comprehensive alerting through major notification services like Slack, PagerDuty, Opsgenie, email, webhooks, and more.

Identify performance trends

Leverage performance trends to improve performance based on business growth, seasonal changes, anticipated events, etc.

Look at infrastructure and application performance over time (hours, days, weeks, or months).

Reduce MTTR

AppOptics is a converged APM tool bringing application and infrastructure monitoring together on common dashboards.

Monitoring infrastructure and application metrics side by side, reduces the time it takes to identify what part of the stack is failing, so you can quickly get to root cause.

Accelerate troubleshooting

Thread the needle on application performance issues by identifying the functioning code causing the problem.

Code profiling combined with distributed trace enables you to go beyond the poor-performing service and identify the line of code causing the poor performance.

Align performance goals with business goals

Use AppOptics to measure application and business performance and their impact on your business

Incorporate custom metrics to combine business metrics side by side with system metrics. See and measure the impact infrastructure and application performance has on your business performance.

Optimize for speed, efficiency, and cost

Correlate resource allocation to application performance, so you use what you need—no more, no less.

Leverage out-of-the-box dashboards and analytics to maximize performance while ensuring you don’t overspend or under-utilize precious on-premises and cloud resources.

Benefits

• Having complete visibility into the health and performance of our applications and their
  underlying infrastructure
• Knowing our applications are meeting service-level objectives and performing the way they
  were designed
• Being able to quickly pinpoint the root cause of performance issues across the stack, down
  to a poor-performing line of code
• Reducing downtime and impact to users due to application errors, resource allocation, or
  performance issues across the stack
• Be confident our performance monitoring tool can scale with you technically and
  cost-effectively

New features and improvements in APM

Released September 18, 2018

Last Updated August 19, 2019

SolarWinds AppOptics APM is a Software-as-a-Service (SaaS)-delivered product that extends the application monitoring capabilities of SolarWinds SAM. It delivers advanced performance metrics for IIS-based, .NET applications that are integrated into the SAM experience.

You can use AppOptics APM to cross-reference application, server, and infrastructure metrics side-by-side in the same dashboard, and trace requests through your entire application and infrastructure. AppOptics APM also includes heat maps to isolate outliers and trends, which you can use to drill down to traces for root cause.

Adding this extra layer of information to the Orion Platform enables you to:

• Gain a code-level look into your AppInsight for IIS applications.
• Monitor the performance of custom IIS-based, .NET applications.
• Check the status of application stacks and see how tiers of applications communicate with each other.
• Determine the databases and users that applications send information to.
• Identify if issues are network-, server-, or application-related so you can route details to the right team.

System Requirements

Updated August 16, 2019

SolarWinds AppOptics APM is a Software-as-a-Service (SaaS)-based application performance management tool that displays advanced performance metrics for IIS-based, .NET applications when integrated with nodes in SolarWinds SAM.

Supported Platforms

This page includes a summary of all currently supported platforms:

Getting started

AppOptics can be attached to a Heroku application via the add-on page or via CLI. Here’s an example of provisioning the add-on with the segue plan tier:

heroku addons:create appoptics:segue
Creating appoptics-elliptical-89104... done, (segue)
Adding appoptics-elliptical-89104 to myappname... done

After successful provisioning, the native log-based metrics described below will automatically begin flowing into your AppOptics account and into a dashboard we’ve pre-configured to provide an overview of your application and database performance. It may take 2-3 minutes for data to appear. You can go to the AppOptics interface via your control panel to check things out, or read on to learn more!

Enabling APM

AppOptics provides agents that automatically monitor applications, generating both metrics about an application’s performance (APM), detailed distributed transaction traces (tracing), and error data.

Installing APM is as simple as adding our agent/SDK to your application and re-deploying. See below for instructions in each supported language.

Note that enabling the add-on will configure the agent to report into a new AppOptics account dedicated to the application, by automatically setting a APPOPTICS_SERVICE_KEY that points to the new account. If you have an existing AppOptics account and wish to use that instead, you can override the APPOPTICS_SERVICE_KEY on the dyno:

heroku config:set APPOPTICS_SERVICE_KEY="<your service key>" --app <app_name>

The service key should be in the format of <API Token>:<Service Name> where the API Token is from your existing AppOptics account and the service name can be any custom value, for example 123...def:my-heroku-service.

Ruby

The Ruby agent is hosted on RubyGems, making installation as simple as gem install appoptics_apm or adding gem 'appoptics_apm' to the end of your Gemfile if the application manages gems via bundler.

Ideally all application gems are required by Bundler.require, which guarantees loading in the order they appear in the Gemfile. Otherwise make sure to require 'appoptics_apm' only after all gems that need instrumentation are loaded.

Rails / Rack example

To install the agent on anything rack-based, like Rails, add the gem to the end of the application’s Gemfile:

source 'https://rubygems.org'
ruby '2.5.2'

…

gem 'appoptics_apm’

Then re-deploy your application:

git commit -a -m "add appoptics monitoring"
git push heroku master

Sinatra example

To install the agent, add the gem to the end of the application’s Gemfile:

source 'https://rubygems.org'
ruby '2.5.2'

gem 'sinatra'
gem 'appoptics_apm’

and require it after sinatra in the application code:

require 'sinatra'
require 'appoptics_apm'

get '/' do
  "Hello world!"
end

Optionally, update the dependencies locally with bundler:

bundle install

(Note that on OS X and other unsupported platforms the agent will install but go into no-op mode.)

When ready, re-deploy the application:

git commit -a -m "add appoptics monitoring"
git push heroku master

Other ruby frameworks

More information about our Ruby agent, including examples for Grape, Padrino, Rails / Rails metal controllers, and Sinatra can be found in the Ruby agent documentation.

Node.js

The Node.js agent is distributed as an npm package and can be installed via npm install appoptics-apm, then added to your application by requiring it before any other modules.

To install the agent, add it to the application’s package.json file:

{
  "name": "node-js-getting-started",
  "version": "0.3.0",
  ...
  "engines": {
    "node": "8.x”
  },
  "dependencies": {
    "ejs": "^2.5.6",
    "express": "^4.15.2",
    “appoptics-apm”: “5.9.2”
  },
  ...
}

and require it in the application entry point file before any other require() calls:

// must be first require
require('appoptics-apm')
const express = require('express')
const path = require('path')
const PORT = process.env.PORT || 5000

express()
  .use(express.static(path.join(__dirname, 'public')))
  .set('views', path.join(__dirname, 'views'))
  .set('view engine', 'ejs')
  .get('/', (req, res) => res.render('pages/index'))
  .listen(PORT, () => console.log(`Listening on ${ PORT }`))

Optionally, update the dependencies locally with npm:

npm install

(Note that on OS X and other unsupported platforms the agent will install but go into no-op mode.)

When ready, re-deploy the application:

git commit -a -m "add appoptics monitoring"
git push heroku master

Python

The Python agent is distributed as a PyPI package and can be installed via pip install appoptics-apm, then added to your application by importing its middleware or handler wrapper.

Django example

To install the agent, add it to the application’s requirements.txt file:

django
gunicorn
django-heroku
appoptics-apm

and import its django middleware wrapper in the application’s settings.py file:

import os
import django_heroku
from appoptics_apm import djangoware
...

Optionally, update the dependencies locally with pip.

pip install -r requirements.txt

(Note that on OS X and other unsupported platforms the agent will install but go into no-op mode.)

When ready, re-deploy the application:

git commit -a -m "add appoptics instrumentation"
git push heroku master

Other python frameworks

More information about our Python agent, including examples for Django, Flask and generic WSGI, and Tornado can be found in the Python agent documentation.

PHP

The PHP agent is distributed as an extension .so binary file that supports many popular Linux distros, and can be easily installed for your application by using the custom buildpack.

Note that our custom buildpack is a fork of the official heroku/php buildpack, and should be used to replace it. This can be done using the Heroku web console under the ‘Settings > Buildpacks’ tab, or with the Heroku CLI as shown below.

List current buildpack:

$ heroku buildpacks
=== app-name-12345 Buildpack URL
heroku/php

Replace the default php buildpack with our custom one:

$ heroku buildpacks:set https://github.com/appoptics/heroku-buildpack-apm-php.git
Buildpack set. Next release on app-name-12345 will use https://github.com/appoptics/heroku-buildpack-apm-php.git.
Run git push heroku master to create a new release using this buildpack.

It is possible to customize the agent install by setting these optional Config Vars:

• APPOPTICS_EXTENSION_DIR: PHP extension directory (will attempt to detect automatically if not set)
• APPOPTICS_INI_DIR: PHP INI directory (will attempt to detect automatically if not set)
• APPOPTICS_THREAD_SAFETY: install thread-safe (TS) or non-thread-safe (NTS) version (will attempt to detect automatically if not set)
• APPOPTICS_VERSION: version to install (default: latest)
• APPOPTICS_URL: URL to get the PHP agent from (default: https://files.appoptics.com/php)

After the buildpack has been added and all desired Config Vars have been set, the app can be re-deployed as usual and will have the PHP instrumentation enabled.

Java

The Java agent is distributed as the appoptics-agent.jar file that should be loaded on JVM startup via the -javaagent option.

Deploying with Git

To install the agent, create a dedicated sub-directory, e.g. appoptics-agent, under your local application source tree and change into that directory:

$ pwd
/home/example/heroku/java-getting-started
$ mkdir appoptics-agent/
$ cd appoptics-agent

Download appoptics-agent.jar and its sha256 from the desired release directory, e.g. latest, verify the checksum, then git add the agent jar so that it will be part of the application resource:

curl -LO https://files.appoptics.com/java/latest/appoptics-agent.jar
curl -LO https://files.appoptics.com/java/latest/appoptics-agent.jar.sha256
…
git add appoptics-agent.jar

Now edit the application Procfile to include the -javaagent option which should point to the downloaded agent jar file. An example of the Procfile content:

web: java -javaagent:appoptics-agent/appoptics-agent.jar -jar target/java-getting-started-1.0.jar

When ready, re-deploy the application:

git commit -a -m "add appoptics instrumentation"
git push heroku master

Deploying as war with Heroku CLI

Download appoptics-agent.jar and its sha256 from the desired release directory, e.g. latest, verify the checksum

curl -LO https://files.appoptics.com/java/latest/appoptics-agent.jar
curl -LO https://files.appoptics.com/java/latest/appoptics-agent.jar.sha256

Include the agent jar as an additional file in your slug with the --includes option. For example:

heroku war:deploy <path_to_war_file> --includes appoptics-agent.jar --app <app_name>

Configure the JAVA_OPTS:

heroku config:set JAVA_OPTS="-javaagent:appoptics-agent.jar" --app <app_name>

The dyno will restart with our Java agent enabled.

Optional files

Note that additionally, each release comes with optional files:

• appoptics-sdk.jar: the instrumentation SDK jar file that can be used to provide custom tracing calls to the application
• javaagent.json: JSON configuration file to customize agent behavior

Golang

The Go agent is an open-source project that provides wrappers and SDK calls that your application can use to instrument web requests and other workloads. Instrumentation is possible using our simple hooks, or with OpenTracing 1.0 API support.

The example below uses a very simple web app, the full source can be found in the agent’s example directory.

To install the agent, add the github.com/appoptics/appoptics-apm-go/v1/ao package to the application, below is an example using govendor:

govendor fetch github.com/appoptics/appoptics-apm-go/v1/ao

Then, use its wrappers or SDK calls to instrument the app. Relevant code snippets from the example web app:

// Wraps a standard HTTP handler with AppOptics instrumentation
package main

import (
        ...
        "github.com/appoptics/appoptics-apm-go/v1/ao"
)
…
func slowHandler(w http.ResponseWriter, r *http.Request) {
        time.Sleep(normalAround(2*time.Second, 100*time.Millisecond))
        w.WriteHeader(404)
        fmt.Fprintf(w, "Slow request... Path: %s", r.URL.Path)
}
...
func main() {
        http.HandleFunc("/slow", ao.HTTPHandler(slowHandler))
        ...

When ready, re-deploy the application:

git commit -a -m "add appoptics instrumentation"
git push heroku master

Custom metrics

In addition to seamlessly extracting native metrics provided by Heroku, AppOptics’ log integration enables you to publish your own custom metrics directly to AppOptics through the Heroku log stream. Simply write your measurements to standard output (stdout) using one of the formats specified below and AppOptics will automatically detect and collect your custom metrics.

Once you collect custom metrics, you can build dashboards and set up alerts with them.

The examples below are in Ruby syntax but can be replicated in any language that permits writing to standard output.

Custom counting

Count log lines are used to submit increments to AppOptics. You can submit increments as frequently as desired and every minute the current total will be flushed to AppOptics and reset to zero. For example the following combination would result in a value of 5 being recorded for the enclosing minute in a AppOptics metric named user.clicks:

$stdout.puts("count#user.clicks=1")
$stdout.puts("count#user.clicks=1")
$stdout.puts("count#user.clicks=3")

Aggregation is done on a per-tag basis; in the following case, user.clicks for user_id=1 receives a count of 1, while user_id=2 receives a count of 4:

$stdout.puts("count#user.clicks=1 tag#user_id=2")
$stdout.puts("count#user.clicks=1 tag#user_id=1")
$stdout.puts("count#user.clicks=3 tag#user_id=2 ")

Custom distributions

Measure log lines are used to submit individual measurements that comprise a statistical distribution. The most common use case are timings i.e. latency measurements, but it can also be used to represent non-temporal distributions such as counts. You can submit as many measures as you’d like (typically they are submitted per-request) and every minute AppOptics will calculate/record a complete set of summary statistics over the measures submitted in that interval:

• min – The smallest value.
• max – The largest value.
• average – The average of all values.
• median – The value that falls precisely in the middle of all values. Considered a good indicator of typical application performance.
• 95th percentile – The value that is larger than 95% of all values.
• 99th percentile – The value that is larger than 99% of all values.
• sum – The total of all values.
• count – The total number of values submitted.

Measures are submitted in a similar fashion to counts:

$stdout.puts("measure#database.query=200ms")

Note that measures create 4 separate metrics in AppOptics based on the submitted measure name. Using the example above, you would find:

• database.query – A complex metric that contains the min/mean/max/sum/count values. You can switch between them using AppOptics’ summary statistics support.
• database.query.median – The median value.
• database.query.perc95 – The 95th percentile value.
• database.query.perc99 – The 99th percentile value.

Custom sample metrics

Sample metrics are used to convey simple key/numerical value pairs when you are already calculating some kind of summary statistic in your app and merely need a simple transport mechanism to AppOptics. Typically you would submit sample metrics on some periodic tick and set said period on the metric in AppOptics. This example would create a metric in AppOptics with the name database.size:

$stdout.puts("sample#database.size=40.9MB")

Tagged metrics

AppOptics metrics support multidimensional tagging, meaning that you can associate a particular observation with one or more attributes you would like to be able to query/report on. In Heroku logging, this is done using a syntax of tag#tag1=value1 tag#tag2=value2 on the same line as the measurement being recorded (before or after the measurement itself).

For example, we can track the rate of requests to our API by both endpoint and customer, as well as by both:

$stdout.puts("count#api.volume=1 tag#api_endpoint=lookup tag#customer_id=123")

Native log-based metrics

After provisioning the add-on, AppOptics immediately begins collecting native Heroku metrics directly from your application’s log stream.

Router metrics are present in every Heroku application log stream and provide detailed information (including timing) about every request serviced by your application. As these details are sourced directly from the Heroku routing layer itself, it’s the truest measure of performance as experienced by your customers, accounting for any delays introduced by Heroku in addition to your application’s processing. AppOptics calculates a rich set of summary statistics (described below in custom distributions) including median, 95th percentile, and 99th percentile latency that provide unprecedented insight into your application’s performance. The metrics (described in the linked Heroku router docs) available in AppOptics currently are:

• heroku.router.http.requests [service, dyno, status_code, method] – Request count by application, dyno, HTTP status code, and method
• heroku.router.service.response_time [service, dyno, method] – Amount of time in millisections for the application to respond to a HTTP request
  • heroku.router.service.response_time.p50 [service, dyno, method] – 50th percentile response time
  • heroku.router.service.response_time.p95 [service, dyno, method] – 95th percentile response time
  • heroku.router.service.response_time.p99 [service, dyno, method] – 99th percentile response time
• heroku.router.connect.latency [service, dyno] – Amount of time in milliseconds spent establishing a connection to the backend web process
  • heroku.router.connect.latency.p50 [service, dyno] – 50th percentile connect latency
  • heroku.router.connect.latency.p95 [service, dyno] – 95th percentile connect latency
  • heroku.router.connect.latency.p99 [service, dyno] – 99th percentile connect latency

The heroku.router.http.requests metrics capture all 200, 300, 400 and 500-level status codes from the Heroku Router for your applications.

Additionally, these metrics include counts and breakdowns of common runtime errors, tagged by error code:

• heroku.errors.http [service, code]
• heroku.errors.logging [service, code]
• heroku.errors.runtime [service, code]

Postgres metrics are present in the log stream of any Heroku application with a provisioned Heroku Postgres database on a standard or premium tier. These metrics are summary statistics gathered by the Heroku Postgres service and are reported directly into AppOptics:

• heroku.postgres.load.load1 [service, database, addon]
• heroku.postgres.load.load5 [service, database, addon]
• heroku.postgres.load.load15 [service, database, addon]
• heroku.postgres.mem.total [service, database, addon]
• heroku.postgres.mem.free [service, database, addon]
• heroku.postgres.mem.cached [service, database, addon]
• heroku.postgres.mem.postgres [service, database, addon]
• heroku.postgres.db.size [service, database, addon]
• heroku.postgres.db.tables [service, database, addon]
• heroku.postgres.db.connections_active [service, database, addon]
• heroku.postgres.db.connections_waiting [service, database, addon]
• heroku.postgres.db.current_transaction [service, database, addon]
• heroku.postgres.db.index_cache_hit_rate [service, database, addon]
• heroku.postgres.db.table_cache_hit_rate [service, database, addon]
• heroku.postgres.db.read_iops [service, database, addon]
• heroku.postgres.db.write_iops [service, database, addon]

Tags: service is the connected app name, database is the database attachment name, and addon is the addon name.

Per-dyno runtime metrics are available to any Heroku application, but must be enabled by the user, as they are disabled by default. They provide insight into both memory usage and CPU load of each dyno and are reported with @host tag values of the dyno’s logical role e.g. web.1 or worker.2. You can enable them with the following commands:

heroku labs:enable log-runtime-metrics
Enabling log-runtime-metrics for ... done
heroku restart

Once enabled, your per-dyno runtime metrics will automatically begin showing up in your AppOptics account. These metrics are summary statistics gathered by the Heroku Runtime layer and are reported directly into AppOptics under their given names.

• heroku.dyno.load.load1 [service, dyno]
• heroku.dyno.load.load5 [service, dyno]
• heroku.dyno.load.load15 [service, dyno]
• heroku.dyno.mem.rss [service, dyno]
• heroku.dyno.mem.swap [service, dyno]
• heroku.dyno.mem.cache [service, dyno]
• heroku.dyno.mem.total [service, dyno]
• heroku.dyno.mem.pgout [service, dyno]
• heroku.dyno.mem.pgin [service, dyno]
• heroku.dyno.mem.quota [service, dyno]

AppOptics Interface

The interface can be accessed via the CLI:

$ heroku addons:open appoptics
Opening appoptics for sharp-mountain-4005...

or by visiting the Heroku Dashboard and selecting the application in question. Select AppOptics from the Add-ons menu.

Alerts

With AppOptics alerts, we’ll notify you when one of your key metrics exceeds a threshold or stops reporting. You can be notified via email, PagerDuty, Slack, webhook, or a number of other possible services.

To get you started, your Add-On account comes preconfigured with two alerts (which are disabled by default), designed to help you stay on top of problematic code deploys:

• Your application returns errors (HTTP 5xx response codes) for more than 1% of requests
• Your application’s response time averages over 1 second

These alerts are configured to email the Heroku account owner. To customize and enable these alerts, first visit the alerts page. Then, to enable an alert, click on the alert name or on the edit button. On the page that opens, click on enabled, then save.

General troubleshooting

It may take 2-3 minutes for the first results to show up in your AppOptics account after you have deployed your app and the first request has been received.

Note that if Heroku idles your application, measurements will not be sent until it receives another request and is restarted. If you see intermittent gaps in your measurements during periods of low traffic, this is the most likely cause.

Picking a plan

Depending on your intended usage level there are several plans to choose from. Plans are sized in terms of average dyno use. (You won’t be penalized for scaling up or down in the short term.)

AppOptics plans include a number of custom metrics with each plan tier, increasing roughly in proportion with the number of dynos monitored. Note that extensive use of custom metrics may cause your usage to fall into a higher plan tier–we’ll reach out if that happens.

All plans include the native router, dyno, and database metrics, as well as APM and custom metrics. Dashboarding, alerting, and API too!

Migrating between plans

You can migrate between plans at any time without any interruption to your metrics.

Use the heroku addons:upgrade command to migrate to a new plan.

$ heroku addons:upgrade appoptics:andromeda
-----> Upgrading appoptics:sombrero to sharp-mountain-4005... done, v18 ($139/mo)
       Your plan has been updated to: appoptics:andromeda

Removing the add-on

AppOptics can be removed via the CLI.

$ heroku addons:destroy appoptics
-----> Removing appoptics from sharp-mountain-4005... done, v20 (segue)

Before removing AppOptics, data can be exported through the AppOptics API.

Terms of Service

By installing the AppOptics Add-on, you agree to the Software Service Agreement.

Support

All AppOptics support and runtime issues should be submitted via one of the Heroku Support channels.