Disable TraceView

Disabling the instrumentation in your web application is a simple process. Because decisions about initiating traces are made in the webserver, you just disable the TraceView module for your web server, or change the tracing mode to ‘never’ in oboe.conf . With tracing disabled, impact on system performance is essentially negligible; no extra work is performed in processing web requests.

Change the tracing mode

Disable webserver instrumentation

On apache simply disable the module and restart the service.

a2dismod libapache2-mod-oboe
service apache2 restart

Uninstall TraceView

Uninstallation procedure is essentially the same as removing any other software from your system. It will vary depending on your distribution and the installation method used.

Error: No package available

If you’ve successfully run our install_traceview.sh or manually added our yum repo, and are able to install some of our rpm packages but not others, e.g., libapache2-mod-oboe but not php-oboe, you might have exclude directives in /etc/yum.conf. On rhel and centos, yum excludes are sometimes used by system administrators to prevent certain rpm packages from being considered by yum for installation or upgrade. For example, this guide suggests users add an exclude=php* kernel* line to prevent the linux kernel and php from receiving upgrades. This config prevents a specific rpm or version from being replaced by packages from the repositories managed by yum.

To verify if this is the case you can do a couple things: compare yum info _package_ against yum --disableexcludes=all info _package_ for the ostensibly missing package. If exclude= is configured, the former will not find the package while the latter will. The other thing you can do is check /etc/yum.conf for exclude directives using grep exclude /etc/yum.conf.

If exclude directives are in place, you can as a workaround use yum --disableexcludes=all install _package_ to temporarily ignore them. But, the workaround must be used every time you install or update an excluded package, which isn’t convenient when it comes time to upgrade. The best solution is to move your global exclude directives from /etc/yum.conf to the per-repository configuration files in the directory /etc/yum.repos.d. Do this for every repo except for the TraceView repo, of course. You can use the same syntax at the repo level as in /etc/yum.conf, as described here, you can use the same exclude= syntax as in /etc/yum.conf.

Traces aren’t showing up

There is normally only a delay of a few seconds between the time a request is made to your web stack and when the trace is visible in your dashboard. But haven’t got any traces coming in, double-check the following:

  1. First, check that our service is up at status.tv.solarwinds.com.
  2. Are requests coming in? In production, there is typically enough incoming traffic already. In development or QA environments however, you might have to generate some traffic manually. Just clicking around your web app will generally create enough requests to starting seeing traces. Keep in mind that cached pages won’t be traced.
  3. Is the Tracelyzer installed, and running, and connected? See this section in installation overview.
  4. Is web server instrumentation installed? By default, language instrumentation only continues traces that are started at the webserver level. Make sure that webserver instrumentation is installed, see the next step. If there’s no webserver sitting in front of your app, then you’ll have change the tracing mode for your language instrumentation.
  5. If apache instrumentation is installed:
    • If you’re using apache on rhel or with a custom apache config, in combination with mod_rewrite or mod_proxy, make sure that mod_oboe is loaded before all other modules.
    • mod_oboe does not trace correctly when Passenger, formerly mod_rails, is installed with PassengerHighPerformance mode turned on. PassengerHighPerformance mode is known to break many apache modules, including mod_rewrite. We are not aware of a workaround at this time. For more information, see www.modrails.com.
  6. Is component instrumentation installed, running, and connected? See this section in installation overview.
  7. Still nothing? Open a support ticket.

Troubleshooting rum

If auto-rum isn’t working as intended then you may find the diagnostic log messages helpful. These are controlled by the apache LogLeve, ErrorLog, ErrorLogFormat configuration directives and the output goes to the http server error log. grep for ‘Pagespeed traceview_rum’ and ‘TraceviewRumFilter’.

LogLevel info
Records when auto-rum is enabled and disabled.
LogLevel warn
Records problems in the generated html that prevent auto-rum from functioning correctly.
LogLevel debug
Records critical steps in the auto-rum processing.

Page caching, TraceView, and RUM

Caching, TraceView, and cookies

The request volume displayed by TraceView represents the quantity of the collected sample of requests hitting the application server. This means that if you are employing full-page caching (varnish, etc), subsequent requests for cached pages aren’t traced on the server side.
The upshot of this is that if you are using full page caching, certain endpoints (like the homepage) by almost always being cached will very rarely be traced. This effect can make your application appear to perform slowly, since the pages which are frequent cache misses (and thereby traced more often) tend to be dynamic such as login pages, which take a longer time to build.

Caching and real user monitoring

Our RUM instrumentation uses cookies to aid in measuring browser performance, like Google Analytics and other browser-based user analytics services. We use only first-party cookies, not third-party cookies — this means that all cookies used for RUM performance measurements for your domain send data only to the servers for your domain.
The cookie used by RUM begins with an underscore and is named tly. It is set by the client, not the backend, and thus will not appear as a server side Set-Cookie header in the HTTP response.
If your caching configuration bypasses cache when receiving client side prefixed cookies you may wish to consider changing this behaviour to maintain cache performance (for Varnish see below). If your caching configuration does not bypass cache when receiving client side prefixed cookies every user who hits that cached page will be counted together with the initial trace.

Interpreting cached RUM traces

When a RUM traced page winds up in a full page cache which then serves it up many times over each successive RUM trace will be associated with the original application trace. The diagram below shows the life cycle of this situation.

A single un-cached RUM trace looks much like a single application trace. One pass through the browser, timing for each stage of the client side experience, pretty straightforward.

But because cached RUM pages are handled a little differently they are also displayed a little differently. In this case time values plotted for the RUM trace are cumulative across every cache hit, and we break them down by location and browser demographics.

Caching specifics

Varnish

Varnish is a reverse proxy server capable of caching pages to reduce the load on the web server sitting behind it. Many TraceView users benefit from deploying Varnish in their environments, but there are some points to be aware of when using the two together.

Varnish and traces

The request volume within TraceView represents the size of the collected sample of requests hitting the application server. The sampling rate affects the raw request volume numbers, but doesn’t change trends in request volume. Because Varnish prevents requests from hitting the application server, it reduces the amount of traces generated within TraceView. Unlike sample rate, though, Varnish alters request volume trends in a non-random way due to the selective caching mechanisms enabled by VCLs.

The upshot of this is that if you are using Varnish, you may see very few or even no requests to certain endpoints (like the homepage) because those pages will always be in the cache. This effect can even appear to make your application perform slowly, since the pages not found in Varnish are often dynamic pages (such as login pages or shopping carts) that take a longer time to build.

Varnish and real user monitoring

If you are using Varnish along with our real user monitoring, you may wish to read about our real user monitoring cookies and edit your VCL files to take them into account.

An additional consideration when using both real user monitoring and Varnish (or similar full HTML page caching systems) is that a cached pageview executes the same JavaScript. As a result, many real user pageviews may point to one application-side trace, and the number of pageviews may even exceed the number of traces.

Varnish users can follow this official cookie configuration tutorial for an example of how to prevent underscore-prefixed cookies from skipping the cache.

Extending TraceView: Customizing

Though no modification of code is required to begin tracing requests through your web application, it can be advantageous to use TraceView as you would a normal application logging facility. Doing so combines the flexibility of ad hoc logging of application-specific data with the structure and context provided by TraceView.

It’s quite simple to do. In every application language that we instrument, a set of logging calls that look similar to native logging mechanisms are provided. By using these, you can add extra events with your application-specific information to a given trace. That data will then be accessible via the TraceView dashboard.

For some languages, we automatically capture outputs to the native logging callbacks (e.g. PHP’s error_log function).

Profiling

For most situations where you want to analyze a specific region of code, it will be easiest and more informative to use our function profiling interface. The alternative is custom layers, which should generally be reserved for calls to external services that our instrumentation would not normally recognize as needing their own layer.

For most languages, profiles can be added using annotations or decorators. In the case of needing to manually create profiles, see ‘Layers and Function Profiles’ further down this page.

Custom layers

Each event in a trace has an associated layer. Layers are logical groupings of events that occur in components of a web application. For instance, each separate component, like the webserver or the database, is considered to be part of a separate layer. A single extent (a bar as viewed on the traceview) represents a span of time spent in a layer during a particular call or request. It’s defined by an entry event and a matching exit event. In-between can be any number of other events for the same layer or even sub-layers.

Sometimes it is useful to designate layers manually, so that they will be treated as separate sections in the TraceView dashboard. An example of this might be a subsection of your application code that functions as a discrete service without being external to the process.

Using custom logging facilities, one can add annotations that mark a section of code as belonging to a separate layer. We offer native facilities in each supported language for making layer designations–eg. in Python, context managers and decorators.

To designate layers manually, use the variant of the language specific logging call that supports the optional layer parameter. However, there is a convention that must be obeyed in designating new layer: the first logging call marking entry into a new layer should have a label of “entry” and the final logging call marking the exit from the user-defined layer should have a label of “exit”. This allows us to gather the most accurate analytics and timing about your layer.

Example:

function myLogicallySeparateTask(...) {
oboe_log("serviceX", "entry", ...);
.. // do stuff
oboe_log("serviceX", "exit", ...);
}

While it is perhaps best to explicitly declare the layer name in each oboe_log(…) call, the logging methods do support inheritance. This means that the layer name need only be declared for the entry log. All subsequent logs before another entry log will inherit the correct layer.
Example:

function alotOfLogicallySeparateTaks(int choice){
switch(choice){
case 0:
oboe_log("LayerA","entry",...)
doWorkA(...)
break
case 1:
oboe_log("LayerB","entry",...)
doWorkB(...)
break
// etc ...
}
if(ERROR)
oboe_log(NULL, "error",...)
oboe_log(NULL, "exit", ...)      //Both these logs inherit their Layer
}

In certain languages, wrappers for methods are provided that simplify the task of specifying entry and exit events for a particular logging section. Check the instrumentation page for your language for more details.

Special interpretation

How does TraceView decide whether a layer is a query, a service call, or just part of an application? Certain types of extents receive special treatment. Below is information on the specifications for special extents.

Primer: extents

A single memcache call would be identified by a single extent: the entry and exit events determine the timing, and we also need to get certain information so we know it’s a cache request. The keys we need to get to know it’s a cache request (KVOp, KVKey are mandatory) can be located in either the entry event, the exit event, an event inside the layer, or any combination thereof.

Layers and function profiles

TraceView supports function profiling to provide more granular information about specific layers. Several instrumentation languages provide the option of using ‘addLayer’ or ‘addProfile’ methods, decorators, or annotations as appropriate. In some cases, though, the desired extent must be constructed more explicitly. Follow the specification described under ‘Custom layers’ above on this page, but define extents as follows:

To flag an extent as a layer:

  • Label - string - ‘entry’, later followed by ‘exit’
  • Layer - string - the name you want the layer to appear as in the interface (e.g., ‘drupal_views’)

To flag an extent as a profile:

  • Label - string - ‘profile_entry’, later followed by ‘profile_exit’
  • ProfileName - string - the name you want the profile to appear as in the interface (e.g., ‘render’)

For an example of using the PHP instrumentation’s oboe_log function to create a profile, see our function profiling page.

Errors

In order to be recognized as an error extent, an extent must report:

  • ErrorClass - string - the type of error
  • ErrorMsg - string - a more specific message

A backtrace will be automatically added. Many types of errors are already caught – eg. web server errors, PHP’s error log, or any exceptions that reach the Rails/Django/WSGI top layers.

Key-value store

In order to be recognized as a key-value store operation, an extent must report:

  • KVOp - string - the type of operation (get, set, etc)
  • KVHit - boolean, optional - whether the operation was a hit/miss – only makes sense for gets really
  • KVNoDuration - boolean, optional - should the duration of this event be measured - default true

  • RemoteHost - string, optional - the remote host connected to for this operation (deprecated: remote_host)

Controller/action

To report information about the controller and action hit by a request, the following keys should be present in in the MVC extent of your app:

  • Controller - string - name of controller
  • Action - string - name of action

Instrumentation for Rails and Django adds these values automatically.

HTTP server

HTTP servers typically serve as the entry point to a service. This is generally the entry point of each trace, but more complex architectures may have more than one HTTP call (internal proxy, external service, etc.). To instrument a layer as an HTTP server, report the following keys:

  • HTTP-Host - string - HTTP host, e.g. a domain name

Additionally, the following keys are also recognized if HTTP-Host is present:

  • URL - string - The full URL requested by the client of this layer.
  • Status - number - The HTTP status code returned.
  • Method - string - The HTTP request method (GET, POST, PUT, etc.). Only standard methods are recognized – nonstandard values will be ignored.
  • ClientIP - string, optional - client IP address ie. ‘REMOTE_ADDR’
  • Forwarded-For - string, optional - forwarded IP from proxy server, if the X-Header exists ie. ‘X_FORWARDED_FOR’
  • Forwarded-Host - string, optional - forwarded hostname from proxy server, if the X-Header exists ie. ‘X_FORWARDED_HOST’

Remote service call

A remote service call might be cURL, Thrift, or any other RPC/HTTP transport. It’s identified by the following (mandatory) keys:

  • RemoteProtocol - string - eg. HTTP / HTTPS / Thrift
  • RemoteHost - string - hostname of remote host
  • RemoteController - string - name of transport implementation (eg. cURL or thrift service name)
  • IsService - boolean - must be set to True
  • RemoteAction - string, optional - name of method call / destination used for categorization - defaults to RemoteHost
  • ServiceArg - string, optional - additional arguments / HTTP params

SQL Database Query

If an event has a Query key (and not Flavor = ‘mongodb’), it will be interpreted as part of a call representing a SQL database query.

  • Query - string - the query text
  • Flavor - string, optional - the type of database being accessed. Currently supported vaules: “mysql”, “postgresql”. Default is “mysql”.
  • RemoteHost - string - the hostname or IP of the machine running the database server

MongoDB Query

For mongodb instrumentation, use the following KV spec:

  • Flavor - string - must be mongodb
  • QueryOp - string - mongo operation, one of eg. command, count, create_collection, create_index, distinct, drop, drop_collection, drop_index, drop_indexes, ensure_index, find, find_and_modify, group, index_information, inline_map_reduce, insert, map_reduce, options, reindex, remove, rename, save, sort, update
  • Query - string - json query action, eg. {“type”:”MyType”,”name”:”MyName”,”info”:{“y”:”102”,”x”:203},”count”:1}
  • Limit - int - if limit clause used, attach limit kv (optional)
  • Database - string - mongo database being used
  • Collection - string - mongo collection (optional; mandatory unless not provided, eg. in command)
  • RemoteHost - string - mongo database host (optional but very highly desired)

Redis query

KVOp
string - one of these names: http://redis.io/commands
KVKey
string, optional - key referenced if applicable, don’t retrieve if more than 1
RemoteHost
string, optional - remote host, if possible
KVHit
boolean, optional - for get command, was it a hit? — not covered in documentation, but I suspect this also applies to getbit, getrange, getset
Script
string, optional - for script-related commands, the first few (100?) bytes of the script, or the script hash