Python support matrix

Python versions

cPython
2.6.x
cPython
2.7.x

Python frameworks

Python databases

  • SQLAlchemy
  • Django ORM
  • PyMongo
  • pycassa 1.7.1

Python RPC clients

Apache Thrift
0.5, 0.6, 0.8
httplib
httplib2
urllib
urllib2
urllib3
requests

Other Python components

Greenlet
0.3.1, 0.3.4
gevent
via Greenlet
Eventlet
via Greenlet
Memcache
pylibmc

Installing instrumentation

Our python module called oboe gets detailed insights out of the box, without any code modification. The oboe package also provides the oboeware module, which contains middleware components and instrumentation helpers for Django, Tornado, and other supported components.

When you install language level instrumentation, here are the things you should be thinking about: first, install the instrumentation. Then, go back to your dashboard and take a look at your traces. Are you getting the level of detail you need? If you’re not getting a sufficient level of detail from out-of-box instrumented php, then you’ll want to consider configuration options.

Python instrumentation is provided by the oboe package, which can either be downloaded manually from its home on PyPI or installed automatically with pip. If you haven’t used pip before you might need to first install python-pip, g++, and python-dev using your favorite package manager.

pip install oboe

Add django instrumentation

Add the following import to your django settings.py as well as your WSGI file, if you have one. If you’re deploying with uWSGI and are specifying the WSGI handler module as django.core.handlers.wsgi:WSGIHandler(), you can use our handler module instead, which is a drop-in replacement. Just specify oboeware.djangoware:OboeWSGIHandler() and you won’t need to modify your settings.py.

import oboeware.djangoware

Add greenlet instrumentation

Support for event-based services is provided through a modified greenlet module.

pip install --extra-index-url=https://pypi.tv.solarwinds.com greenlet==0.4.5-oboe1

Add tornado instrumentation

TraceView supports web apps written with the tornado web framework for python. The tornado instrumentation relies on the insertion of a handful of one-line calls to the oboeware library, and thus requires patching the tornado library. Patched tornado packages are available from our pypi repository, or you can download a git patch from files.tv.solarwinds.com and apply it yourself. To install our tornado packages using pip, specify the version and pypi address.

pip install --extra-index-url=http://pypi.tv.solarwinds.com tornado==2.3-oboe3

Add WSGI instrumentation

Many python apps are implemented atop WSGI, which provides a uniform interface through which web frameworks and middleware libraries interoperate. TraceView offers a WSGI middleware component called OboeMiddleware, which wraps your WSGI app to collect performance statistics. Install middleware instrumentation as you would any other middleware by adding it to the file that instantiates your app. But put it at the end of the list, closest to the outside of the middleware stack.

Set the tracing mode: If you don’t have an instrumented webserver in front of your app, you’ll also need to set the tracing mode.

 1 # flask example
 2 from flask import Flask
 3 from oboeware import OboeMiddleware
 4  
 5 # the flask app
 6 app = Flask(__name__)
 7 
 8 app.wsgi_app = OboeMiddleware(app.wsgi_app, {'tracing_mode':'always'})
 9  
10 @app.route("/")
11 def hello():
12     return "Hello World!"

Upgrading instrumentation

The python instrumentation package is managed by the pip command regardless of platform.

pip install oboe --upgrade

Configuring instrumentation

Common configuration tasks:

Report backtraces

The instrumented libraries have the ability to collect backtraces; some of them collect backtraces by default, others don’t. See complete list of python configuration options. Many public API methods also take backtrace parameters.

oboe.config['inst']['memcache']['collect_backtraces'] = True

Suppress reporting for sensitive data

Python instrumentation has the ability to sanitize sql database queries in case you don’t want query parameters displayed in your account. By default this functionality is disabled, but you can use the sanitize_sql flag to configure the instrumentation to stop collecting and reporting query parameters.

oboe.config['sanitize_sql'] = True

Disable debug logging

By default, the oboe module emits debug output on stderr, which for most environments is mixed into the web application’s logs. You can check there for debug information if something doesn’t seem to be working as expected. This output can also be disabled via an environment variable. Running your application with OBOE_DISABLE_DEFAULT_LOGGER set in your environment disables the logging entirely.

Complete config options

The following is the complete list of python instrumentation configuration options.


    oboe.config[‘tracing_mode’]

    config option
    oboe.config[‘tracing_mode’]
    description
    Specify when traces should be initiated for incoming requests. This should be set to ‘always’ when you don’t have an instrumented webserver in front of your app.
    possible values
    ‘always’
    (Default) Continue existing traces, otherwise attempt to start a new one.
    ‘through’
    Continue existing traces, but never start them. This mode assumes that a higher layer makes the decision about whether to trace.
    ‘never’
    Never continue existing traces or start new ones.
    history
    Introduced with the original python instrumentation.
    example
    oboe.config['tracing_mode'] = 'always'

    oboe.config[‘inst_enabled’]

    config option
    oboe.config[‘inst_enabled’]
    description
    To disable an individual instrumentation module, simply add the following lines before loading the middleware into your app.
    parameters
    [‘<module-name>’]
    A module name. Valid options: django_orm, httplib, memcache, pymongo, redis, sqlalchemy.
    history
    Introduced in version 0.4.2.
    example
    oboe.config['inst_enabled']['memcache'] = False

    oboe.config[‘sanitize_sql’]

    config option
    oboe.config[‘sanitize_sql’]
    description
    Do not collect or report sql query parameters to TraceView.
    history
    Introduced in version 1.4.0.
    example
    oboe.config['sanitize_sql'] = True

    oboe.config[‘inst’][‘django_orm’]

    config option
    oboe.config[‘inst’][‘django_orm’]
    description
    Configuration options for the django ORM instrumentation.
    parameters
    [‘collect_backtraces’]
    Default: True.
    history
    Introduced with [‘collect_backtraces’] in version 1.5.0.
    example
    oboe.config['inst']['django_orm']['collect_backtraces'] = True

    oboe.config[‘inst’][‘httplib’]

    config option
    oboe.config[‘inst’][‘httplib’]
    description
    Configuration options for httplib instrumentation.
    parameters
    [‘collect_backtraces’]
    Default: True.
    history
    Introduced with [‘collect_backtraces’] in version 1.5.0.
    example
    oboe.config['inst']['httplib']['collect_backtraces'] = True

    oboe.config[‘inst’][‘memcache’]

    config option
    oboe.config[‘inst’][‘memcache’]
    description
    Configuration options for memcache instrumentation.
    parameters
    [‘collect_backtraces’]
    Default: False.
    history
    Introduced with [‘collect_backtraces’] in version 1.5.0.
    example
    oboe.config['inst']['memcache']['collect_backtraces'] = True

    oboe.config[‘inst’][‘pymongo’]

    config option
    oboe.config[‘inst’][‘pymongo’]
    description
    Configuration options for pymongo instrumentation.
    parameters
    [‘collect_backtraces’]
    Default: True.
    history
    Introduced with [‘collect_backtraces’] in version 1.5.0.
    example
    oboe.config['inst']['pymongo']['collect_backtraces'] = True

    oboe.config[‘inst’][‘redis’]

    config option
    oboe.config[‘inst’][‘redis’]
    description
    Configuration options for redis instrumentation.
    parameters
    [‘collect_backtraces’]
    Default: False.
    history
    Introduced with [‘collect_backtraces’] in version 1.5.0.
    example
    oboe.config['inst']['redis']['collect_backtraces'] = True

    oboe.config[‘inst’][‘sqlalchemy’]

    config option
    oboe.config[‘inst’][‘sqlalchemy’]
    description
    Configuration options for sqlalchemy instrumentation.
    parameters
    [‘collect_backtraces’]
    Default: True.
    history
    Introduced with [‘collect_backtraces’] in version 1.5.0.
    example
    oboe.config['inst']['sqlalchemy']['collect_backtraces'] = True

    Customizing instrumentation

    Customization involves adding hooks from our public API to your code so that you can to take advantage of additional filtering capabilities on the dashboard, change how requests are traced, or capture additional information during the trace.

      Python custom layers

      Python instrumentation creates some layers by default, e.g., ‘django’, ‘wsgi’, ‘pylibmc’, which more or less map to the components you’ll find in the support matrix. If these layers don’t provide enough visibility, you can further divide your code into sub-layers. How you segment your application into layers is entirely up to you. For example the out-of-box instrumentation might not recognize calls to external processes; or, maybe there’s a subsection of your code that functions as a discrete service without being external to the process. In any case, python instrumentation offers two facilities for manually creating layers: one is a decorator, for use when you want to represent a particular function as a layer; this other is an API method that’s better used when the block of code isn’t neatly contained.

      Custom layer using the logging decorator

      Follow the link to the API documentation for complete syntax and usage information.

      1 @oboe.log_method('slow_thing')
      2     def my_possibly_slow_method(...)

      Custom layer using the logging API method

      There’s a convention that must be followed when defining a new layer: the logging call which marks the entry into a new layer must be labeled ‘entry’; the logging call which marks the exit from the user-defined layer must be labeled ‘exit’. Follow the link to the API documentation for complete syntax and usage information.

      1 def myLogicallySeparateTask(...):
      2     oboe.log('entry', 'serviceX', {'key':'value'})
      3     ... // do stuff
      4     oboe.log('exit', 'serviceX', {'key':'value'})

      Python function profiling

      For most situations where you want to analyze a specific region of code, it will be easiest and more informative to use our function profiling interface. The alternative is custom layers, which should generally be reserved for calls to external services that our instrumentation would not normally recognize as needing their own layer. A function profile enables you to drill down on the performance of a specific region of code within the language layer of your app, seeing how often they are hit, in what calls and traces, and how long they take. Python instrumentation offers two facilities for creating manually creating layers: one is a decorator, for use when you want to represent a particular function; this other is an API method that’s better used when the block of code isn’t neatly defined. Follow the link to the API documentation for complete syntax and usage information.

      Function profiling using a decorator

      1 @oboe.profile_function('_profile_name_', Store_return=False, Store_args=False)
      2 def my_function(...):

      Block profiling using a context manager

      1 with oboe.profile_block('_profile-name_'):
      2   ...

      Add info events

      There are two reasons you might want to create an info event: the first is to simply to attach any metadata that may be of interest to you during later analysis of a trace. The other reason is to take full advantage of TraceView filtering capability. In short, you can classify extents by attaching a pre-defined set of specified key/value pairs. Then from the dashboard, you can filter for traces with that extent type. See special interpretation, for more on this.

      To add info to an extent, call oboe.log anywhere within it, as shown. You may do this at multiple points if necessary. The information events are displayed on the raw span data tab on the trace details page. If all of the key/value pairs for special interpretation are present, the extent type is reflected in the span details tab. Follow the link to the API documentation for complete syntax and usage information.

      1 oboe.log('info', {'key':'value', None, 'key': 'value'})

      Python partitions

      With hundreds of traces flowing into your dashboard, you need a way to keep them organized. For this, TraceView provides ‘partitions’, a programmatic way of classifying traces. For example, you can place traces into the following classes: anonymous, authenticated, and administrator. On the dashboard, partitions appear as a filter on the same level as urls and controller/action pairs, so in this example you could choose to limit the display to just requests of authenticated users. Traces are not assigned to any partition by default, and you are free to partition requests however you need to feel that your app is logically arranged. To assign a trace to a partition, just fire an info event anywhere within the trace and attach the partition key/value pair. Partition names may have a maximum length of 40 characters, anything longer is truncated to 40. They’re also limited to alphanumeric characters, or underscores. Any characters that don’t fit this criteria are silently converted to underscores. Follow the link to the API documentation for complete syntax and usage information.

      1 oboe.log('info', None, {'Partition':'_partition-name_'})

      Python controllers filter

      Python instrumentation supports these frameworks. If we don’t support your framework, add controller/action information by reporting an info event with the ‘controller’ and ‘action’ keys. Even if your framework doesn’t have a conventional notion of controllers and actions, you can still take advantage of this filtering capability by simply attaching whatever values that make sense to you.

      1 oboe.log('info', {'Controller':'your_controller', 'Action':'do_something'})

      Report errors and exceptions

      Many types of errors are already caught, e.g., web server errors, or any exceptions that reach the django or WSGI top layers. Those that aren’t caught by instrumentation can be reported manually using either oboe.log_error or oboe.log_exception. Use the former for any error not necessarily resulting in an exception, use the latter inside of a handler to collect and report information about built-in exceptions. Despite the different applications, both report two key pieces of information—the type of error and an error message—which enable TraceView to classify the extent as an error extent. Error events are marked in the trace view 50x50_trans.png and corresponding information is shown in the errors panel. Follow the link to the API documentation for complete syntax and usage information.

      1 if call_method() == None:
      2   oboe.log_error('nothing was returned!')
      3 
      4 try:
      5   call_method()
      6 except Exception as e:
      7   oboe.log_exception()

      RUM for python

      Do you have an instrumented webserver? These instructions involve enabling RUM via your page template files. If apache or nginx sits in front of your app, you don’t need to enable RUM here. It’s already provided at the web server level via our auto-rum support.

      TraceView provides optional javascript-based client-side performance data via real user monitoring (RUM). In order to gather real user timing data from the browser, you’ll need to insert two scripts into your page templates. These are provided in django as custom template tags. For other web frameworks, use the provided helper functions which automatically generate the required javascript.

      1. Place oboe.rum_header just after any <meta> tags in your <head> block. It should be the first non-meta tag inside <head>.
      2. Place oboe.rum_footer immediately before the closing </body> tag.
      Embedding RUM js in django
       1 {% load oboe %} {# include our tag library then just reference them #}
       2 ...
       3 
       4 <head>
       5 <meta ... >
       6 {% oboe_rum_header %}
       7 ...
       8 </head>
       9 
      10 <body>
      11 ...
      12 {% oboe_rum_footer %}
      13 </body>
      Embedding RUM in other python web frameworks
       1 {# stash the tag strings in two variables for your templates, then reference them #}
       2 header_code = oboe.rum_header() 
       3 footer_code = oboe.rum_footer()
       4 
       5 <head>
       6 <meta ... >
       7 
       8 ...
       9 </head>
      10 
      11 <body>
      12 ...
      13 
      14 </body>