Make sure you’ve installed base packages before installing this instrumentation.

Don’t be alarmed by the amount of documentation on this page. 99% of you need only the section immediately following this one which is installing nginx using a package manager like apt or yum. This is the fastest and easiest way to get nginx with TraceView instrumentation. But, if you want to build nginx or .deb packages from source, you’ll find that here too.

Installing from packages

Instrumentation for nginx is provided as a module. Because nginx modules are compiled into the main server binary—nginx does not load modules from shared libraries—we provide modified versions of the default nginx packages for your distro. These packages can be installed using your package manager once you’ve installed our base packages. This is because when you ran our install script install_traceview.sh, one of the things it did was add an AppNeta repository and float it to the top of your sources list.

Installing from packages on rhel/centos

We have built nginx packages with TraceView instrumentation using the source rpms from the epel repository. You must have the epel repository enabled in order to install the dependencies required for nginx. On rhel you might need to install and enable the repo, on centos it’s installed and enabled by default.

# use the --enablerep option if you're on rhel
sudo yum [--enablerepo=epel] install nginx
sudo service nginx restart

Older than trusty, debian squeeze

A simple apt-get install grabs the instrumented build from the AppNeta repo.

# for nginx-full
sudo apt-get install nginx
# for nginx-light or nginx-extras
sudo apt-get install {nginx-light | nginx-extras} nginx
sudo service nginx restart

Trusty and newer, debian wheezy

These versions provide three nginx packages, nginx-light, nginx-full, and nginx-extras, each of which as a different combination of core and extra modules. All three are available with TraceView instrumentation. We build our instrumented nginx using the most recent nginx version available which is usually provided by the backports repo. If you’re on wheezy stable, you’ll first need to add the this repo. On ubuntu trusty, trusty-backports is enabled by default.

# for nginx-full
sudo apt-get install nginx
# for nginx-light or nginx-extras
sudo apt-get install {nginx-light | nginx-extras} nginx
sudo service nginx restart

Installing from dotdeb packages

We’ve provided an instrumented build using dotdeb sources too. To get it on a debian distro, you’ll need to append the dotdeb repo to the AppNeta source list. Then follow the install instructions from one of the previous sections.

# for debian distros
sudo nano /etc/apt/sources.list.d/appneta.list
# append the dotdeb repo to the existing deb command
deb http://apt.tv.solarwinds.com/<access_key> wheezy main dotdeb 

To get it on ubuntu distros, modify the AppNeta sources list as shown below, and then follow the install instructions from one of the previous ubuntu sections. Note this dotdeb disclaimer.

# for ubuntu distros
# original source list
deb http://apt.tv.solarwinds.com/<access_key> precise main
# modified source list
deb http://apt.tv.solarwinds.com/<access_key> wheezy dotdeb

Building your own .deb package

If you are using a third-party apt repository then you can build your own oboe-enabled .deb package based on that repository’s source packages. We provide a build script which automates the process of installing apt build dependencies, downloading the source package, adding the traceview instrumentation and pagespeed modules to the package configuration, and building a binary package from source. This method only works if you are building nginx version 0.8 or above.

  1. First, you’ll need the source to our nginx oboe instrumentation module. nginx doesn’t support dynamically loading modules from shared libraries, so all modules must be configured and built into the server binary.

    # for 64-bit systems
    wget https://files.tv.solarwinds.com/src/nginx_oboe-latest.x86_64.tar.gz -P /tmp
    # for 32-bit systems
    wget https://files.tv.solarwinds.com/src/nginx_oboe-latest.i686.tar.gz -P /tmp
    
  2. Extract the archive. In this example we use the 64-bit nginx_oboe option, which results in the directory /tmp/nginx_oboe.

    tar -xzvf /tmp/nginx_oboe-latest.x86_64.tar.gz -C /tmp
    
  3. Pre-install these known dependencies which are not covered in the build script.

    sudo apt-get install libdistro-info-perl libgd2-noxpm
    
  4. Download and execute the build script.

    wget https://files.tv.solarwinds.com/src/build_nginx_deb.sh -P /tmp
    sh /tmp/build_nginx_deb.sh /tmp/nginx_oboe
    
  5. The script first attempts to install build dependencies. If running as a non-root user, you’ll be prompted to enter your sudo password. After this stage, if the script exposes any additional dependencies, it will exit. In this case just apt-get install the required packages and relaunch.

    === Welcome to the Tracelytics build script for Nginx!
    === This script will help you build .deb packages for Nginx which support
    === tracing HTTP requests using the Tracelytics liboboe instrumentation library.
    ...
    [sudo] password for <username>:
    
  6. Ignore this debchange warning.

    === Adding Tracelytics to the Debian changelog and bumping version...
    debchange warning: Recognised distributions are: unstable, testing, stable,
    oldstable, experimental, {testing-,stable-,oldstable-,}proposed-updates,
    {testing,stable,oldstable}-security, wheezy-backports and UNRELEASED.
    Using your request anyway.
    debchange: Did you see that warning?  Press RETURN to continue...
    
  7. Once built, the new .deb packages will be the same version as in your apt repository, but with +tracelytics inserted into their names.

    === dpkg-buildpackage has completed.  The built .deb packages are:
    ===   ./nginx_1.2.1-2.2+wheezy3+tracelytics201504211445wheezy_all.deb
    ./nginx-common_1.2.1-2.2+wheezy3+tracelytics201504211445wheezy_all.deb
    ./nginx-doc_1.2.1-2.2+wheezy3+tracelytics201504211445wheezy_all.deb
    ./nginx-extras_1.2.1-2.2+wheezy3+tracelytics201504211445wheezy_amd64.deb
    ./nginx-extras-dbg_1.2.1-2.2+wheezy3+tracelytics201504211445wheezy_amd64.deb
    ./nginx-full_1.2.1-2.2+wheezy3+tracelytics201504211445wheezy_amd64.deb
    ./nginx-full-dbg_1.2.1-2.2+wheezy3+tracelytics201504211445wheezy_amd64.deb
    ./nginx-light_1.2.1-2.2+wheezy3+tracelytics201504211445wheezy_amd64.deb
    ./nginx-light-dbg_1.2.1-2.2+wheezy3+tracelytics201504211445wheezy_amd64.deb
    ./nginx-naxsi_1.2.1-2.2+wheezy3+tracelytics201504211445wheezy_amd64.deb
    ./nginx-naxsi-dbg_1.2.1-2.2+wheezy3+tracelytics201504211445wheezy_amd64.deb
    ./nginx-naxsi-ui_1.2.1-2.2+wheezy3+tracelytics201504211445wheezy_all.deb.
    ===
    ...
    
  8. Install nginx-light, nginx-full, or nginx-extras. In this example we chose nginx-light.

    sudo dpkg -i nginx-common_*.deb nginx-light_*.deb nginx_*.deb
    
  9. Verify that nginx was installed with the nginx_oboe and pagespeed modules. nginx_oboe is the TraceView instrumentation and pagespeed was included in the build so that you can benefit from auto-rum.

    nginx -V
    nginx version: nginx/1.2.1
    TLS SNI support enabled
    configure arguments: --with-debug --add-module=/tmp/nginx_oboe --add-module=/tmp/nginx_oboe/psol/ngx_pagespeed --prefix=/etc/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-log-path=/var/log/nginx/access.log --http-proxy-temp-path=/var/lib/nginx/proxy --lock-path=/var/lock/nginx.lock --pid-path=/var/run/nginx.pid --with-pcre-jit --with-http_gzip_static_module --with-http_ssl_module --with-ipv6 --without-http_browser_module --without-http_geo_module --without-http_limit_req_module --without-http_limit_zone_module --without-http_memcached_module --without-http_referer_module --without-http_scgi_module --without-http_split_clients_module --with-http_stub_status_module --without-http_ssi_module --without-http_userid_module --without-http_uwsgi_module --add-module=/root/nginx-1.2.1/debian/modules/nginx-echo
    

Building from source

Do you build nginx from source so that you can use your own custom set of modules? If so then here is what you need to add TraceView instrumentation to your build.

Building nginx with nginx_oboe

Building nginx from source follows a configure, make, make install sequence, which is common in linux. If you need a refresher, check out this old article for some additional context on the commands that follow.

  1. Install the packages required to build nginx.

    # installing required packages on debian/ubuntu
    sudo apt-get install build-essential libpcre3-dev libssl-dev
    # installing required packages on rhel/centos
    sudo yum groupinstall 'Development Tools'
    sudo yum install pcre-devel openssl-devel
    
  2. Download the source code for nginx to a temporary location.

    wget http://nginx.org/download/nginx-1.6.3.tar.gz -P /tmp
    
  3. Download the source code for our nginx_oboe instrumentation module to a temporary location. The module comes in 32- and 64-bit versions.

    # for 64-bit systems
    wget https://files.tv.solarwinds.com/src/nginx_oboe-latest.x86_64.tar.gz -P /tmp
    # for 32-bit systems
    wget https://files.tv.solarwinds.com/src/nginx_oboe-latest.i686.tar.gz -P /tmp
    
  4. Extract both archives. In this example we use the 64-bit nginx_oboe option, which results in the directories /tmp/nginx-1.6.3 and /tmp/nginx_oboe.

    tar -xzvf /tmp/nginx-1.6.3.tar.gz -C /tmp
    tar -xzvf /tmp/nginx_oboe-latest.x86_64.tar.gz -C /tmp
    
  5. From the nginx source directory, and as root, execute the configure script with the add-module and debug options as shown. The command is slightly different based on the nginx version you’re compiling. For nginx version 0.8 and newer, we want you to include the pagespeed module so you can benefit from auto-rum. Be sure to tack on any other desired config options.

    cd /tmp/nginx-1.6.3
    sudo su
    #nginx version 0.8 and newer
    ./configure --with-http_ssl_module --add-module=../nginx_oboe --add-module=/tmp/nginx_oboe/psol/ngx_pagespeed/ --with-debug
    #nginx versions older than 0.8
    ./configure --with-http_ssl_module --add-module=../nginx_oboe --with-debug
    
  6. The configure script results in a makefile. From inside the nginx source directory, execute ‘make && make install’. This command chain runs the makefile and then copies the resulting files to the appropriate locations.

    make && sudo make install
    
  7. Download and install an init script if you don’t already have one, we suggest using one of the many that nginx offers.

  8. Start nginx and verify that it’s running.

    # start nginx
    service nginx start
    Starting nginx: nginx.
    # verify that nginx is running
    ps aux | grep nginx | grep -v grep
    root     23021  0.0  0.2  51868  1308 ?        Ss   13:15   0:00 nginx: master process /opt/nginx/sbin/nginx
    nginx    23022  0.0  0.3  52276  1984 ?        S    13:15   0:00 nginx: worker process
    
  9. Manually place the liboboe and autorum configuration files, and at minimum set the nginx tracing mode in nginx.conf. See configuring nginx instrumentation.

Building with passenger and nginx_oboe

To build nginx with the passenger module, you can use the passenger-install-nginx-module script as usual and provide the TraceView instrumentation module as an extra parameter. The following instructions compile passenger for ruby apps. You should perform these steps as root because you’ll be instructed to install the binary after compilation.

  1. Install the packages required to build nginx. The passenger installer might expose dependencies not covered here. In this case the script will instruct you on how to install the required packages and relaunch the script.

    # installing required packages on debian/ubuntu
    sudo apt-get install build-essential ruby-dev libcurl4-openssl-dev
    # installing required packages on rhel/centos
    sudo yum groupinstall 'Development Tools'
    sudo yum ruby-devel openssl-devel libcurl-devel
    
  2. The passenger installer also depends on the rack gem.

    /usr/bin/gem install rack
    
  3. On centos/rhel only, you’ll need to install the rack gem separately.

    # the passenger installer wants you to install rake via your epel repo
    yum install [--enablerepo=remi] rubygem-rake
    # if that's not available just use rubygem
    /usr/bin/gem install rake
    
  4. Download the phusion passenger module, and the nginx source files to a temporary location.

    wget http://nginx.org/download/nginx-1.6.3.tar.gz -P /tmp
    wget http://s3.amazonaws.com/phusion-passenger/releases/passenger-4.0.59.tar.gz -P /tmp
    
  5. Download the source code for our nginx_oboe instrumentation module to a temporary location. The module comes in 32- and 64-bit versions.

    # for 64-bit systems
    wget https://files.tv.solarwinds.com/src/nginx_oboe-latest.x86_64.tar.gz -P /tmp
    # for 32-bit systems
    wget https://files.tv.solarwinds.com/src/nginx_oboe-latest.i686.tar.gz -P /tmp
    
  6. Extract each of the archives to the same location, in this example /tmp, which results in the the following directories: /tmp/nginx-1.6.3, /tmp/nginx_oboe, and /tmp/passenger-4.0.59.

    tar -xzvf /tmp/nginx_oboe-latest.x86_64.tar.gz -C /tmp
    tar -xzvf /tmp/nginx-1.6.3.tar.gz -C /tmp
    tar -xzvf /tmp/passenger-4.0.59.tar.gz -C /tmp
    
  7. Run the phusion passenger installer and follow the on-screen instructions.

    /tmp/passenger-4.0.59/bin/passenger-install-nginx-module
    Welcome to the Phusion Passenger Nginx module installer, v4.0.59.
    
    This installer will guide you through the entire installation process. It
    shouldn't take more than 5 minutes in total.
    ...
    Press Enter to continue, or Ctrl-C to abort.
    
  8. When prompted to for standard or custom installation, enter ‘2’.

    Automatically download and install Nginx?
    ...
    Do you want this installer to download, compile and install Nginx for you?
     1. Yes: download, compile and install Nginx for me...
     2. No: I want to customize my Nginx installation...
    ...
    Enter your choice (1 or 2) or press Ctrl-C to abort: 2
    
  9. This example extracted the nginx source to /tmp/nginx-1.6.3, so we provide that when prompted.

    Where is your Nginx source code located?
    Please specify the directory: /tmp/nginx-1.6.3
    
  10. Keep the default installation directory by just pressing enter.

    Where do you want to install Nginx to?
    Please specify a prefix directory [/opt/nginx]:
    
  11. Add extra arguments add for nginx_oboe and pagespeed when prompted.

    Extra Nginx configure options
    
    If you want to pass extra arguments to the Nginx 'configure' script, then
    please specify them. If not, then specify nothing and press Enter.
    ...
    Extra arguments to pass to configure script: --add-module=/tmp/nginx_oboe --add-module=/tmp/nginx_oboe/psol/ngx_pagespeed/ --with-debug
    
  12. When the script completes verify that nginx was complied with the right modules.

    nginx version: nginx/1.6.3
    built by gcc 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5)
    TLS SNI support enabled
    configure arguments: --with-http_ssl_module --with-http_gzip_static_module --with-http_stub_status_module --with-cc-opt=-Wno-error --add-module=/tmp/passenger-4.0.59/ext/nginx --add-module=/tmp/nginx_oboe --add-module=/tmp/nginx_oboe/psol/ngx_pagespeed/ --with-debug
    
  13. Download and install an init script if you don’t already have one, we suggest using one of the many that nginx offers.

    # start nginx
    service nginx start
    Starting nginx: nginx.
    # verify that nginx is running
    ps aux | grep nginx | grep -v grep
    root     23021  0.0  0.2  51868  1308 ?        Ss   13:15   0:00 nginx: master process /opt/nginx/sbin/nginx
    nginx    23022  0.0  0.3  52276  1984 ?        S    13:15   0:00 nginx: worker process
    

Building with a fabric task

If you use Fabric for Python, a build task is available thanks to user imlucas. Note that you’ll still need to configure nginx (below). Add or include the following in your fabfile.py file changing the value of NGINX_VERSION as desired and of ARCH to ‘i686’ for 32-bit systems.

def install_nginx():
    """Install nginx from source with the oboe and oboe_ps modules."""
    NGINX_VERSION = "1.6.0"
    ARCH = "x86_64"
    sudo("apt-get install -y build-essential zlib1g-dev libpcre3 libpcre3-dev libgd2-noxpm-dev libgeoip-dev")
    run("curl -O http://nginx.org/download/nginx-%s.tar.gz" % NGINX_VERSION)
    run("tar -xvzf nginx-%s.tar.gz" % NGINX_VERSION)
    run("wget https://files.tv.solarwinds.com/src/nginx_oboe-latest.%s.tar.gz" % ARCH)
    run("tar -xvzf nginx_oboe-latest.tar.gz")
  
    with cd("nginx-%s" % NGINX_VERSION):
        run("./configure --prefix=/etc/nginx --add-module=../nginx_oboe/ --add-module=../nginx_oboe/psol/ngx_pagespeed/ --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-log-path=/var/log/nginx/access.log --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --lock-path=/var/lock/nginx.lock --pid-path=/var/run/nginx.pid --with-debug --with-http_addition_module --with-http_dav_module --with-http_geoip_module --with-http_gzip_static_module --with-http_image_filter_module --with-http_realip_module --with-http_stub_status_module --with-http_ssl_module --with-http_sub_module --with-http_xslt_module --with-ipv6 --with-sha1=/usr/include/openssl --with-md5=/usr/include/openssl --with-mail --with-mail_ssl_module")
        sudo("make && make install")

Upgrading instrumentation

redhat/centos

  1. To check for updates to our instrumented nginx package run:

    yum info nginx
    
  2. Pending releases will be listed under ‘available packages’. If you don’t see this you’re all up to date! If you there is a new version available upgrade it by running:

    yum -y update nginx
    

debian/ubuntu

  1. To check for updates to our instrumented nginx package run:

    apt-get update
    apt-cache policy nginx
    
  2. If the displayed Latest Version matches the current Candidate Version you’re all up to date! If there is a new version available it can be upgraded by running:

    apt-get install --only-upgrade nginx
    

Configuring instrumentation

When you install instrumented nginx from packages, the liboboe and autorum configuration files are installed for you. When you install instrumented nginx from source, you need to place these files manually and edit your nginx.conf files accordingly. This means that at minimum you must complete set the nginx tracing mode.

oboe.conf
Determines when traces should be initiated for incoming requests.
autorum.conf
There are no user-configurable directives in the autorum.conf.
autorum_server.conf
The autorum_server.conf is not needed for normal auto-rum operation, but it can be used to customize the operation of the pagespeed subsystem.

As of 1.3.0 autorum_default_conf is deprecated and any references to it should be removed from your nginx configuration.

Set the tracing mode

If you built instrumented nginx from source, you’ll need to add a oboe_tracing_mode directive to the http block of nginx.conf. You can do this by either copy/pasting the entire contents of oboe.conf, or by adding an include directive as shown.

su -
mkdir /usr/local/nginx/conf.d
# assuming nginx_oboe was extracted to /tmp and the default nginx installation directory
cp /tmp/nginx_oboe/oboe.conf /usr/local/nginx/conf.d
nano /usr/local/nginx/conf/nginx.conf
# add this line to the bottom of the http block
include /usr/local/nginx/conf.d/*.conf;

Change the tracing mode

You can change the nginx tracing mode by editing the oboe_tracing_mode directive in oboe.conf. The tracing modes are ‘always’, ‘through’, and ‘never’. If you installed instrumented nginx from packages, you’ll find in /etc/nginx/conf.d/oboe.conf.

always
Initiate new traces. This is the default.
through
Continue traces from an upstream source, do not initiate new ones.
never
Do not allow traces through and do not initiate traces.

Disable tracing for static files

Tracing requests for static files will likely not be interesting and may skew the averages in certain displays of tracing data. For this reason, many users choose to exclude requests for static files from tracing. Here’s a simple config snippet that disables tracing for requests to most static file types. Append it to nginx.conf.

location ~* ^.+\.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|pdf|txt|tar|wav|bmp|rtf|js|flv|swf|ttf|woff|svg)$
{
   oboe_tracing_mode never;
}

Tracing and rewrites

The configuration inheritance model for nginx is such that rewrites effectively clear configuration associated with the ‘from’ location, and only respect configuration at the ‘to’ location. That means that tracing configuration directives are sensitive to location rewrites. This has led to some confusion as to why oboe_tracing_mode appears to not be respected in certain configuration scenarios. For more information see this blog post by martin fjord.

Enable real-user monitoring

auto-rum is a switch provided on a per-app basis that enables you to turn rum on and off without having to change your nginx config and reload. When toggled, the TraceView dashboard sends a signal to instrumented nginx based on directives in nginx.conf.

  • If you installed instrumented nginx from packages double-check that auto-rum is supported on your platform and nginx version. If it is, then simply enable rum by checking a box on the real-user monitoring page.

  • If auto-rum isn’t supported on your platform or nginx version, then this section isn’t for you. rum can’t be implemented at the web server level and instead you’ll have to manually add some javascript to your page templates.

  • If you have a custom nginx.conf, add the required configuration directives so that nginx can interface with the auto-rum switch in the TraceView dashboard. Do this either by copy/pasting the entire contents of autorum.conf into the http block of nginx.conf, or copying autorum.conf to the conf.d installation sub-directory.

  • If you built instrumented nginx from source, and you’re on a platform and nginx version that supports auto-rum, then you’ll need make a few configuration changes before you can use the switch in the TraceView dashboard.

    1. Add the required configuration directives so that nginx can interface with the auto-rum switch in the TraceView dashboard. Do this either by copy/pasting the entire contents of autorum.conf into the http block of nginx.conf, or copying autorum.conf to the conf.d installation sub-directory.
    2. (Optional) autorum_server.conf contains directives to configure virtual pages that report on internal statistics or support other functionality within the pagespeed sub-system. If you need these enabled then the following directives need to be included in or copied into each server block that you want to have serving these special pages. You won’t normally want the statistics pages to be seen by the public so they are restricted to local users by default.
    3. Reload the nginx configuration.
    4. Once nginx is configured you can enable rum via the real-user monitoring page in the TraceView dashboard.
  • If auto-rum isn’t working as intended then you may find the debug log messages helpful. These are controlled by the nginx error_log configuration directive, and the output goes to the nginx error log. Look for any references to ‘Pagespeed traceview_rum’ and ‘TraceviewRumFilter’. See the nginx core functionality documentation for more details about these directives and its debugging log for suggestions on its use.

  • A log level of at least info error_log /path/to/log info records when auto-rum is enabled and disabled.
  • A level of at least warn error_log /path/to/log warn warns of problems in the generated html that prevent auto-rum from functioning correctly.
  • A level of at least debug error_log /path/to/log debug records critical steps in the auto-rum processing.