Graphing Running Averages in RRDTool

There are a number of times where the data collected may appear so erratic that it is difficult to identify any trends.  While use of a VDEF can provide a gross average of a data set, it doesn’t provide the utility of a true running average.  The running average can provide a consistent calculation regardless as to the time scale displayed in the graph.  Fortunately, RRDTool provides a flexible means via its TREND operator for calculating a running average to address this need.

To begin, setup a chart that graphs the desired data in a typical manner before adding in the running average.  The following graph command creates a somewhat typical visualization of the last day’s network traffic as illustrated by the following graph:

rrdtool graph example.png \
  --title "Network Traffic" \
  --width=500 \
  --slope-mode \
  --end=now \
  --start=end-24h \
  --base=1000 \
  DEF:bytesIn=network.rrd:bytesIn:AVERAGE \
  DEF:bytesOut=network.rrd:bytesOut:AVERAGE \
  CDEF:bpsIn=bytesIn,8,* \
  CDEF:bpsOut=bytesOut,8,* \
  CDEF:bpsOutNeg=bpsOut,-1,* \
  VDEF:bpsInTot=bpsIn,TOTAL \
  VDEF:bpsOutTot=bpsOut,TOTAL \
  COMMENT:"              current     min        max      total\n" \
  AREA:bpsIn#FF880044:"Bits/s In " \
  GPRINT:bpsIn:LAST:"%6.1lf%s\t" \
  GPRINT:bpsIn:MIN:"%6.1lf%s\t" \
  GPRINT:bpsIn:MAX:"%6.1lf%s\t" \
  GPRINT:bpsInTot:"%6.1lf%s\n" \
  LINE1:bpsIn#FF8800CC \
  AREA:bpsOutNeg#44C80044:"Bits/s Out" \
  GPRINT:bpsOut:LAST:"%6.1lf%s\t" \
  GPRINT:bpsOut:MIN:"%6.1lf%s\t" \
  GPRINT:bpsOut:MAX:"%6.1lf%s\t" \
  GPRINT:bpsOutTot:"%6.1lf%s\n" \
  LINE1:bpsOutNeg#44C800CC \
  HRULE:0#000000

With the basic graph setup, adding the elements necessary for the running average can now begin.  In this example, it appears there is a roughly hourly cycle (Time Machine backups) that defines the period so a reasonable start is to work with a minimum average calculated over 60 minutes.

First, additional CDEF statements need to be declared applying the TREND operator to the desired data.  The format for this operation is straightforward:

CDEF:label=data_source,time_span,TREND

In this case, two new data sources are defined and are given the labels bspInTrend and bpsOutTrendNeg.  The data sources for these are the previously declared bpsIn and bpsOutNeg and the time span for them is 1 hour (3600 seconds).

  CDEF:bpsInTrend=bpsIn,3600,TREND \
  CDEF:bpsOutTrendNeg=bpsOutNeg,3600,TREND \

With the running averages now defined, it is possible to graph the data.  For the purposes of this example, the running average is graphed as a “shadow” on top of the base data.

  LINE2:bpsInTrend#000000CC \
  LINE2:bpsOutTrendNeg#000000CC \

The graph shown below illustrates the new running averages.

The running averages are now shown and it illustrates that despite the hourly peaks, the overall trend is fairly flat with the slight exception around 00:15.  However, the running averages also show a gap at the beginning of the graph. This gap is because it takes an hour of data before the hourly running average can be computed.  While this may be acceptable, it doesn’t look polished.

A minor tweak of the original DEF statements can provide “padding” necessary to eliminate the displayed gap in the running average lines.  This can be accomplished by extending the start time of the data sources by at least the same duration specified in the running averages.  Note that the display area is not affected by altering the start time of the DEF statements — that is controlled by the --start and --end options.  For the example data, the DEF statements need to extend the range of the data set by 1 hour so as to encompass 25 hours instead of the display area’s 24 hours:

  DEF:bytesIn=network.rrd:bytesIn:AVERAGE:start=end-25h \
  DEF:bytesOut=network.rrd:bytesOut:AVERAGE:start=end-25h \

Putting it together, the following command will pull in all the data necessary to calculate a running average for the extent of the display area, calculate the running averages, and display the results:

rrdtool graph example.png \
  --title "Network Traffic" \
  --width=500 \
  --slope-mode \
  --end=now \
  --start=end-24h \
  --base=1000 \
  DEF:bytesIn=network.rrd:bytesIn:AVERAGE:start=end-25h \
  DEF:bytesOut=network.rrd:bytesOut:AVERAGE:start=end-25h \
  CDEF:bpsIn=bytesIn,8,* \
  CDEF:bpsOut=bytesOut,8,* \
  CDEF:bpsOutNeg=bpsOut,-1,* \
  CDEF:bpsInTrend=bpsIn,3600,TREND \
  CDEF:bpsOutTrendNeg=bpsOutNeg,3600,TREND \
  VDEF:bpsInTot=bpsIn,TOTAL \
  VDEF:bpsOutTot=bpsOut,TOTAL \
  COMMENT:"              current     min        max      total\n" \
  AREA:bpsIn#FF880044:"Bits/s In " \
  GPRINT:bpsIn:LAST:"%6.1lf%s\t" \
  GPRINT:bpsIn:MIN:"%6.1lf%s\t" \
  GPRINT:bpsIn:MAX:"%6.1lf%s\t" \
  GPRINT:bpsInTot:"%6.1lf%s\n" \
  LINE1:bpsIn#FF8800CC \
  AREA:bpsOutNeg#44C80044:"Bits/s Out" \
  GPRINT:bpsOut:LAST:"%6.1lf%s\t" \
  GPRINT:bpsOut:MIN:"%6.1lf%s\t" \
  GPRINT:bpsOut:MAX:"%6.1lf%s\t" \
  GPRINT:bpsOutTot:"%6.1lf%s\n" \
  LINE1:bpsOutNeg#44C800CC \
  LINE2:bpsInTrend#000000CC \
  LINE2:bpsOutTrendNeg#000000CC \
  HRULE:0#000000

 

Graphing Time Shifted Data in RRDTool

While visualizing performance or activity data using a well-designed chart can be useful, it is frequently desired to be able to compare recent performance against an earlier period of time.  Comparisons against historic data help identify changes in trends or other anomalous performance behaviors.  In some cases, this can be done simply by extending the time span of the graph.  However, expansion of the graphed time span can lead to either an unacceptable loss of resolution or an unacceptably large image.  Fortunately, RRDTool can make this type of comparison chart fairly easily without resulting in a loss of fidelity or unwieldy image size.

To begin, it helps to setup a chart that graphs the desired data in a typical manner before adding in the time shifted overlays.  The following graph command creates a somewhat typical visualization of the last hour’s network traffic as illustrated by Example 1 graph:

rrdtool graph example.png \
  --title "Network Traffic" \
  --width=500 \
  --slope-mode \
  --end=now \
  --start=end-1h \
  --base=1000 \
  DEF:bytesIn=network.rrd:bytesIn:AVERAGE \
  DEF:bytesOut=network.rrd:bytesOut:AVERAGE \
  CDEF:bpsIn=bytesIn,8,* \
  CDEF:bpsOut=bytesOut,8,* \
  CDEF:bpsOutNeg=bpsOut,-1,* \
  VDEF:bpsInTot=bpsIn,TOTAL \
  VDEF:bpsOutTot=bpsOut,TOTAL \
  COMMENT:"            current     min        max      total\n" \
  AREA:bpsIn#FF880044:"Bits/s In " \
  GPRINT:bpsIn:LAST:"%6.1lf%s\t" \
  GPRINT:bpsIn:MIN:"%6.1lf%s\t" \
  GPRINT:bpsIn:MAX:"%6.1lf%s\t" \
  GPRINT:bpsInTot:"%6.1lf%s\n" \
  LINE1:bpsIn#FF8800CC \
  AREA:bpsOutNeg#44C80044:"Bits/s Out" \
  GPRINT:bpsOut:LAST:"%6.1lf%s\t" \
  GPRINT:bpsOut:MIN:"%6.1lf%s\t" \
  GPRINT:bpsOut:MAX:"%6.1lf%s\t" \
  GPRINT:bpsOutTot:"%6.1lf%s\n" \
  LINE1:bpsOutNeg#44C800CC \
  HRULE:0#000000

With the basic chart setup as desired, now the command can be modified to support a time shifted overlay.

First, additional DEF statements need to be defined for time range for the source data to encompass the time span desired for comparison.  Note that this is not the same as the graph’s display range; those options (--start and --end) should remain unchanged.  The override start time should extend the time frame to the start of the time shifted data.  For example, the following would be suitable for comparison of the previous 1 hour:

  DEF:bytesInPrevHour=network.rrd:bytesIn:AVERAGE:start=end-2h \
  DEF:bytesOutPrevHour=network.rrd:bytesOut:AVERAGE:start=end-2h \

As another example, the following would be suitable for a comparison of the same hour, 1 day (24 hours) ago:

  DEF:bytesInPrevDay=network.rrd:bytesIn:AVERAGE:start=end-25h \
  DEF:bytesOutPrevDay=network.rrd:bytesOut:AVERAGE:start=end-25h \

Next, a new SHIFT statement needs to be introduced that shifts the data being displayed by the specified number of seconds.  The SHIFT statements must be specified after the DEF statements they are shifting.  The following statements would shift the newly inserted DEF statements by 1 hour (3600 seconds):

  SHIFT:bytesInPrevHour:3600 \
  SHIFT:bytesOutPrevHour:3600 \

Next, the CDEF and VDEF operations need to be supplied in order to transform the new data in a consistent manner with the original data.

  CDEF:bpsInPrev=bytesInPrevHour,8,* \
  CDEF:bpsOutPrev=bytesOutInPrevHour,8,* \
  CDEF:bpsOutPrevNeg=bpsOutPrev,-1,* \

Finally, the new elements are ready for graphing.  Care should be made to ensure the shifted data is displayed in a distinct manner from the current data so as to prevent visual confusion or obscuring the current data.

  LINE1:bpsInPrev#00000088 \ 
  LINE1:bpsOutPrevNeg#00000088 \

Pulling it all together, the following command will pull in the data from the extended time frame, shift it forward by the appropriate amount of time so that it aligns with the displayed time frame, perform the desired transformation options, and finally display the results:

rrdtool graph example.png \
  --title "Network Traffic" \
  --width=500 \
  --slope-mode \
  --end=now \
  --start=end-1h \
  --base=1000 \
  DEF:bytesIn=network.rrd:bytesIn:AVERAGE \
  DEF:bytesOut=network.rrd:bytesOut:AVERAGE \
  DEF:bytesInPrevHour=network.rrd:bytesIn:AVERAGE:start=end-2h \
  DEF:bytesOutPrevHour=network.rrd:bytesOut:AVERAGE:start=end-2h \
  SHIFT:bytesInPrevHour:3600 \
  SHIFT:bytesOutPrevHour:3600 \
  CDEF:bpsIn=bytesIn,8,* \
  CDEF:bpsOut=bytesOut,8,* \
  CDEF:bpsOutNeg=bpsOut,-1,* \
  CDEF:bpsInPrev=bytesInPrevHour,8,* \
  CDEF:bpsOutPrev=bytesOutInPrevHour,8,* \
  CDEF:bpsOutPrevNeg=bpsOutPrev,-1,* \
  VDEF:bpsInTot=bpsIn,TOTAL \
  VDEF:bpsOutTot=bpsOut,TOTAL \
  COMMENT:"            current     min        max      total\n" \
  AREA:bpsIn#FF880044:"Bits/s In " \
  GPRINT:bpsIn:LAST:"%6.1lf%s\t" \
  GPRINT:bpsIn:MIN:"%6.1lf%s\t" \
  GPRINT:bpsIn:MAX:"%6.1lf%s\t" \
  GPRINT:bpsInTot:"%6.1lf%s\n" \
  LINE1:bpsIn#FF8800CC \
  AREA:bpsOutNeg#44C80044:"Bits/s Out" \
  GPRINT:bpsOut:LAST:"%6.1lf%s\t" \
  GPRINT:bpsOut:MIN:"%6.1lf%s\t" \
  GPRINT:bpsOut:MAX:"%6.1lf%s\t" \
  GPRINT:bpsOutTot:"%6.1lf%s\n" \
  LINE1:bpsOutNeg#44C800CC \
  LINE1:bpsInPrev#00000088 \ 
  LINE1:bpsOutPrevNeg#00000088 \
  HRULE:0#000000

 

Advanced Color Graphing Techniques using RRDTool

Introduction

This section covers some of the more advanced uses of color in a RRDTool graph.  These techniques can not only help in providing an additional bit of polish to an otherwise ordinary graph, but can also be useful in clarifying interpretation and providing additional information.

Despite the availability of the techniques outlined below, it is still essential that proper color conventions and patterns be observed.  For example, red typically denotes “hot” in the context of temperature or “severe” in a notification/aberration detection.  Using red to denote a cool temperature on a temperature graph or a “normal” operating condition is very likely to lead to viewer confusion.  For more information, please be sure to consult a good reference on color theory.

Examples

Example 1

This is a simple example of a “stock” graph.  It uses no special color treatments but it is still able to clearly convey the necessary information.  In this case, it simply presents a graph of the system temperature.

rrdtool graph "Example 1 Colors.png" \
--start "end-48 hours" --end "12am Nov 1, 2009" \
--imgformat PNG --width 500 --height 120 \
--title "Example 1" \
--vertical-label "Temperature" \
DEF:temp=sysinfo.rrd:temperature:AVERAGE \
AREA:temp#FF0000

Example 1 Colors

Example 2

In this example, the use of an “alpha channel” in the color specification is introduced.  The easiest way to think of an alpha channel is as a transparency measure.  A alpha channel of “FF” would be completely opaque, while an alpha channel of “00″ would be completely transparent.  The alpha channel is specified in hexadecimal form at the end of the RGB color value.  In the example below, the color value of “FF000044″ has specified an alpha channel value of “44″.  If no alpha channel is specified, then a default value of “FF” (fully opaque) is used.

rrdtool graph "Example 2 Colors.png" \
--start "end-48 hours" --end "12am Nov 1, 2009" \
--imgformat PNG --width 500 --height 120 \
--title "Example 2" \
--vertical-label "Temperature" \
DEF:temp=sysinfo.rrd:temperature:AVERAGE \
AREA:temp#FF000044

Example 2 Colors

Example 3

This example illustrates the use of the “layer cake” effect.  This technique can help to provide additional context to a graph, as the colored layers can help clearly delineate when a system is operating within tolerances or not.  Thus, the health of the system can be determined at a glance and is not dependent on the viewer being intimate with the operational thresholds.  This example breaks down the temperature readings into four layers (cold, cool, warm, and hot) of 50 degrees each, but it would be a trivial extension to increase/decrease the number of layers.

The CDEF for the middle layers can be somewhat intimidating for those who are not experts in Reverse Polish Notation.  In this example, each layer relies on being stacked and so the appropriate calculation is determining the portion of the temperature (if any) that makes up the layer.  The following breakdown may help make it more palatable:

cool=temp,50,GT,temp,100,GT,50,temp,50,-,IF,UNKN,IF
if (temp > 50) then
  if (temp > 100) then
    cool = 50
  else
    cool = temp - 50
else
  cool = UNKN

As each layer is a maximum of 50 degrees, the trick is to determine how much (if any) of a layer falls within the designation.  If the actual temperature exceeds that of the layer, then simply use the maximum value (50).  If the temperature falls within the layer, then the value should be the temperature less the total of any previous bands.  If the temperature is less than the minimal temperature for this layer, then simply return the “unknown” value to prevent any graphing.

rrdtool graph "Example 3 Colors.png" \
--start "end-48 hours" --end "12am Dec 5, 2009" \
--imgformat PNG --width 500 --height 120 \
--title "Example 3" \
--vertical-label "Temperature" \
--lower-limit 0 --rigid \
DEF:temp=sysinfo.rrd:temperature:AVERAGE \
CDEF:cold=temp,50,LE,temp,50,IF \
CDEF:cool=temp,50,GT,temp,100,GT,50,temp,50,-,IF,UNKN,IF \
CDEF:warm=temp,100,GT,temp,150,GT,50,temp,100,-,IF,UNKN,IF \
CDEF:hot=temp,150,GT,temp,150,-,UNKN,IF \
AREA:cold#0000FFAA:cold:STACK \
AREA:cool#0000FF44:cool:STACK \
AREA:warm#FF000044:warm:STACK \
AREA:hot#FF0000AA:hot:STACK

Example 3 Colors

Example 4

There are several techniques for “feathering” the colors in a graph as shown in this example.  The technique illustrated in this example is suitable for a representing a single color palette with the gradient lightest at the top and darkest at the bottom.  It is achieved by simply overlaying the graph with the same color selection at selected proportions and relying on the alpha channel to “build up” as the layers overlap.  Care should be made when using this technique not too make the top layers so translucent they become difficult to discern.

This example simply maps the set of data values into sets for 1/4, 1/2 and 3/4 values and then overlays the original value graph.  Additional looks can also be achieved through the use of alternative data transformation maps/ratios.

rrdtool graph "Example 4 Colors.png" \
--start "end-48 hours" --end "12am Dec 5, 2009" \
--imgformat PNG --width 500 --height 120 \
--title "Example 4" \
--vertical-label "Temperature" \
--lower-limit 0 --rigid \
DEF:temp=sysinfo.rrd:temperature:AVERAGE \
CDEF:tier1=temp,4,/ \
CDEF:tier2=temp,2,/ \
CDEF:tier3=temp,4,/,3,* \
AREA:temp#FF000022: \
AREA:tier3#FF000022: \
AREA:tier2#FF000022: \
AREA:tier1#FF000022:

Example 4 Colors

Example 5

This example illustrates another method of “feathering” the colors in the graph.  In this case, the color gradient is lightest at the bottom and darkest at the top.  In order to achieve this, the value is simply divided up and then each layer is stacked on top of the other while steadily increasing the alpha channel value.

rrdtool graph "Example 5 Colors.png" \
--start "end-48 hours" --end "12am Dec 5, 2009" \
--imgformat PNG --width 500 --height 120 \
--title "Example 5" \
--vertical-label "Temperature" \
--lower-limit 0 --rigid \
DEF:temp=sysinfo.rrd:temperature:AVERAGE \
CDEF:tier=temp,4,/ \
AREA:tier#FF000022::STACK \
AREA:tier#FF000044::STACK \
AREA:tier#FF000066::STACK \
AREA:tier#FF000088::STACK

Example 5 Colors

Example 6

This example illustrates the use of highlights to clearly delineate the borders between stacked area graphs.  It allows the use of a softer color palette without having to resort to a clashing color scheme to define the borders.

The highlight lines should be specified after all the area graphs have been declared.  Each highlight should be specified in the same order as its corresponding area graph in order to ensure the proper color is “on top” should the data sets have any overlap.  It is typically easiest to maintain the same color scheme by using the same RGB value as the area graph but specifying high alpha channel value.

rrdtool graph "Example 6 Colors.png" \
--start "end-48 hours" --end "12am Jan 15, 2009" \
--imgformat PNG --width 500 --height 120 \
--title "Example 6" \
--vertical-label "Bytes" \
--lower-limit 0 --rigid \
DEF:disk1=sysinfo.rrd:disk_used:AVERAGE \
DEF:disk2=sysinfo.rrd:disk2_used:AVERAGE \
AREA:disk1#0000FF22:: \
AREA:disk2#00F00022::STACK \
LINE1:disk1#0000FFAA:"Disk 1" \
LINE1:disk2#00F000AA:"Disk 2":STACK

Example 6 Colors