Tag Archives: monitoring

Enable Remote Logging on Mac OS X

Posted on by
1

Background

It is often desirable to collect the system logs from various devices onto a central “logging host”.  This can simplify monitoring needs considerably as tools and scripts need only work on a single host.  It also provides some security benefits as hackers will have a harder time masking their activities if they do not have access to the logging host.

On Mac OS X, there is a logging daemon (the Unix standard syslogd utility) that can be used to write both local logs and receive logs from remote hosts.  Unfortunately, the ability to receive remote logs is turned off in the default installation.  However, it is a trivial task to enable this functionality.

Setup (for 10.5 and older systems)

  1. Login as administrator to the logging host
  2. Open a terminal session using the Terminal utility
  3. Navigate to the LaunchDaemons directory
    cd /System/Library/LaunchDaemons
  4. Edit the com.apple.syslogd.plist file
    sudo vi com.apple.syslogd.plist
  5. Remove the comment delimiters (<!-- and -->) surrounding the NetworkListener block and save the changes
    <!--
 <key>NetworkListener</key>
 <dict>
 <key>SockServiceName</key>
 <string>syslog</string>
 <key>SockType</key>
 <string>dgram</string>
 </dict>
-->
  6. Stop the currently running instance of the syslog daemon
    sudo launchctl unload /System/Library/LaunchDaemons/com.apple.syslogd.plist
  7. Restart the syslog daemon to pick up the changes in the LaunchDaemon configuration
    sudo launchctl load /System/Library/LaunchDaemons/com.apple.syslogd.plist

The logging facility on the log host should now be available for remote devices to use.

Setup (for 10.6 and later systems)

  1. Login as administrator to the logging host
  2. Open a terminal session using the Terminal utility
  3. Navigate to the LaunchDaemons directory
    cd /System/Library/LaunchDaemons
  4. Convert the com.apple.syslogd.plist to XML format
    sudo plutil -convert xml1 com.apple.syslogd.plist
  5. Edit the com.apple.syslogd.plist file
    sudo vi com.apple.syslogd.plist
  6. Add the NetworkListener dict entry after the end of the BSDSystemLogger dict entry and save the changes
    <key>NetworkListener</key>
<dict>
<key>SockServiceName</key>
<string>syslog</string>
<key>SockType</key>
<string>dgram</string>
</dict>
  7. Convert the com.apple.syslogd.plist file back to the binary format
    sudo plutil -convert binary1 com.apple.syslogd.plist
  8. Stop the currently running instance of the syslog daemon
    sudo launchctl unload /System/Library/LaunchDaemons/com.apple.syslogd.plist
  9. Restart the syslog daemon to pick up the changes in the LaunchDaemon configuration
    sudo launchctl load /System/Library/LaunchDaemons/com.apple.syslogd.plist

The logging facility on the log host should now be available for remote devices to use.

Setup (using the Property List Editor)

Apple provides the Property List Editor utility as part of its developer tools.  The developer tools are usually a separate install from the operating system and is frequently not installed on systems.  If it is present, then using the Property List Editor may be the most convenient means of modifying the com.apple.syslogd.plist file.

  1. Login as administrator to the logging host
  2. Launch the Property List Editor utility (typically located in /Developer/Applications/Utilities)
  3. Open the /System/Library/LaunchDaemons/com.apple.syslogd.plist file
  4. Select the “Sockets” dictionary entry and “Add Item” to create a new key in the “Sockets” dictionary
  5. Change the name of the item to “NetworkListener” and set its type to “Dictionary”
  6. Select the “NetworkListener” dictionary entry and “Add Item” to create a new key in the “NetworkListener” dictionary
  7. Change the name of the item to “SockServiceName”, its type to “String” and its value to “syslog”
  8. Select the “NetworkListener” dictionary entry and “Add Item” to create another new key in the “NetworkListener” dictionary
  9. Change the name of the item to “SockType”, its type to “String”, and its value to “dgram”
  10. Save the file and quit the Property List Editor utility.  The property list should resemble the following example:Example Property List for com.apple.syslogd.plist
  11. Open a terminal session using the Terminal utility
  12. Navigate to the LaunchDaemons directory
    cd /System/Library/LaunchDaemons
  13. Stop the currently running instance of the syslog daemon
    sudo launchctl unload /System/Library/LaunchDaemons/com.apple.syslogd.plist
  14. Restart the syslog daemon to pick up the changes in the LaunchDaemon configuration
    sudo launchctl load /System/Library/LaunchDaemons/com.apple.syslogd.plist

Creating a Round Robin Database with RRDTool

Posted on by
2

Introduction

There are two essential components that should be defined when creating a Round-Robin Database (RRD) using rrdtool:  the Data Source (DS) and the Round-Robin Archive (RRA).  The Data Source definition dictates how the data is entered into the database and can provide rules for the interpretation, limits, and expected frequency of updates.  The Round-Robin Archive definition dictates the storage of the data including how the data is consolidated, the time resolution, and the span of time the archive covers.  A single RRD instance must contain at least one Data Source and Round-Robin Archive, but may contain more.

Data Source Definition

The general format for a DS specification is:

DS:Label:Type:Heartbeat:Min:Max

Label is the name of the Data Source.  It may be from 1-19 characters long and consists of characters in the set [a-zA-Z0-9_].  The label should be descriptive of the data being collected.  Example labels include “temperature”, “BytesIn”, “bytes_out”, etc.

Type is one of rrdtool’s defined data types:  GAUGE, COUNTER, DERIVE, ABSOLUTE, or COMPUTE.

GAUGE is for storing simple measurements that are rate independent.  Data types such as temperatures, prices, or system load are suitable for this data type.

COUNTER is for storing continuously incrementing counters.  Data values input into this type of Data Source should never decrease except due to an overflow condition (which is handled by rrdtool).  Data of this type is stored as a per-second rate.  Example measurements that are suitable for this data type are velocity and throughput.

DERIVE stores the derivative of the line between the current and previous data points.  This is useful for converting an input of simple measurements into a rate.  For example, with an input of temperature readings, it will record the rate of temperature change instead of simply the raw temperature readings.

ABSOLUTE is used for counters which are reset upon reading.  This behavior is often found in fast-moving counters that would otherwise overflow frequently.  It is otherwise similar to the COUNTER type.

COMPUTE provides a means of applying a formula to other Data Sources in the RRD.  As such, the DS specification for a COMPUTE type is differs than the general form:

DS:Label:COMPUTE:rpn-expression

The COMPUTE type can be thought of as a “virtual” Data Source that calculates its value based off of the values of other Data Sources.

Heartbeat defines the maximum number of seconds between two data updates before the value of the data is assumed to be unknown.  This determines how rrdtool will interpolate data between readings.

Min and Max establish the expected range values for the data.  Setting appropriate values provides additional safeguards against the accidental inclusion of invalid data.  If the expected range values are unknown a value of “U” can be set instead.

Round Robin Archive Definition

The general format for a RRA specification is:

RRA:Consolidation Function:XFF:Steps:Rows

Consolidation Function (CF) is one of rrdtool’s defined functions:  AVERAGE, MIN, MAX, and LAST.  The different functions provide flexibility when aggregating Primary Data Points (PDP) into a RRA.  Note that regardless of the CF chosen, aggregation will result in the loss of resolution for the data.  For many applications, this is perfectly acceptable as maintaining fine-grained resolution on older data is often not necessary.

AVERAGE stores the average of all the Primary Data Points within the designated time range.

MIN stores the smallest value of all the Primary Data Pointswithinthe designated time range.

MAX stores the largest value of all the Primary Data Pointswithinthe designated time range.

LAST stores only the last (most recent) value of all the Primary Data Points within the designated time range.

XFF is the “X Files Factor”.  This defines how many Primary Data Points may be unknown before the Consolidated Data Point is also defined as unknown.  It is expressed as the ratio of the number of unknown PDP’s over the total number PDP’s for the aggregation period.  As such, valid values are between 0 and 1.  As an example, a XFF of 0.5 dictates that a CDP will be stored as unknown if the number of unknown PDP’s within the time period exceeds 50% of the total number of PDP’s.

Steps defines how many Primary Data Points will be aggregated into a Consolidated Data Point.  The Consolidated Data Point is then recorded in the RRA.  Each RRA Step is a multiple of the Step value associated with the Primary Data Point.  For example, if the Primary Data Point Step has a resolution of 300 seconds and the RRA Step value is 12, then each Consolidated Data Point will be aggregated over 3600 seconds (12 x 300 seconds).

Rows defines how many Consolidated Data Point Steps are stored in the RRA.  As each Row represents the time span defined by the RRA Step, it is a simple calculation to determine the maximum time represented in the archive.  For example, if the Primary Data Point has a Step value of 300 seconds, and the RRA Step value is 12, and the Row value is 8760, then the RRA will be able to store 1 year’s worth of data.  (300s x 12 x 8760 == 1 year)

rrdtool create

Before attempting to create a RRD, it is best to have fully defined the relevant Data Sources and Round Robin Archives the RRD will be storing.  With those defined, there are only a couple other command line options to define in order to successfully create the RRD.

The general format for creating a RRD is as follows:

rrdtool create filename --start time --step secs \
DS:Label:Type:Heartbeat:Min:Max \
RRA:Consolidation Function:XFF:Steps:Rows

filename is any legitmate filename permitted by the filesystem.  It is suggested that a suffix of “.rrd” be used to identify the file as a Round Robin Database.

start is the earliest time represented in the RRD.  The RRD will not accept any times that predate (or equal) this value.  The time is frequently presented as a standard Unix epoch value (number of seconds since Jan 1, 1970 UTC) although it it will also parse other formats including “now”, “now – 1 hour”, and more.  (See the manual page for the at command for more details on the time format rrdtool supports.)  The default value is the current time minus 10 seconds.

step is the number of seconds each Primary Data Point represents.  The default value (300 seconds) provides a resolution of 1 PDP every 5 minutes.

Examples

Example 1

rrdtool create sysinfo.rrd --start now --step 300 \
DS:load:GAUGE:600:0:U \
RRA:AVERAGE:0.5:1:8928

This example creates a RRD with the filename of sysinfo.rrd.  The starting time is “now” which rrdtool will determine at the time the command is executed.  The step value for the Primary Data Points is 300 seconds.

The only Data Source defined in this RRD is labelled “load” and is of type GAUGE.  It has a heartbeat value of 600 seconds, a minimum value of 0 and an unlimited (“U”) maximum value.

There is only one Round Robin Archive defined, and it uses the AVERAGE  method to aggregate Primary Data Points into Consolidated Data Points.  It has an XFF of 0.5, which means that no more than 50% of the PDP’s can be of the “unknown” type or the CDP will also be of “unknown” type.  The RRA step value is 1, which in this case means there is a 1:1 correspondence between a PDP and a CDP.  (i.e. no aggregation is actually performed)  There are 8928 data rows in the RRA, so this archive represents a span of 31 days (8928 x 300 seconds).

Example 2

rrdtool create sysinfo.rrd --start now --step 300 \
DS:load:GAUGE:600:0:U \
RRA:AVERAGE:0.5:1:8928 \
RRA:AVERAGE:0.5:12:8760

This example is identical to Example 1 with the exception that it contains more than one RRA.

In this case, the first RRA defines a “high-resolution” view of the data but only for the last 31 days.  In many applications, the value of the data is highest for the most recent data, and so maintaining it with a high fidelity is warranted.

The second RRA defines a “low-resolution” view of the data for a substantially longer period of time.  In this example, the RRA step value is 12, so the AVERAGE Consolidation Function will be applied to 12 Primary Data Points to generate a single Consolidated Data Point.  As each PDP represents 300 seconds, each step in this RRA will represent 3600 seconds, or 1 hour (300s x 12).  With 8760 rows for this RRA, this archive represents a total time span of 1 year (365 days x 24 hours = 8760).

When generating graphs, rrdtool is smart enough to use the appropriate RRA(s) to provide the appropriate resolution for the time span requested.

Example 3

rrdtool create sysinfo.rrd --start now --step 300 \ DS:load:GAUGE:600:0:U \
DS:diskfree:DERIVE:600:U:U \
RRA:AVERAGE:0.5:1:8928 \
RRA:AVERAGE:0.5:12:8760

This example further extends the setup illustrated in Example 2.  In this case, an additional Data Source has been defined to track the rate of change in the free disk space.  As the amount of free disk space may increase or shrink, the DERIVE data type is required.  Like the GAUGE Data Source, it has a heartbeat value of 600 seconds but both the minimum and maximum values are unknown for this DS.  Note that the RRA’s apply to both of the Data Sources.