Enabling DNS statistics for Mac OS X Server

The DNS server is an essential service for the smooth operations of any site, but there is little insight into how it is being used or its operational health presented in the Apple Server Admin tool.  Thankfully, Apple uses the standard bind9 utility as the underlying implementation for the DNS service and this can provide a great deal of statistics for analysis.  Unfortunately, the statistics are currently disabled in the default installation and there is no means to enable them using the Server Admin tool but it is relatively straight-forward to edit a couple configuration files in order to unleash this capability.

First, the /etc/named.conf file needs to be modified to specify both the statistics file and the appropriate control mechanisms.  As this is an essential configuration file, create a backup copy of the existing (working) file before editing in case something goes wrong.

In the named.conf file, ensure that there is an configuration block for the “controls”.  By default, it should already be present and look similar to the following:

controls  {
        inet 127.0.0.1 port 54 allow    {any;   }
        keys    { "rndc-key";    };
   };

This section defines the access control for the stats tool (among other things).  The essential elements for this purpose is that it defines the communications port as port 54 and it uses the key identified as “rndc-key” as its security token.  The key is actually defined in a separate file and read into the configuration via the include "/etc/rndc.key" statement earlier in the named.conf file.

There should also be a configuration block for the “options”.  A typical installation will look like the following:

options  {
        include "/etc/dns/options.conf.apple";
                /*
         * If there is a firewall between you and nameservers you want
         * to talk to, you might need to uncomment the query-source
         * directive below.  Previous versions of BIND always asked
         * questions using port 53, but BIND 8.1 uses an unprivileged
         * port by default.
         */
        // query-source address * port 53;
   };

This section requires some minor modification to specify the stats file.  In the block, insert a line specifying statistics-file "/full/path/to/the/named.stats";.  It is best to make the addition after the include statement so that the edited result looks similar to the following:

options  {
        include "/etc/dns/options.conf.apple";

        // Addition to enable statistics
        statistics-file "/var/log/named.stats";

                /*
         * If there is a firewall between you and nameservers you want
         * to talk to, you might need to uncomment the query-source
         * directive below.  Previous versions of BIND always asked
         * questions using port 53, but BIND 8.1 uses an unprivileged
         * port by default.
         */
        // query-source address * port 53;
   };

After the named.conf file has been configured properly, a new file must created for the configuration of the rndc utility.  Create the /etc/rndc.conf file and edit it with the following settings:

include "/etc/rndc.key";

options {
    default-key "rndc-key";
    default-server localhost;
    default-port 54;
};

The include statement must match the same included file specified in the named.conf file as the security key (“rndc-key”) must match or the rndc utility used to trigger the stats dump will not be authorized to connect.  The options block specifies the key name and the default server and port.  The assumption here is that the rndc tool will be run on the same host (localhost) as the DNS server.  The port should match the port specified in the controls block of the named.conf file.

At this point, the named daemon can be safely signaled to pick up the configuration changes.  This can be done easily via the Server Admin tool by restarting the DNS service or, for the more advanced sysadmin, a SIGHUP can be sent to the process.

To trigger a stats dump, simply call the rndc stats and look for the stats file specified in the named.conf file.  If the file isn’t found, there may be a authorization issue:  try running the command as an administrator or call the command via sudo (e.g. sudo rndc stats).

The stats can be triggered on an ongoing basis by specifying an appropriate entry for the launchd system.  A simple plist formatted file located in /Library/LaunchDaemons can serve to repeatedly call the stats file for ongoing monitoring.  (See launchd tips and tricks for more information.)

The following example would be named com.hostname.namedstats.plist and specifies that the stats should be generated every 5 minutes (300 seconds).

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.hostname.namedstats</string>
    <key>StartInterval</key>
    <integer>300</integer>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/sbin/rndc</string>
        <string>stats</string>
    </array>
</dict>
</plist>

Note that every time the stats are generated they are appended to the existing file which can quickly result in a very large, unwieldy file.  If recurring stats are desired, then ensuring the stats file is periodically rotated is essential.

 

Automatic Log Rotation in Mac OS X

Log rotation is a standard practice to ensure that log files are periodically restarted so that they don’t grow without bounds.  In the past, when disk space was at a premium and maximum file sizes were limited, this was an absolute necessity.  For more modern systems, these constraints may be less relevant although they may continue to be a valid concern for very active systems or with verbose logging enabled.  Even with abundant disk space, appropriate managing of the system logs helps ensure the system remains healthy, logs remain available for diagnostics and analysis, and operational overhead is reduced.

In Linux, the typical log rotation utility is, appropriately enough, logrotate.  However, Mac OS X uses a different tool, newsyslog, to manage the log rotation responsibilities.  This utility is run automatically by the OS at frequently intervals and evaluates the rules defined in its configuration file — /etc/newsyslog.conf.  For the log files defined in the config, it can evaluate the need to rotate based on a number of criteria, such as file size, at a specific time, at a specific interval, etc.  It can also specify additional actions to take on the rotated log, including compression of the old log, maintaining a fixed number of previous log files, setting appropriate permissions, etc.  Full documentation is available in the man page for newsyslog.conf.

Each entry in the newsyslog.conf file is specified as a single line with space-delimited fields for the various options.  The general format is as follows (with [] surrounding the optional fields):

logfilename [owner:group] mode count size when flags [/pid_file] [sig_num]

An example entry for rotating the named.stats log file can be specified as follows:

/var/log/named.stats                    640  5     1000 *     J

In the above example, the full path is specified for the target log file — /var/log/named.stats.  Permissions (640) are specified in the typical Unix manner, in this case enabling read/write for the owner and read-only for the group.  This log file will maintain a maximum of 5 files, removing the oldest file as newer archived logs are created.  This log file is also set to rotate at a fixed size — 1000 kilobytes and can rotate at any time (“*”) this size threshold is exceeded.  The “J” flag specified calls for the the older log files to be compressed (further reducing used disk space) using the bzip2 utility.

A special note for binary (not text-based) logs:  by default, newsyslog will inject a line at the beginning of a file indicating when and why the log rotated.  For most text-based logs, this can be easily skipped over but for binary logs this might result in an unreadable file.  In this case, be sure to specify the “B” flag which suppresses the informational line from being injected.

Enable Remote Logging on Mac OS X

Background

It is often desirable to collect the system logs from various devices onto a central “logging host”.  This can simplify monitoring needs considerably as tools and scripts need only work on a single host.  It also provides some security benefits as hackers will have a harder time masking their activities if they do not have access to the logging host.

On Mac OS X, there is a logging daemon (the Unix standard syslogd utility) that can be used to write both local logs and receive logs from remote hosts.  Unfortunately, the ability to receive remote logs is turned off in the default installation.  However, it is a trivial task to enable this functionality.

Setup (for 10.5 and older systems)

  1. Login as administrator to the logging host
  2. Open a terminal session using the Terminal utility
  3. Navigate to the LaunchDaemons directory
    cd /System/Library/LaunchDaemons
  4. Edit the com.apple.syslogd.plist file
    sudo vi com.apple.syslogd.plist
  5. Remove the comment delimiters (<!-- and -->) surrounding the NetworkListener block and save the changes
    <!--
     <key>NetworkListener</key>
     <dict>
     <key>SockServiceName</key>
     <string>syslog</string>
     <key>SockType</key>
     <string>dgram</string>
     </dict>
    -->
  6. Stop the currently running instance of the syslog daemon
    sudo launchctl unload /System/Library/LaunchDaemons/com.apple.syslogd.plist
  7. Restart the syslog daemon to pick up the changes in the LaunchDaemon configuration
    sudo launchctl load /System/Library/LaunchDaemons/com.apple.syslogd.plist

The logging facility on the log host should now be available for remote devices to use.

Setup (for 10.6 and later systems)

  1. Login as administrator to the logging host
  2. Open a terminal session using the Terminal utility
  3. Navigate to the LaunchDaemons directory
    cd /System/Library/LaunchDaemons
  4. Convert the com.apple.syslogd.plist to XML format
    sudo plutil -convert xml1 com.apple.syslogd.plist
  5. Edit the com.apple.syslogd.plist file
    sudo vi com.apple.syslogd.plist
  6. Add the NetworkListener dict entry after the end of the BSDSystemLogger dict entry and save the changes
    <key>NetworkListener</key>
    <dict>
    <key>SockServiceName</key>
    <string>syslog</string>
    <key>SockType</key>
    <string>dgram</string>
    </dict>
  7. Convert the com.apple.syslogd.plist file back to the binary format
    sudo plutil -convert binary1 com.apple.syslogd.plist
  8. Stop the currently running instance of the syslog daemon
    sudo launchctl unload /System/Library/LaunchDaemons/com.apple.syslogd.plist
  9. Restart the syslog daemon to pick up the changes in the LaunchDaemon configuration
    sudo launchctl load /System/Library/LaunchDaemons/com.apple.syslogd.plist

The logging facility on the log host should now be available for remote devices to use.

Setup (using the Property List Editor)

Apple provides the Property List Editor utility as part of its developer tools.  The developer tools are usually a separate install from the operating system and is frequently not installed on systems.  If it is present, then using the Property List Editor may be the most convenient means of modifying the com.apple.syslogd.plist file.

  1. Login as administrator to the logging host
  2. Launch the Property List Editor utility (typically located in /Developer/Applications/Utilities)
  3. Open the /System/Library/LaunchDaemons/com.apple.syslogd.plist file
  4. Select the “Sockets” dictionary entry and “Add Item” to create a new key in the “Sockets” dictionary
  5. Change the name of the item to “NetworkListener” and set its type to “Dictionary”
  6. Select the “NetworkListener” dictionary entry and “Add Item” to create a new key in the “NetworkListener” dictionary
  7. Change the name of the item to “SockServiceName”, its type to “String” and its value to “syslog”
  8. Select the “NetworkListener” dictionary entry and “Add Item” to create another new key in the “NetworkListener” dictionary
  9. Change the name of the item to “SockType”, its type to “String”, and its value to “dgram”
  10. Save the file and quit the Property List Editor utility.  The property list should resemble the following example:Example Property List for com.apple.syslogd.plist
  11. Open a terminal session using the Terminal utility
  12. Navigate to the LaunchDaemons directory
    cd /System/Library/LaunchDaemons
  13. Stop the currently running instance of the syslog daemon
    sudo launchctl unload /System/Library/LaunchDaemons/com.apple.syslogd.plist
  14. Restart the syslog daemon to pick up the changes in the LaunchDaemon configuration
    sudo launchctl load /System/Library/LaunchDaemons/com.apple.syslogd.plist