« Oh come on now... | Main | Yes I know... »

A Primer to SNMP on Mac OS X 10.5 and Mac OS X 10.5 Server

"Simple Network Management Protocol"

Has there ever been a more misleading or misunderstood name? I don't think so. But, SNMP is one of the most widely used, if not the most widely-used network management protocol. If you wrap your head around it, you can do some really neat stuff to make your life easier. If you don't, well, you'll end up using it anyway, you just won't know what's going on.

Since I see a lot of SNMP questions, I felt it was a good idea to write something up on SNMP that went into some depth. First, just in case it comes up, I have read Andrina Kelly's excellent article on SNMP at MacEnterprise.org. Her article was focused on using snmpconf to do the basic setup in Mac OS X, and it's solid. This article is something of a superset of that, and is correctly viewed as a supplement to that article. In fact, I owe Andrina some thanks for that article, since it motivated me to write this one. (I'd say I owe her beer, but she's Canadian, and I'm not that rich.)

So first off, what's the deal with SNMP? Well, the 'deal' is that people wanted a flexible, relatively low overhead way to manage their network. They wanted a way to read and set information, and have devices inform them when something goes wrong, or is about to go wrong. The result is SNMP. Now, why "Simple", when, as we shall see, it's actually pretty complex. Well, it's "Simple" in the sense that there's not much to it. There's about three things that can happen in SNMP:

  1. You can query a value and get a reply

  2. You can set a value and get a return code

  3. An SNMP-enabled device can send you a notification, or "trap"

That's pretty much the entire range of SNMP right there, and of the three, two is the least common. It's what you do with those numbers that gives you the power, or rather, what utilities like Nagios, Cacti, and Lithium do with the numbers that make things interesting.

To really understand SNMP, we have to understand two things: Object Identifiers, or OIDs, and Management Information Bases, or MIBs.

OIDs and MIBs

Both OIDs and MIBs are defined by the Structure of Management Information, (SMI) RFCs, (1155 for SMIv1 which ties to SNMPv1, 2578 for SMIv2, which ties to SNMPv2. There are two follow-on SMI RFCs, 3780 and 3781, but they are not bound to SNMP in particular, although 3781 is a set of SNMP-specific extensions to 3780. (Note that there are about 75 RFCs that apply to SNMP overall, so I'm not listing them. If you want to read them, the IETF is the keeper of the keys, as it were, for RFCs, and is the best place to go for RFC information.)

Within the SMI(s), you have three basic items

  1. The Name, or OID, which is how you identify SNMP objects in a structured, unique manner.

  2. The type and syntax, (defined via Abstract Syntax Notation One, (ASN.1), which is what defines how data is represented and transmitted. This handles things like byte-order/endian issues, so that Intel systems can use SNMP with SPARC systems, and not have to worry about data formatting issues.

  3. Encoding, which defines how SNMP information is encoded/decoded for transmission over networks.

The part we care most about is the OID. OIDs are, for better clarity, (and believe me, OIDs need all the clarity they can get), arranged in a tree structure. So each part of an OID number tells you on what part of the overall OID 'tree' that data lives on. Each OID branch has a name and a purpose. For 99.9% of all uses, the "root" of the OID tree is .1.3.6.1, or iso(1).org(3).dod(6).internet(1). There are other structures in the OID tree, but for our needs, "root" is .1.3.6.1.

Within the .1.3.6.1 OID, there are four sub-branches:

  1. directory(1)

  2. mgmt(2)

  3. experimental(3)

  4. private(4)

Of these, we only care about 2 and 4. mgmt(2) is for 'standard' management objects, and private(4) is for company-specific objects. So standard OIDs for calculating traffic flow or CPU load would go under .1.3.6.1.2, but company - specific OIDs for say, the Airport Extreme Base station go under 1.3.6.1.4. This separation gives companies a lot of room to develop their own SNMP implementations without interfering with the standard ones.

Within each OID entry, you have a specific data type for the value returned by the OID. For example:

.1.3.6.1.2.1.1.3.0 = Timeticks: (1902335) 5:17:03.35

.1.3.6.1.2.1.1.4.0 = STRING: John C. Welch <jwelch@bynkii.com>

.1.3.6.1.2.1.1.5.0 = STRING: localhost

.1.3.6.1.2.1.1.6.0 = STRING: John Welch's cube

Since the first entry is the uptime of my system, that's returned as a Timeticks value. Each "tick" in this case is 1/100th of a second. The rest are strings, as they hold string data. Other values can include IpAddress, use for IPv4 addresses, NetworkAddress, which can be used for MAC addresses, Counter, a 32-bit counter, (in SNMPv2, Counter is Counter32, so as to distinguish it from the 64-bit Counter64), and Gauge, which is similar to a counter, but rather than constantly incrementing, it is used to show instantaneous value, such as the current traffic rate in bps of a router. (If you think of Counter as an odometer, and Gauge as a speedometer/tachometer, you've got the right mental picture.)

There are something like eighteen different data types in SNMP, and to be honest, you could easily reduce that to two or three. But, there's an advantage to more specific datatypes, and that is in processing the results. It's much easier to get a list of IP addresses in my routing table out of SNMP results if I know that the label for the data is going to be "IpAddress: <data>" That's not the only reason, but it comes in right handy.

So now, if an OID is a specific instance of a SNMP value, then what do we need MIBs for?

Well, something needs to exist to tell us what that OID is. The examples I used above are pretty obvious, and even if you weren't up on SNMP, you could probably guess at what the OID that returns "John Welch's cube" is used for. However, what happens when you get this: .1.3.6.1.2.1.4.14.0 = Counter32: 2 ? That's not real obvious. Well, luckily, OIDs have text labels as well as numerical labels. Changing a formatting parameter in your snmpwalk gets you:

.iso.org.dod.internet.mgmt.mib-2.system.sysUpTime.sysUpTimeInstance = Timeticks: (1990007) 5:31:40.07

.iso.org.dod.internet.mgmt.mib-2.system.sysContact.0 = STRING: John C. Welch <jwelch@bynkii.com>

.iso.org.dod.internet.mgmt.mib-2.system.sysName.0 = STRING: localhost

.iso.org.dod.internet.mgmt.mib-2.system.sysLocation.0 = STRING: John Welch's cube

That's pretty cool, as it's much easier to tell what something is when you have the text labels. So now, our previously inscrutable numerical OID becomes: .iso.org.dod.internet.mgmt.mib-2.ip.ipReasmReqds.0 = Counter32: 2. Okay, so it's not exactly crystal clear to a newbie, it's still easier to figure out.

Well, that is what you use MIBs for. MIBs help define and organize OIDs. If OIDs are thought of as languages, then MIBs are translators. This is especially important when you start talking about private MIBs, like ones used by vendors for their own products. For example, if you query a Netgear WG102 wireless base station without the proper MIBs, you get a lot of OIDs that look like:

.iso.org.dod.internet.private.enterprises.4526.4.3.1.1.0 = STRING: "00146C689987"

.iso.org.dod.internet.private.enterprises.4526.4.3.1.2.0 = STRING: "Version 4.0 Release 16 NA"

.iso.org.dod.internet.private.enterprises.4526.4.3.1.3.0 = STRING: "myap1"

.iso.org.dod.internet.private.enterprises.4526.4.3.1.7.0 = INTEGER: 840

So you can guess at what those values mean, but that's a pain, because you're never sure just what the numerical parts of that OID really mean. Now, the same OIDs, but with the proper MIB:

.iso.org.dod.internet.private.enterprises.netgear.wireless.wg102.sysSettings.sysMacAddress.0 = STRING: 00146C689987

.iso.org.dod.internet.private.enterprises.netgear.wireless.wg102.sysSettings.sysVersion.0 = STRING: Version 4.0 Release 16 NA

.iso.org.dod.internet.private.enterprises.netgear.wireless.wg102.sysSettings.sysAPName.0 = STRING: myap1

.iso.org.dod.internet.private.enterprises.netgear.wireless.wg102.sysSettings.sysCountryRegion.0 = INTEGER: unitedStates(840)

With the MIB, we can see that .enterprises.4526.4.3.1.1.0 is really .enterprises.netgear.wireless.wg102.sysSettings.sysMacAddress.0. For fairly obvious things, the MIB isn't a big deal. But when you really need to see what a specific OID is measuring, (critical when you're deciding what SNMP values you need to monitor), the MIB is critical. Luckily, most networking companies are pretty cool about MIBs. Some have them available for download, others give you updates when you download firmware updates, etc.

But what do you do when you get a company that's not always so...nice, about MIBs? Well, you can try to search for them on the vendor's site, however, I've found that it's easier to go to one of the MIB aggregation sites, like mibDepot, or one of a dozen others, and just snag the MIB from them. Since MIBs are just text files, they're easy enough to read. However, having the MIB is not the same as having the OID.

For example, I have the MIBs for quite a few Windows Server services, such as Exchange et al. That doesn't mean that all the Exchange information in a MIB, even a Microsoft MIB is going to be available. The vendor for the product you're trying to measure has to allow for their SNMP implementation to expose the information you seek. If they don't, then the MIB does you no good. MIBs only help you read what is there already, they don't create new data or information for you. Bearing that in mind, let's take a look at a MIB that ships with Leopard. To see what MIBs are standard in Mac OS X, go to /usr/share/snmp/mibs/ and take a look. You'll see a bunch of them, but for this example, we're going to take a look at the MIB for the WG102, primarily because it's short.:

WG102 DEFINITIONS ::= BEGIN



     IMPORTS

          OBJECT-GROUP

               FROM SNMPv2-CONF

          MODULE-IDENTITY, OBJECT-TYPE, Unsigned32, enterprises, IpAddress


               FROM SNMPv2-SMI

          DisplayString, TruthValue

               FROM SNMPv2-TC;

The first line tells us the name of the MIB, in this case, "WG102". The next part starts the definitions of the MIB itself. In this case, the WG102 MIB imports some data it needs from other MIBs, such as OBJECT-GROUP from SNMPv2-CONF, MODULE-IDENTITY, OBJECT-TYPE, Unsigned32, enterprises, and IpAddress from SNMPv2-SMI, PhysAddress from RFC1213-MIB, and DisplayString, TruthValue from SNMPv2-TC. Importing here works the way it does in Perl, Python, AppleScript, etc., and allows a MIB to use data and datatypes defined in other MIBs without having to explicitly identify that data and those datatypes itself.

wg102 MODULE-IDENTITY

          LAST-UPDATED "200509291000Z" -- Sep 29, 2005 at 10:00 GMT

          ORGANIZATION

               "NETGEAR Inc."

          CONTACT-INFO

               "4500 Great America Parkway

               Santa Clara, California 95054

               Phone: (408) 907-8000

               Fax: (408) 907-8097

               Web Site: http://www.netgear.com"

          DESCRIPTION

               "The MIB module for 802.11b/g ProSafe Wireless Access Point entities.

               iso(1).org(3).dod(6).internet(1).private(4).

               enterprises(1).netgear(4526).wireless(4).wg102(3)"

          ::= { wireless 3 }



--

-- Node definitions

--



          netgear OBJECT IDENTIFIER ::= { enterprises 4526 }





          wireless OBJECT IDENTIFIER ::= { netgear 4 }

This section provides more basic information about the MIB itself, such as information on the vendor, when it was updated last, and the OIDs it's going to identify. In a MIB, -- is the single line comment identifier, so when you see it used, it's for well, comments. In the case of this MIB, pretty much everything here is part of the netgear.wireless OID, or the .1.3.6.1.4.1.4526.4.3 tree. Looking at the first part of the first section we see:

sysSettings OBJECT IDENTIFIER ::= { wg102 1 }

Since this part comes after the 4526.4, we can tell that for this mib, sys(tem)Settings are in the .1.3.6.1.4.1.4526.4.3.1 tree.

          sysMacAddress OBJECT-TYPE

               SYNTAX DisplayString (SIZE(6..17))

               MAX-ACCESS read-only

               STATUS current

               DESCRIPTION

                    "This field indicates the system MAC Address."

               ::= { sysSettings 1 }

Looking at this first entry, we see that it's the sysMACAddress object type. The value it returns is a string, and Max-Access tells us it's a read-only value. The Status indicates this is a current value, (i.e. not deprecated or obsolete). Finally, we have a Description, and the actual number of the sysMacAddress in the .1.3.6.1.4.1.4526.4.3.1 tree. If we look at that OID on a WG102, we get: .1.3.6.1.4.1.4526.4.3.1.1.0 = STRING: 00146C689987, which is exactly what the MIB told us we'd see. Now, some of you see that OID and are wondering, "What's the deal with that 0? That's not in the MIB". First, correct, it's not in the MIB. Here's the deal. When you're looking at a listing of OIDs, there are two ways to show things. One is a list of vaguely related items, or scalar objects A scalar object can be related to the objects above and below it in a tree, but it doesn't have to be. The other way to display objects is as a table, where each individual item in the table has its own OID value, but it's part of the table OID. With a scalar object, there's only ever one. So, since computers start counting at 0, to get a scalar object's information, you end the OID with ".0". With a table, that last number would be the specific row in a table. So, if you wanted the third row of a given table, you'd use .2 on the end. In this case, the sysMacAddress is a scalar value, so we end the OID with a .0

An example of a table entry in a MIB is seen below:

          wlanSettings OBJECT IDENTIFIER ::= { wg102 2 }





-- **********************************************************************

-- * Wireless Settings Table

-- **********************************************************************

          wlanSettingTable OBJECT-TYPE

               SYNTAX SEQUENCE OF WlanSettingEntry

               MAX-ACCESS not-accessible

               STATUS current

               DESCRIPTION

                    "wlanSettingsallow for multiple instances on an agent."

               ::= { wlanSettings 1 }





          wlanSettingEntry OBJECT-TYPE

               SYNTAX WlanSettingEntry

               MAX-ACCESS not-accessible

               STATUS current

               DESCRIPTION

                    "An entry in the wlanSettingTable. It is possible for there

                    to be multiple AP interfaces on one agent, each with its

                    unique MAC address. The relationship between an AP

                    interface and an interface in the context of the Internet-

                    standard MIB is one-to-one. As such, the value of an

                    radioIndex object instance can be directly used to identify

                    corresponding instances of the objects defined herein. "

               INDEX { radioIndex }

               ::= { wlanSettingTable 1 }

The first two entries here are not accessible via SNMP, but help to identify the overall OID for the table. In this case, we see that the base OID for this table is .1.3.6.1.4.1.4526.4.3.2.1.1 (If you can't see this easily, don't worry. I'm only using specific parts of the MIB here, it's really rather large.)


          WlanSettingEntry ::=

               SEQUENCE {

                    radioIndex

                         INTEGER,

                    radioEnable

                         TruthValue,

                    wirelessMode

                         INTEGER,

                    channel

                         INTEGER,

                    txRate

                         INTEGER,

                    txPower

                         INTEGER,

                    beaconInterval

                         INTEGER,

                    dtimInterval

                         INTEGER,

                    rtsThreshold

                         INTEGER,

                    fragmentationThreshold

                         INTEGER,

                    dot11bPreamble

                         INTEGER,

                    superMode

                         TruthValue,

                    wmm
     
                         TruthValue,

                    wmmNoAck

                         TruthValue,
          
                    acEnabled

                         TruthValue,

                    acEnhancedRFSecurity

                         INTEGER,
     
                    acRogueDevDetection

                         TruthValue,

                    accessControlMode

                         INTEGER

                }

Here we have the row identifiers, and their basic types. From this we can tell that the table has 18 rows, starting with radioIndex, and ending with accessControlMode. From here, we go into the details for each row:

          radioIndex OBJECT-TYPE

               SYNTAX INTEGER

                    {

                    dot11a(0),

                    dot11bg(1)

                    }

               MAX-ACCESS read-only

               STATUS current

               DESCRIPTION

                    "This attribute shall indicate the radio's band."

               ::= { wlanSettingEntry 1 }

Our first row, radioIndex is kind of neat. First, it's an integer, but can have two possible values, 0, (for 802.11a) and 1, (for 802.11b/g). It's read-only, a current object, and has a row number of 1. (I'll explain where the 0 went in a bit) If we look at the specific entry for this row with just the numerical OID, and the text labels we see:

.1.3.6.1.4.1.4526.4.3.2.1.1.1.1 = INTEGER: dot11bg(1)
.iso.org.dod.internet.private.enterprises.netgear.wireless.wg102.wlanSettings.wlanSettingTable.wlanSettingEntry.radioIndex.dot11bg = INTEGER: dot11bg(1)

So, from just that, we can see this base station is set to work with 802.11b/g. The OID tells us what the device knows, the MIB explains what the OID means. Looking down the rest of the entries, we see just what all this table does:

     radioEnable OBJECT-TYPE

          SYNTAX TruthValue

          MAX-ACCESS read-write

          STATUS current

          DESCRIPTION

               "This attribute shall indicate whether the radio is enabled."

          ::= { wlanSettingEntry 2 }


     wirelessMode OBJECT-TYPE

          SYNTAX INTEGER

               {

               auto(0),

               dot11a(1),

               dot11b(2),

               dot11g(3)

               }

          MAX-ACCESS read-write

          STATUS current

          DESCRIPTION

               "This attribute shall indicate the desired wireless

               operating mode.

               
               Options are:

                auto - Both 802.11g and 802.11b wireless stations can be

                used.

               dot11a - Only 802.11a wireless stations can be used.

               dot11b - All 802.11b wireless stations can be used.

                     802.11g wireless stations can still be used

                     if they can operate in 802.11b mode.

               dot11g - Only 802.11g wireless stations can be used."

          ::= { wlanSettingEntry 5 }


     channel OBJECT-TYPE

          SYNTAX INTEGER (0..165)

          MAX-ACCESS read-write

          STATUS current

          DESCRIPTION

               "This attribute shall indicate the channel number to be

               used. And the zero indicates that auto channel selection is enabled."

          ::= { wlanSettingEntry 6 }


     txRate OBJECT-TYPE

          SYNTAX INTEGER

               {

               best(0),

               rate1Mbps(1),

               rate2Mbps(2),

               rate5dot5Mbps(3),

               rate6Mbps(4),

               rate9Mbps(5),

               rate11Mbps(6),

               rate12Mbps(7),

               rate18Mbps(8),

               rate24Mbps(9),

               rate36Mbps(10),

               rate48Mbps(11),

               rate54Mbps(12)

               }

          MAX-ACCESS read-write

          STATUS current

          DESCRIPTION

               "This attribute shall indicate the transmite rate. When

               the operatemode is:

               
                auto(0) - can set to 0:best, 1:1Mb/s, 2:2Mb/s,

                5.5:5.5Mb/s, 11:11 Mb/s, 6:6Mb/s, 9:9Mb/s,

                12:12Mb/s, 18:18Mb/s, 24:24Mb/s, 36:36Mb/s,

                48:48Mb/s, and 54:54Mb/s.

               
                dot11a(1) - can set to 0:best, 6:6Mb/s, 9:9Mb/s,

                12:12Mb/s, 18:18Mb/s, 24:24Mb/s,

                36:36Mb/s, 48:48Mb/s, 54:54Mb/s.

               
                dot11b(2) - can set to 0:best, 1:1Mb/s, 2:2Mb/s,

                5.5:5.5Mb/s, and 11:11 Mb/s.

               
                dot11g(3) - can set to 0:best, 6:6Mb/s, 9:9Mb/s,

                12:12Mb/s, 18:18Mb/s, 24:24Mb/s,

                36:36Mb/s, 48:48Mb/s, 54:54Mb/s."

          ::= { wlanSettingEntry 7 }


     txPower OBJECT-TYPE

          SYNTAX INTEGER

               {

               full(0),

               half(1),

               quarter(2),

               eighth(3),

               min(4)

               }

          MAX-ACCESS read-write

          STATUS current

          DESCRIPTION

               "This attribute shall indicate the transmitting power."

          ::= { wlanSettingEntry 8 }


     beaconInterval OBJECT-TYPE

          SYNTAX INTEGER (20..1000)

          UNITS "1024 microsecond"

          MAX-ACCESS read-write

          STATUS current

          DESCRIPTION

               "This attribute shall indicate the beacon interval."

          ::= { wlanSettingEntry 9 }


     dtimInterval OBJECT-TYPE

          SYNTAX INTEGER (1..255)

          MAX-ACCESS read-write

          STATUS current

          DESCRIPTION

               "This attribute shall indicate the DTIM period."

          ::= { wlanSettingEntry 10 }


     rtsThreshold OBJECT-TYPE

          SYNTAX INTEGER (0..2346)

          MAX-ACCESS read-write

          STATUS current

          DESCRIPTION

               "This attribute shall indicate the RTS threshold."

          ::= { wlanSettingEntry 11 }


     fragmentationThreshold OBJECT-TYPE

          SYNTAX INTEGER (256..2346)

          MAX-ACCESS read-write

          STATUS current

          DESCRIPTION

               "This attribute shall indicate the fragmentation threshold."

          ::= { wlanSettingEntry 12 }


     dot11bPreamble OBJECT-TYPE

          SYNTAX INTEGER

               {

               long(0),

               auto(1)

               }

          MAX-ACCESS read-write

          STATUS current

          DESCRIPTION

               "This attribute shall indicate the preamble setting.

               This setting is only applicable to 802.11b mode."

          ::= { wlanSettingEntry 13 }


     superMode OBJECT-TYPE

          SYNTAX TruthValue

          MAX-ACCESS read-write

          STATUS current

          DESCRIPTION

               "This attribute shall indicate whether super mode (super-A

               for 11a radio, Super-G for 11g radio) is enabled."

          ::= { wlanSettingEntry 14 }


     wmm OBJECT-TYPE

          SYNTAX TruthValue

          MAX-ACCESS read-write

          STATUS current

          DESCRIPTION

               "This attribute shall indicate whether wmm is enabled."

          ::= { wlanSettingEntry 15 }


     wmmNoAck OBJECT-TYPE

          SYNTAX TruthValue

          MAX-ACCESS read-write

          STATUS current

          DESCRIPTION

               "This attribute shall indicate whether wmm with no ack is enabled."

          ::= { wlanSettingEntry 16 }


     acEnabled OBJECT-TYPE

          SYNTAX TruthValue

          MAX-ACCESS read-write

          STATUS current

          DESCRIPTION

               "This field indicates whether enable AutoCell."

     ::= { wlanSettingEntry 17 }


     acEnhancedRFSecurity OBJECT-TYPE

          SYNTAX INTEGER

               {

               disable(0),

               enable(3)

               }

          MAX-ACCESS read-write

          STATUS current

          DESCRIPTION

               "This field indicates whether enable Enhanced RF Security."

          ::= { wlanSettingEntry 18 }


     acRogueDevDetection OBJECT-TYPE

          SYNTAX TruthValue

          MAX-ACCESS read-write

          STATUS current

          DESCRIPTION

               "This field indicates whether enable Rogue Device Detection."

          ::= { wlanSettingEntry 19 }


     accessControlMode OBJECT-TYPE

          SYNTAX INTEGER

               {

               disabled(0),

               local(2),

               server(3)

               }

          MAX-ACCESS read-write

          STATUS current

          DESCRIPTION

               "The field indicates whether the access control list is

               enabled and the source of the database of the access

               control list."

          ::= { wlanSettingEntry 20 }

When we look at the values for each OID in the table, we get:

.1.3.6.1.4.1.4526.4.3.2.1.1.1.1 = INTEGER: dot11bg(1)

.1.3.6.1.4.1.4526.4.3.2.1.1.2.1 = INTEGER: true(1)

.1.3.6.1.4.1.4526.4.3.2.1.1.5.1 = INTEGER: dot11g(3)

.1.3.6.1.4.1.4526.4.3.2.1.1.6.1 = INTEGER: 0

.1.3.6.1.4.1.4526.4.3.2.1.1.7.1 = INTEGER: rate54Mbps(12)

.1.3.6.1.4.1.4526.4.3.2.1.1.8.1 = INTEGER: full(0)

.1.3.6.1.4.1.4526.4.3.2.1.1.9.1 = INTEGER: 100 1024 microsecond

.1.3.6.1.4.1.4526.4.3.2.1.1.10.1 = INTEGER: 1

.1.3.6.1.4.1.4526.4.3.2.1.1.11.1 = INTEGER: 2346

.1.3.6.1.4.1.4526.4.3.2.1.1.12.1 = INTEGER: 2346

.1.3.6.1.4.1.4526.4.3.2.1.1.13.1 = INTEGER: auto(1)

.1.3.6.1.4.1.4526.4.3.2.1.1.14.1 = INTEGER: false(2)

.1.3.6.1.4.1.4526.4.3.2.1.1.15.1 = INTEGER: false(2)

.1.3.6.1.4.1.4526.4.3.2.1.1.16.1 = INTEGER: false(2)

.1.3.6.1.4.1.4526.4.3.2.1.1.17.1 = INTEGER: true(1)

.1.3.6.1.4.1.4526.4.3.2.1.1.18.1 = INTEGER: disable(0)

.1.3.6.1.4.1.4526.4.3.2.1.1.19.1 = INTEGER: true(1)

.1.3.6.1.4.1.4526.4.3.2.1.1.20.1 = INTEGER: local(2)

Thanks to the MIB, we can see exactly what this table is telling us about this base station. Now, what about the 0? Well, in a table, you don't use it. 0's are for scalar values, so tables don't use it. However, since these are table values, you don't need the trailing .1 either. Looking for .1.3.6.1.4.1.4526.4.3.2.1.1.2 or .1.3.6.1.4.1.4526.4.3.2.1.1.2.1 will get you the same result.

Okay, enough with the OIDs and MIBs, lets get into actually using SNMP.

SNMP Commands

There are about twenty SNMP commands, and while it is good to learn what all of them do, in practice, there aren't that many you use regularly. Out of all of them, I tend to use snmpget and snmpwalk the most. There's a few others, but those two are about 95% of my direct SNMP command usage. There are a few things you want to keep in mind about SNMP commands, some common tips and tricks. Luckily, SNMP commands have a lot of commonality, so the options are pretty much the same for all. "man snmpcmd" will give you most of these.

There are quite a few more options, but these are the ones I use the most.

Oh, one other thing. Normally, SNMP uses UDP port 161. While you can use TCP, I really recommend you don't. SNMP does not care that much about reliability, nor does it send or get a lot of data. However, in even a medium - sized monitoring setup, you can be issuing hundreds of SNMP requests every few minutes. The overhead of TCP, combined with no real advantage to using TCP makes it a rather poor choice for SNMP over UDP.

snmpget

This is the most ubiquitous of the lot. It's the way you get single values from another device, and is probably the most commonly used SNMP command. Using snmpget is pretty simple:

snmpget options <IP address or DNS name of the target> <OID you're querying>

There are a lot of options for snmpget, (run snmpget with no parameters to see them all), but the common ones are the ones I listed above. The target is the machine you're querying. If you want to just play around with SNMP on your own machine, you can use localhost for the target. The OID can be specified either numerically or textually. If you're going to do it textually, then you want to use as small an OID specifier as possible. So, all put together, it looks like:

(Yes, I know, there's no .0 at the end of this OID. Sometimes, SNMP is like english. You get used to some rules, and then BAM! Neither "bomb", "tomb" or "comb" rhyme, and you're left wondering "What's up with that?")

snmpwalk

If you need to get a list of all the OIDs supported by a given target, or the supported elements in a specific OID tree, then you would use snmpwalk. The options for snmpwalk are the same as for snmpget, the difference being snmpget gives you a single result for an OID, the snmpwalk command returns you a list of OID values. You can list every supported OID on a machine by setting the starting OID to .1, or you can query a specific OID branch by being more specific, and either numerical or textual OID specifiers are valid:

Numerical OID specifier:

Textual OID specifier:One thing to be careful of, depending on the OID, snmpwalk can put a whack on a CPU, so if you're doing an entire OID tree, you probably want to pipe the output to more, and pause a bit in between hitting the space bar, or play with the timeout variables in the command. If you know you're going to be getting a lot of information in an snmpwalk command, you might consider using snmpbulkwalk instead, as it can be more efficient.

snmptable

This is a command I don't use that often, but it can be handy when you want to see every entry in a table without having to parse it out of snmpwalk. To use snmptable, you have to pass it the OID of a table, which can be a bit confusing at first. For example, let's say you want to look at the contents of the sysORTable with snmptable. If you look at it with snmpwalk, you'd get:

That gives you all the information, but you have to do a lot of matching between the three elements of the table.

In comparison, with snmptable, you get:

standard snmptable

Same information, but laid out nicer.

The options for snmptable are a bit different too. While you can use all the -On options, where you get a lot of use out of snmptable is in the -Cn options, as these deal with the table formatting. For example, snmptable with the -Cf , option gives you a comma-delimited table:

comma delimited snmptable

If you use the -Cl option, the data is left-justified:

left-justified snmp table

Keep in mind that you're querying the table, so there's no ending element specifier:

snmptable localhost .1.3.6.1.2.1.1.9

snmptable -Cf , localhost .1.3.6.1.2.1.1.9

snmptable -Cl localhost .1.3.6.1.2.1.1.9

snmpset

Finally, you have the way to set values, snmpset. This one is what I use when I have to deal with a lot of end machines in a hurry. For the most part, there's not a lot you use this for. You can set names, locations, and other minor information with it. Some vendors use it for interesting things. For example, there are some Netgear access points that you can reboot by setting the right OID to any non-0 number. The options for snmpset have one important difference, the type. The type is used to specify what kind of value you're attempting to set the OID to. The syntax for snmpset is:

snmpset <options> OID <type> value

The types for snmpset are:

i INTEGER

u UNSIGNED

s STRING

a IPADDRESS

b BITS

Those commands are pretty much 90% of my manual snmp command usage. However, before you use SNMP, you have to set it up.

Setting up SNMP on Mac OS X

While there are a number of ways to set up SNMP, including the ever-popular "just edit the conf files", the one I prefer is still snmpconf. Andrina did a good job of introducing it, now I'm going to beat it to death, because, well, that's what I do. One change you'll see from Mac OS X 10.5 is that the default snmp conf files are no longer kept in /usr/share/snmp. In Mac OS X 10.5, the defaults are kept in /etc/snmp/snmpd.conf.default. Neither location is wrong, but snmpconf favors /usr/share/snmp, and so do I. (Using the SNMP conventions here makes getting tips from people on other *nix platforms much easier.)

One thing with snmpconf that I like is its ability to automatically put the files you create in the "correct" place once you're done. To do that, you use the -i switch when you run it, like so: sudo snmpconf -i. snmpconf is, for those who care, a big, well-written perl script that helps you set up snmp, snmpd, and snmptrapd parameters. It's smart enough to notice that you have existing snmp conf files, including the ones in /etc/snmp, and will ask you if you want to merge those files with any you create. If you aren't confident about your ability to read these conf files and parse out extraneous/bad entries, then you'll want to choose "None" when asked. Since snmpconf takes the conf files in a specific order, (seen below) I will too. Note that I am not going to go over every option in snmpconf, just the most common ones.

I can create the following types of configuration files for you.

Select the file type you wish to create:

(you can create more than one as you run this program)



   1:  snmpd.conf

   2:  snmptrapd.conf

   3:  snmp.conf



Other options: quit



Select File:

snmpd.conf

This file is what configures snmpd, or the daemon that answers snmp queries from other computers. Taking that option, we see:

The configuration information which can be put into snmpd.conf is divided

into sections. Select a configuration section for snmpd.conf

that you wish to create:



   1:  Access Control Setup

   2:  Extending the Agent

   3:  Monitor Various Aspects of the Running Host

   4:  Agent Operating Mode

   5:  System Information Setup

   6:  Trap Destinations



Other options: finished

Option 1, Access Control Setup is fairly self-explanatory, it's where we set up who and what can query this machine. Taking this option gives us:

Section: Access Control Setup

Description:

  This section defines who is allowed to talk to your running

  snmp agent.



Select from:



   1:  a SNMPv3 read-write user

   2:  a SNMPv3 read-only user

   3:  a SNMPv1/SNMPv2c read-only access community name

   4:  a SNMPv1/SNMPv2c read-write access community name



Other options: finished, list

Again, this article is not going to deal with SNMPv3 except for an explanation at the end that briefly summarizes it. It's different enough to need its own article. Therefore the two options we care about the most are 3 and 4. Note, that while you have to have a read-only community to really use SNMP at all, read-write is not a major requirement, although a lot of config utilities, most notably HP use RW SNMP. Since these community strings are used in a completely unencrypted fashion, don't use a community string that is the same as a password you actually care about. Since the options for both are the same, we'll just look at option 4.

The first item you're asked for is the community name. Enter whatever you like, just avoid spaces or illegal chars like / or :. The next question is for a hostname or IP address to accept this community name from. Unless you want any device on your network to be able to use this community string, you want to limit it to boxes actually using SNMP queries and applications. If nothing else, auditors LOVE to jump on SNMP setups where you're not controlling access to SNMP information. The next option will ask if you want to restrict the OIDs that this community string can see. Again, answer in the way that best fits your needs. If you're not sure, hit RETURN for all of them. You can always lock it down later if you need to, and again, SNMP v1/2 is really insecure anyway. This is just keeping the stoops out.

Once you've set your options for the read-only and read-write community strings, enter "finished" to jump back to the main snmpd.conf menu.

Item 3 in that menu, Monitor Various Aspects of the Running Host is how you set up the computer to check certain items, and send traps if parameters you set are violated. This is similar to jobs you can set up using say, cron or launchd, but is designed to notify a remote system if something goes "wrong". The menu is below:

Select from:



   1:  Check for processes that should be running.

   2:  Check for disk space usage of a partition.

   3:  Check for unreasonable load average values.

   4:  Check on the size of a file.



Other options: finished, list

The options are fairly clear, so we'll just take a look at option two, Check for disk space usage of a partition. Selecting option 2 brings us to:

Select section: 2



Configuring: disk

Description:

  Check for disk space usage of a partition.

    The agent can check the amount of available disk space, and make

    sure it is above a set limit.  

    

     disk PATH [MIN=100000]

    

     PATH:  mount path to the disk in question.

     MIN:   Disks with space below this value will have the Mib's errorFlag set.

            Can be a raw byte value or a percentage followed by the %

            symbol.  Default value = 100000.

    

    The results are reported in the dskTable section of the UCD-SNMP-MIB tree



Enter the mount point for the disk partion to be checked on:

Pretty self-evident. Enter the mount point for the drive, then enter the minimum amount of space that should be available on that mount point. Once you do that, you go back to the menu. Use any other options you need, then type "finished" to go back to the main snmpd.conf menu.

Option 4 on the main menu, Agent Operating Mode should be used carefully, as it can do some very odd things to your snmpd setup. If you aren't sure what you are doing here, leave it alone.

Option 5, System Information Setup, is where you enter in the location of the system, the contact information for the administrator, and the proper value for the sysServices object. The sysServices object is how you define what kind of services the computer you're setting up provides, and somewhat corresponds to the OSI seven - layer model. If you aren't sure about how to set this up, leave it alone and take the defaults, the work in a majority of cases.

Option 6, Trap Destinations, is important if you're trying to set up traps, as this is where you tell the system you're configuring, where to send traps. Selecting it brings you to this menu:

Select section: 6



Section: Trap Destinations

Description:

Here we define who the agent will send traps to.



Select from:



   1:  A SNMPv1 trap receiver

   2:  A SNMPv2c trap receiver

   3:  A SNMPv2c inform (acknowledged trap) receiver

   4:  A generic trap receiver defined using snmpcmd style arguments.

   5:  Default trap sink community to use

   6:  Should we send traps when authentication failures occur



Other options: finished, list



Select section:

Items 1-3 here are fairly straightforward. You enter in the hostname of your network monitoring server, a community to use, and an optional port number. Option 4 is for more advanced setup. Option 5 lets you set a default trap community, to be used if one is not explicitly provided, and option 6 is for sending traps if an SNMP auth failure occurs. Considering the insecure nature of SNMP v1/v2, this is not a bad idea. Once you're done here, enter "finished" to get back to the main snmpd.conf menu. Since this was the last option, enter "finished" again to get back to the main snmpconf menu.

snmtrapdp.conf

Since snmptrapd is the daemon that receives traps, this is where you configure that daemon. Obviously, there's no need to do so if the machine you're setting up won't need to act as a trap receiver. The main menu looks like this:

The configuration information which can be put into snmptrapd.conf is divided

into sections. Select a configuration section for snmptrapd.conf

that you wish to create:



   1:  Authentication options

   2:  Output formatting for traps received.

   3:  Logging options

   4:  Runtime options

   5:  Trap Handlers



Other options: finished



Select section:

Option 1 lets you set how the trap receiver deals with authentication traps. 1 or yes sets it to ignore authentication traps, 0 or no sets it to not ignore them. Option 2 lets you set the options for formatting incoming traps. Option 3 deals with how you log traps and where you log them to. Option 4 sets forking and PID file options, and option 5 lets you run a shell program or script when a trap is received. Option 5 is where you can get a lot of power out of trap reception. Depending on how much work you want to do, you can really automate the heck out of how you respond to problems on your network, just with what comes with the OS.

Once you're done here, enter "finished" until you get to the main snmpconf menu. Our last configuration file is snmp.conf, which is where you set defaults for the various SNMP commands we discussed earlier.

snmp.conf

The main snmp.conf menu is shown below:

The configuration information which can be put into snmp.conf is divided

into sections. Select a configuration section for snmp.conf

that you wish to create:



   1:  Default Authentication Options

   2:  Debugging output options

   3:  Textual mib parsing

   4:  Output style options



Other options: finished



Select section:

Option 1 lets you set the default authentication options for the SNMP commands:

Section: Default Authentication Options

Description:

  This section defines the default authentication

  information.  Setting these up properly in your

  ~/.snmp/snmp.conf file will greatly reduce the amount of

  command line arguments you need to type (especially for snmpv3).



Select from:



   1:  The default port number to use

   2:  The default snmp version number to use.

   3:  The default snmpv1 and snmpv2c community name to use when needed.

   4:  The default snmpv3 security name to use when using snmpv3

   5:  The default snmpv3 context name to use

   6:  The default snmpv3 security level to use

   7:  The default snmpv3 authentication type name to use

   8:  The default snmpv3 authentication pass phrase to use

   9:  The default snmpv3 privacy (encryption) type name to use

  10:  The default snmpv3 privacy pass phrase to use



Other options: finished, list



Select section:

Since we aren't dealing with SNMPv3 in this article, we'll talk about options 1-3. The first three options are pretty simple. If you don't want to use the default port of 161, you set that in with option 1. The default version number, 1/2c/3 is set with option 2. Normally, I just set it to 2c. If you have older equipment, you may have to specify -v 1 in your SNMP commands, but for the most part, 2c should work on any equipment you deal with . Option 3 lets you set the default community string for your SNMP commands. For SNMP v1/v2 usage, setting your authentication options lets you set up defaults for the -c and -v options. Enter "finished" from here to get back to the main snmp.conf menu.

I rarely mess with options 2 or 3, so we'll skip to option 4, which sets your -On defaults. In the output style options, set the defaults for how you want the returns from various SNMP commands to be formatted. Once you're done, enter "finished" until you get back to the main snmpconf menu.

Since we're finished, enter "quit", and it will create the files in /usr/share/snmp for you, (the -i option to snmpconf does this), and voila, you're set up to use SNMP. If you want to get an idea for syntax, take a look at the various SNMP conf files, and the man pages for them, they can be of great use in helping you really optimize your SNMP setup.

Getting snmpd started

If you're running Mac OS X Server, then you enable this with Server Admin, in the main Setting tab for the server itself, (not in any of the services).

If you're running Mac OS X 10.4 Client, then you add the SNMPSERVER=-YES- line to /etc/hostconfig, or if the line is in there and set to -NO-, then you set it to -YES-.

In Mac OS X 10.5 Client, since all of this is run by launchd, the easiest way to start snmpd is to just use Lingon. For Mac OS X 10.5, you have to use version 2.0.2 or later. If you want to use Lingon for launchd control in Mac OS X 10.4, (a really good idea), then you need version 1.2.1.

In Lingon, expand the System Daemons, and select "org.net-snmp.snmpd". You'll get a Very Dire Warning from Lingon about mucking around with System Daemons. Since we have to, we smile, nod, and move on. Make sure the "Enabled" checkbox is checked, and that "Keep it running all the time no matter what happens" and "Run it when it is loaded by the system (at startup or login) is selected. Save and exit Lingon.

You can check to see if snmpd started with ps -ax|grep snmpd. If it didn't, you may have to either reboot, or use launchctl to start snmpd. Once it's running, assuming your setup is good, you should be all set with SNMP on that machine. This brings us to the ultimate question:

"I have all this stuff set up, i know how to configure it, but what can I DO with it?" Well, let's find out, hmm?

Stupid SNMP tricks

Keep in mind these are all on Mac OS X 10.5. Mac OS X 10.4 on PPC doesn't have as many hardware monitoring OIDs as Mac OS X 10.5, and Mac OS X 10.4 on Intel barely has any SNMP at all. It's really broke for Mac OS X 10.4 on Intel, so if you want to use SNMP with Intel Macs, your choices are compile your own SNMP setup, or upgrade to Mac OS X 10.5.

Now, we all know about using SNMP to track stuff like network throughput, and the like. Bah, that's boring. I mean, you need it but geez, EVERYONE does THAT. What we want is cools stuff:

For example, want to see the basic hardware SNMP thinks you have?

hrDeviceTable

That's pretty handy, but suppose you have multiple drives on a remote server, and you want to see which item in the hrDeviceTable is the boot device?

Valkyrie:~ jwelch$ snmpwalk -Of localhost .iso.org.dod.internet.mgmt.mib-2.host.hrSystem|more

.iso.org.dod.internet.mgmt.mib-2.host.hrSystem.hrSystemUptime.0 = Timeticks: (1396937) 3:52:49.37

.iso.org.dod.internet.mgmt.mib-2.host.hrSystem.hrSystemDate.0 = STRING: 2007-12-4,22:13:36.0,-5:0

.iso.org.dod.internet.mgmt.mib-2.host.hrSystem.hrSystemInitialLoadDevice.0 = INTEGER: 1536

.iso.org.dod.internet.mgmt.mib-2.host.hrSystem.hrSystemNumUsers.0 = Gauge32: 2

.iso.org.dod.internet.mgmt.mib-2.host.hrSystem.hrSystemMaxProcesses.0 = INTEGER: 532

A couple of points:

  1. hrSystemNumUsers is based on ttys, so be careful how you take that number

  2. hrSystemMaxProcesses reflects the value you would see for kern.maxproc in sysctl

Want more information on local hard drives?

hrStorageTable

CPU load?

Valkyrie:~ jwelch$ snmptable -Cl localhost .iso.org.dod.internet.mgmt.mib-2.host.hrDevice.hrProcessorTable

SNMP table: HOST-RESOURCES-MIB::hrProcessorTable



hrProcessorFrwID        hrProcessorLoad

SNMPv2-SMI::zeroDotZero 11

SNMPv2-SMI::zeroDotZero 13

Disk information on the boot drive without having to do the math yourself:

Valkyrie:~ jwelch$ snmptable -Cl localhost .iso.org.dod.internet.mgmt.mib-2.host.hrDevice.hrDiskStorageTable

SNMP table: HOST-RESOURCES-MIB::hrDiskStorageTable



hrDiskStorageAccess hrDiskStorageMedia hrDiskStorageRemoveble hrDiskStorageCapacity

readWrite           unknown            false                  293036184 KBytes

Boot partition info:

Valkyrie:~ jwelch$ snmptable -Cl localhost .iso.org.dod.internet.mgmt.mib-2.host.hrDevice.hrPartitionTable

SNMP table: HOST-RESOURCES-MIB::hrPartitionTable



hrPartitionIndex hrPartitionLabel       hrPartitionID hrPartitionSize  hrPartitionFSIndex 

1                "EFI System Partition" "0xe000001"   0 KBytes         0

2                "Untitled"             "0xe000002"   187564032 KBytes 1

3                "Untitled"             "0xe000003"   105136236 KBytes 6

Note that the partition sizes may not perfectly match the Finder.

File system information, ala the mount command on a local machine, including bootable info:

Valkyrie:~ jwelch$ snmptable -Cl localhost .iso.org.dod.internet.mgmt.mib-2.host.hrDevice.hrFSTable

SNMP table: HOST-RESOURCES-MIB::hrFSTable



hrFSIndex hrFSMountPoint      hrFSRemoteMountPoint hrFSType                        hrFSAccess hrFSBootable hrFSStorageIndex 

1         "/"                 ""                   HOST-RESOURCES-TYPES::hrFSHFS   readWrite  true         31

2         "/dev"              ""                   HOST-RESOURCES-TYPES::hrFSOther readWrite  false        32

3         "/dev"              ""                   HOST-RESOURCES-TYPES::hrFSOther readWrite  false        33

4         "/net"              ""                   HOST-RESOURCES-TYPES::hrFSOther readWrite  false        34

5         "/home"             ""                   HOST-RESOURCES-TYPES::hrFSOther readWrite  false        35

6         "/Volumes/Untitled" ""                   HOST-RESOURCES-TYPES::hrFSNTFS  readOnly   false        36

7         "/Volumes/jcwelch"  ""                   HOST-RESOURCES-TYPES::hrFSOther readWrite  false        37

/Volumes/jcwelch is my iDisk, and I chopped off the backup date entries, since they don't really work consistently.

You can get a list of running processes...note that I'm showing you one entry from this list, as it's rather long:

Valkyrie:~ jwelch$ snmptable -Cl localhost .iso.org.dod.internet.mgmt.mib-2.host.hrSWRun.hrSWRunTable

SNMP table: HOST-RESOURCES-MIB::hrSWRunTable



hrSWRunIndex hrSWRunName   hrSWRunID               hrSWRunPath      hrSWRunParameters   hrSWRunType hrSWRunStatus 

1            "launchd"     SNMPv2-SMI::zeroDotZero "/sbin/launchd"  ""                  unknown     runnable

The hrSWRunIndex corresponds to the PID of the process.

The CPU time for PID 1:

cpu time for PID 1

The RSIZE for PID 1:

rsize for PID 1

That's just getting numbers. You start factoring in software like Nagios and Cacti, and you can do some neat things with SNMP info. For example, I was able to take the print jobs number for some workgroup multifunction printers we have, and by having Cacti show it as a counter and a gauge, I get not only a graph of the total print jobs on any given MFD, but I also have a graph of print frequency, so we can see when the printer is the busiest. It's a great way to justify new printers.

However, I've already dealt with Nagios, and Cacti is another article in and of itself, so that will wait for next time.

SNMPv3

As I said earlier, one of the big issues with SNMP v1/v2 is that it's completely insecure. Everything is done via plain text, including community strings, (read: "passwords"). That's kind of bad when you consider the kind of information you can get from SNMP, so SNMP v3 was created. It allows for encryption, and far better security in general than earlier versions. The problem is, it's only recently that v3 support is becoming ubiquitous. That's not to say that SNMPv3 should be avoided, but that you may not be able to implement it everywhere, especially where you have older equipment. If I get enough requests, I may go into setting up v3 in a separate article at some point.

Conclusion

As lengthy as this article is, I've barely scratched the surface of what you can do with SNMP, but this should give you a solid idea of what's involved with it, and what you can do with it.

For books, there are two that I use all the time, and are my primary references for SNMP in daily practice, and for this article:

Essential SNMP, Second Edition, by Douglas Mauro and Keven Schmidt from O'Reilly, and SNMP, SNMPv2, SNMPv3, and RMON 1 and 2 (3rd Edition), by William Stallings. Note that the Stallings book is more of a college-level text book. It goes into tons of detail, but is not a casual read by an stretch.


Technorati Tags:
, , , ,


Posted by John C. Welch at 00:27 | Permalink


Comments

Warning for Notes users: The commenting system uses HTML.
I know this will be scary for some of you, especially Notes fans. However, open standards, rah-rah.
If you want to use less-than or greater-than signs, or other similar charachters that HTML reserves,
you'll simply have to learn to do it the HTML way. Luckily, HTML is kind of popular, no matter what
your re-educators have told you, and you can easily find help on the intertubes.

Jeez dude, why didn't you just post this on Twitter, where it belongs.....

Posted by: W. Ian Blanton | December 5, 2007 1:42 AM

Seriously though...this is one of those "Save as Webarchive" posts so I can dig through it with the time it deserves.

Posted by: W. Ian Blanton | December 5, 2007 1:44 AM

It was too short for twitter

Posted by: John C. Welch Author Profile Page | December 5, 2007 4:25 PM

Found a minor mistake:

If they don't, then the MIB does you know good.

should be:

If they don't, then the MIB does you no good.

Posted by: allan McCoy | December 11, 2007 2:01 PM

Fixed. Thanks for the catch Allan.

Posted by: John C. Welch | December 12, 2007 7:40 AM

Thank you for your excellent article! Finally I got my snmp service start up on Mac OS X 10.5.
I have a question, however, I tried to execute the following command:
snmpget -On localhost .1.3.6.1.2.1.1.3.0
The program returned with an error: No community name specified.
Then I added -c public to the command, it worked well :-)
I notice that you never specify a community name in your command. Is there a way to specify a default commnunity?
I'd appreciate your answer very much. Thanks.

Posted by: Chris | December 27, 2007 9:35 PM

Chris,

The way to set a default command is to use snmpconf, and configure the default community for snmp.conf, so that you don't have to specify the community in the command string.

Posted by: John C. Welch | December 28, 2007 8:50 AM

John, is it possible to use SNMP to configure my mac to use 802.11a APs when they are available, instead of 802.11b/g ones? I can see no obvious way to do this in the Network Preferences. Thanks!

Posted by: Carl Youngblood Author Profile Page | April 10, 2008 1:57 PM

Post a comment

Thanks for signing in, . Now you can comment. (sign out)

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)


Remember me?


digital.forest Where Internet solutions grow

There, a PayPal Button.

 
Family
The Artwork of Melissa Findley
Diane Francis @ the National Post Eric Francis @ the Calgary Sun

Apple Amazon Links
Apple Mac OS X Server 10.5 [Unlimited]

Apple Mac OS X Server 10.5 [10-Client]

Apple Mac OS X 10.5 Leopard

Apple Mac OS X 10.5 Leopard [5-User Family Pack]

Amazon Book Links
Legacy of Ashes: The History of the CIA

The Donnas: Bitchin'

Wizards at War (The Young Wizards, Book 8)

The Demon's Sermon on the Martial Arts

The Collected Stories of Arthur C. Clarke

JavaScript and Ajax for the Web, Sixth Edition

Awakening Warrior: Revolution in the Ethics of Warfare

FOB Links

Mac Web Writers

Techie Links

Review Victims