A Quick SNMP Primer

Just a quick disclaimer to go with my quick primer: I'm relatively new to SNMP myself. This article doesn't aim or claim to be comprehensive. It's just meant to help you get started.

SNMP is the Simple Network Management Protocol. It has been said that not only is it not simple, but it's usually used for monitoring more than managing devices. I don't fully agree with this sentiment. It does have some weirdness, but once you get over that it's not so bad.

Structure of Management Information

Before we get into practical usage, we need to talk theory. SNMP partly describes a network protocol, but it also describes an organizational structure. This structure is called the Structure of Management Information, or SMI for short. This is a tree that branches out into various nodes called Object Identidiers, or OIDs. Each level of this tree has its own set of numbers for each node. At the moment, we're only concerned with a specific path in this tree, sometimes referred to as MIB-II, or the Management Information Base. The OID for this node is 1.3.6.1.2.1.

This node is actually the root of where our queries are performed. In fact, a lot of times this OID is just printed as mib-2. The nodes underneath this are what we will be querying.

Some of the nodes that we're interested in here are system (1), interfaces (2), at (3), ip (4) and so on. For example, the OID that describes a system is mib-2.system.sysDescr.0. The long name for this is .iso.org.dod.internet.mgmt.mib-2.system.sysDescr.0, and the numerical notation for this is .1.3.6.1.2.1.1.1.0. On my Linux box, this is often the output of the "uname -a" command.

If you need some time to digest this part, go for it. I think this is the biggest thing to overcome with SNMP. When you're ready, come back for the next part.

SNMP Versions

There are three main versions of SNMP that you will run into: 1, 2c and 3. Version 2c is the community version of version 2, and while it does provide a little extra information, it isn't much of an improvement over version 1.

These two versions use passwords called community strings to provide access. There is a read-only community used for monitoring (reading values from the device), and a read/write community used for management (setting values on the device). The most common default community string for read-only access is "public". The most common default community string for read/write access is "private".

Most SNMP-enabled devices have these as their defaults, and as we all know, most sysadmins are still too lazy to change the defaults. Fortunately, the Net-SNMP package in Linux has a little bit better default configuration. It is set to only listen to requests from the local machine. Even better, the Net-SNMP server isn't installed by default on most Linux distributions, so don't worry about your system being vulnerable like that out of the box.

SNMP version 3 introduced the concept of usernames and passwords, rather than just communities. But versions 1, 2c and 3 all suffer from one major shortcoming: they send their authentication and their data in the clear. Fortunately, version 3 also supports encryption for either authentication or data transfer or both.

Setting Up SNMP

Installation is easy, but there are a couple of different packages that you want. On RHEL5 and Fedora 10:

yum install net-snmp net-snmp-utils

On Ubuntu:

apt-get install snmpd snmp

For those of you who are interested, net-snmp == snmpd and net-snmp-utils == snmp.

Once you have it installed, go ahead and blow away the configuration file:

mv /etc/snmp/snmpd.conf /etc/snmp/snmpd.conf.orig

The default configuration is a lot to deal with. We're going to keep it simple. Create a new /etc/snmp/snmpd.conf file with the following line:

rocommunity yourcommunitystring

...where yourcommunitystring is, of course, whatever community string you want to use. This is really all the configuration you need for versions 1 and 2c, though you can add more if you like.

If you want version 3 instead, the configuration is a little different. There are actually two files to edit here, and the second one probably doesn't exist by default. First, edit the /etc/snmp/snmpd.conf file and add the following line:

rouser username

...where username is of course the username that you want to use. You'll also need to set up a password. Of course, we would never want to use a plaintext password if we didn't want to. Depending on your version of Net-SNMP, we have a few options available to us. We can use either MD5 or SHA for authentication, and DES (or AES in more recent versions) for data transfer. I'm going to assume that you want to use SHA and DES. Create if necessary and edit the /var/net-snmp/snmpd.conf file and add the following line:

createUser username SHA password1 DES password2

...where username is of course the username that you specified in the /etc/snmp/snmpd.conf file, password1 is your SHA password and password2 is your DES password. I know it seems kind of scary to leave passwords out in the open in plain text files like this, but don't worry; as soon as SNMP starts up, it will rewrite this file and encrypt the passwords. I should also note that this is the only time you will ever edit this file by hand, and the createUser line is the only line that you will ever add to it.

When you're ready to start the SNMP service, type in:

chkconfig snmpd on (RHEL/Fedora systems only)
/etc/init.d/snmpd start

Viewing SNMP Data

There are several commands available for viewing SNMP data, but I'll start you off with just the snmpwalk and snmpget commands for now. Once you get the hang of them, you can start exploring other commands. Usage does differ slightly between versions 1 and 2c, and version 3, but only in authentication.

If you added the rocommunity line to your /etc/snmp/snmpd.conf file, use the following command to take a look at your MIB-II tree:

snmpwalk -v 2c -c yourcommunitystring localhost | less

We're piping the output of the command to less because there can be a lot of information here. You've probably guessed that the -v option specifies the version (1 or 2c) and the -c option specifies the community string. The first line probably looks something like this:

SNMPv2-MIB::sysDescr.0 = Linux bourdain 2.6.24-22-generic #1 SMP Mon Nov 24 18:32:42 UTC 2008 i686 GNU/Linux

Let's make the names little more verbose. Hit 'q' to exit out of this, and try this command:

snmpwalk -v 2c -c yourcommunitystring -Of localhost | less

For your reference, that is a capital letter O, not the number zero. When you look at the output of this command, it probably looks more like this:

.iso.org.dod.internet.mgmt.mib-2.system.sysDescr.0 = Linux bourdain 2.6.24-22-generic #1 SMP Mon Nov 24 18:32:42 UTC 2008 i686 GNU/Linux

Hey, you want to really have some fun? Try out this command instead:

snmpwalk -v 2c -c yourcommunitystring -On localhost | less

Now you get the name in numerical format:

.1.3.6.1.2.1.1.1.0 = STRING: Linux bourdain 2.6.24-22-generic #1 SMP Mon Nov 24 18:32:42 UTC 2008 i686 GNU/Linux

The amusement never ends. Going back to the full naming format, you'll notice that the first set of lines is in the system node:

.iso.org.dod.internet.mgmt.mib-2.system...

If you scroll down a little, you'll find the interfaces node:

.iso.org.dod.internet.mgmt.mib-2.interfaces...

We can specify to walk only one of these nodes if we like. Since we're only looking at information under the mib-2 node, we can actually just refer to it by that one group name:

snmpwalk -v 2c -c yourcommunitystring -Of localhost system | less

Now you will only see information in the system group. If you know the exact location of a value that you want to look at, you can use the snmpget command to look at it. The syntax is almost identical to snmpwalk:

snmpget -v 2c -c yourcommunitystring -Of localhost system.sysDescr.0

Now that you know how to get information using community strings, let's switch back to SNMPv3. When we specify "-v 3", we must specify the -u option with a username, instead of the -c option. We also need to let it know what passwords we're using, and how to encrypt them. Your basic snmpwalk command will look something like this:

snmpwalk -v 3 -u username -a SHA -A password1 -x DES -X password2 -l authNoPriv localhost | less

If you were able to follow along okay when we set these passwords, then most of this line already makes sense to you. The big things to remember now are what options go with what values, and the security level.

The "-l" option can specify one of three levels: noAuthNoPriv, authNoPriv or authPriv. If for some unknown reason you decide not to use passwords, you would set this to noAuthNoPriv. Otherwise you would use authNoPriv for read-only access or authPriv for read/write access.

I'm not going to bore you with the output of this command, because you've already seen it. You can use -Of or -On to modify the output, you can specify a group, you can do pretty much whatever you were doing with v1 or v2. The only real difference is authentication. And yes, the snmpget command looks pretty much the same.

In fact, once you have a handle on these couple of commands, this would probably be a good time to check the man page for snmpcmd. There is not actually a command called snmpcmd, this page just shows you the options that are common to the whole suite of Net-SNMP commands.

Hopefully this gives you a good foundation for playing with SNMP on your Linux box. Of course, there are volumes written on the subject, and my little artcle doesn't even attempt to cover what they will. But hopefully they will get you over any initial fears about it and get you up and running.

Monitoring Apache Processes with Cacti

I've been setting up Cacti at work to monitor our servers. Last night it occured to me that it might be useful to monitor Apache processes on our web servers. Cacti already has a graph built in for monitoring processes in general, but I just wanted those that matched "httpd". It was a simple thing that I wanted, but setting it up proved to be non-trivial, so I thought I'd share how I set it up. Keep in mind that this is on a RHEL5.2 box. Your distribution may vary.

First of all, you need a script to gather the information. You already have one mostly built, so use it as a template:

cd /var/lib/cacti/scripts/
cp unix_processes.pl unix_apache_processes.pl

Edit that file so that it looks like this:

#!/usr/bin/perl

open(PROCESS, "ps ax | grep httpd | grep -c : |");
$output = ;
close(PROCESS);
chomp($output);
print $output;

Now you need to set up Cacti to use that script. Go into the Console tab in Cacti and click on "Data Input Methods". Click "Add", and use the following values:

Name: Unix - Get Apache Processes
Input Type: Script/Command
Input String: perl /scripts/unix_apache_processes.pl

Save it, then go back in and click on "Add" for "Output Fields". Use the following values:

Field [Output]: proc
Friendly Name: Number of Apache Processes
Update RRD File: (checked)

Go ahead and save it. Now go into "Graph Templates" and find the "Unix - Processes" template. Put a check next to it, go find the drop-down box next to "Choose an action", set it to "Duplicate" and click "go". Change the title to "Unix - Apache Processes" and save it.

Go back into "Graph Templates" and click on the "Unix - Apache Processes" template that you just created. Find everything that says "Processes" and change it to say "Apache Processes", then save it.

Go into "Data Templates" and find "Unix Processes". Duplicate it like you did with the Graph Template, calling the new template "Apache Processes". Then go back into it and change the "Data Input Method" drop-down to "Unix - Get Apache Processes".

Go back into "Graph Templates", then "Unix - Apache Processes" again and click on "Item # 1". Change the "Data Source" drop-down to "Apache Processes - (proc)" and hit "save". Do the same for "Item # 2", "Item # 3" and "Item # 4".

Now when you go into the "Devices" menu and click on one of your web servers, you should see "Unix - Apache Processes" show up on the drop-down box for "Associated Graph Templates". Go ahead and add it, then go up to the top of the device page and click "Create Graphs for this Host". Put a check next to "Unix - Apache Processes" and click "Create".

Setting up this graph may be pretty straight-forward for the average Cacti user, but there are a lot of steps, and it helps to do them in the right order. This walkthrough should give you a good start for adding more complex graphs to Cacti when you need to.

Pork Enchiladas

I love enchiladas. I remember my dad making them when I grew up. Of course, his were a little different than mine. For instance, he used corn tortillas. And he seemed to like cheese-only enchiladas. I kind of liked 'em like that. But I've discovered that my wife doesn't seem to like cheese-only enchiladas. And I got fed up with corn tortillas long ago. They crack when you try to roll them, they... well, they crack. Isn't that bad enough?

Lately I've been playing with enchilada sauce. Really, isn't it just a thinned out tomato and chile sauce? I decided to try a little experiment: canned tomato sauce plus taco seasoning. Hey, it's actually not bad. It's a little thick, but it's very close in flavor to some of the store-bought enchilada sauces.

So bearing all this in mind, let me share with you an enchilada recipe that my family has been eating on occassion lately. It's not exactly traditional, but we like it. But if you want to go back to corn tortillas or something, go for it. You won't hurt my feelings.

First off, you'll need your enchilada sauce. Whether you use the "real" stuff or you do it my way, it helps to heat it up. I like to use a wide frying pan, to have enough room to dip the tortillas in later. With my version, you need a can of tomato sauce and a couple of Tablespoons of taco seasoning.

Stir it all together and heat it up to about a simmer. You don't need to boil it or anything, it just needs to be heated a little. If you want, you can thin it out with a little bit of liquid. Water would be fine, but broth would be better. Once it's ready, put it together with the rest of your ingredients and equipment.

We have cheddar cheese, chopped pork, sliced olives, red bell peppers, an 8x8 baking pan, four 8-inch flour tortillas, and a pan full of enchilada sauce. Just a note: about one enchilada in, we put the baking pan next to the enchilada sauce. Step one, coat a tortilla with sauce:

Move it to the baking pan and add a hanful of chopped pork:

Olives go in next:

Chopped red bell peppers:

And of course, some cheese:

Roll it up and repeat with the rest of the tortillas. You'll want to use all of the pork inside, but be sure to save some of the olives, peppers and cheese for the top. At some point, it's going to look kind of like this:

Go ahead and cover it with a bit of cheese:

Add the remaining olives:

And the remaining peppers:

Slide it into the oven at 350F, until the cheese starts to bubble. It'll probably take about 15 minutes or so.

Oh, so tasty.

Fun at Work

I'm afraid of being found out.

Seriously, I don't know that I've ever had so much fun at work. Don't get me wrong, I loved working at Guru Labs, but the situation is different here. Let me explain.

At Guru Labs, my bosses seemed to have very realistic expectations of me. They hired me, fully knowing that I had gaps in the technical knowledge that they needed me to have. The promise was that they would help fill those gaps, and I would be ready to teach by the time I needed to. If I was not, then they would give me the opportunity to work elsewhere.

I met the challenge, and before long I was knee-deep in Linux knowledge that I never expected to obtain. There were always new things to learn, and I was always surrounded by people willing to teach me. At the same time, I got to be that person for about a couple dozen students a month. I was having a blast, and I wasn't sure I ever wanted it to end.

There was just one problem. As much as there was to learn, it sometimes seemed like there was never time to learn it. There were always classes to teach, and there was course ware to write, work to be done. I was constantly surrounded by cool new technologies that often felt barely touchable. And suddenly, it was gone, and I was left to look for another job which could never hope to be as much fun.

Fortunately, I had prospects in the queue. By the end of the first week I had applied at close to two dozen companies, interviewed at three of them, and had an offer from one of them. By the end of the second week I had already started work at my new company, and was still politely turning down interviews.

I had actually turned down an offer from this company at the same time I originally accepted an offer from Guru Labs. I had a friend that worked there, but I still had to get hired on my own. When I started, I think the idea was for me to watch my new coworkers and get a feel for what needed to be done. They were programmers who had been doing system administration when it was called for, but it wasn't their thing. Thanks to Guru Labs, I was well-trained in system administration and I just kind of took over.

This is a company who was small (maybe 20 employees) when I first interviewed, and who is now in the process of growing into a larger company (60+ employees at this location, plus the warehouse, plus the call center in Chicago, etc), but who still has the kind of server setup that one might expect at a small company. I have taken it upon myself to make the servers enterprise-grade.

This means I get to play with all of the technologies that I didn't really get to play with before. Well, not all of them, but increasingly more. One of my current projects involves four RHEL 5.2 virtual machines installed on my notebook using KVM, which I am using to test out different MySQL Cluster configurations. This week I also started playing with Cacti, and Nagios is on the horizon.

One day I looked at our current backup server and decided that it needed more hard drive space. Really we needed a brand new backup server, but I was willing to settle for a new hard drive. My boss came in to talk about it, and by the time he left I was pricing out components for a brand new backup server. This is not the first time I've had a boss that knew less than me about what I did, but it's definitely the first time a boss has not only admitted it, but taken my suggestions on what needs to be done.

To be honest, I'm not used to this. I was hired based on technical merit and experience, and then given what feels like free reign to do whatever I want. I don't really have free reign, but it feels like it. My boss realizes that I know what needs to be done, and he's willing to let me do it. He doesn't have some kind of personal agenda, he's not caught up in a power struggle, there's none of that. He's trusting to me do what he hired me to do, and his only concern is that I stay focused on the best interests of the company.

It's so weird. Nearly every day when I drive home from work, I am elated about what happened that day and what's going to happen the next. Actually having my opinion listened to on such a regular basis makes me feel like a rock star. And yes, I'm a little worried that I'm going to be found out. Every so often I wonder if my boss is going to realize how much fun I'm having actually doing my job, and start charging me admission.

I still miss Guru Labs. I certainly wouldn't be in the position I am now without the expert training and unparalleled opportunities that I received there. But I guess fate (if you believe in that sort of thing) decided that it was time for me to cut my teeth elsewhere. I guess it truly is bitter-sweet.