Understanding and configuring logging takes about 30 minutes of training and a couple minutes per device.  While that is an effort, think of it as an investment that will pay off later if there’s ever a severe network problem.  Even for a skilled and knowledgeable network engineer, trying to troubleshoot past network events without logs is very difficult.  Even fancy network monitoring tools will show what and when, but not why or how.

Step 1: Set the Clock

Before worrying about what time it is, first decide on a timezone setting.  By default, devices use UTC and networks with international staff may chose to keep it that way.  However, most would prefer to see the device’s local timezone as it’s more understandable to the average human.  Here’s an example to set a device to Pacific Standard Time and support Daylight Savings:

(config)#clock timezone PST -8
(config)#clock summer-time PDT recurring

Now that that’s configured, we can exit config mode and set the clock.  After all, what’s the point of having logs if you don’t know when they happened?  Note that time is entered in 24-hour format, for example 10 PM = 22 hours.

#clock set 22:33:44 MAR 26 2013

Ideally, you’ll want to have an NTP server available to set and sync time automatically.  My recommendation is having the core or edge routers sync from an server, then  have all devices point to that.  Pointing to an HRSP, VRRP, or GLBP address is ideal since you can point to a single IP yet still have redundancy.

(config)#ntp server 192.168.0.1

It will take 2-3 minutes for the device to fully get sync’d.  You can verify NTP sync status with the “show ntp status” and “show ntp associations” commands.  “debug ntp packet”. NTP is not required, but remember that the clock will lose time when the device is unplugged, so it’s a very good idea.  If it’s impossible to have an NTP source for your network, chose the 2 most reliable devices on your network and set them as NTP peers – they’ll keep each other in check should one of them reboot.

Finally, determine how you’d like to see timestamps on logging and debugging output.  The default behavior is to show the device’s uptime – not very useful in the real world.  Here’s an example where both logging and debugging output shows the date and time with the local time zone:

(config)#service timestamps debug datetime localtime show-timezone
(config)#service timestamps log datetime localtime show-timezone

On the PIX/ASA/FWSM platforms, this command is used instead:

(config)#logging timestamp

OK, now you’ve got nice looking, easy to read log messages.  Exit config mode and type “show logging”.  You should see something like this:

Mar 26 09:34:34 PST: %SYS-5-CONFIG_I: Configured from console by j5

Step 2: Understand logging severities

Before worrying about the logs itself, it’s important to understand the logging levels, or severities as they are called.  Severities are numbered 0 to 7, with the 0 being the most important events and 7 being the least.  Cisco explains the severity levels as follows:

debugging      Debugging messages                (severity=7)
informational  Informational messages            (severity=6)
notifications  Normal but significant conditions (severity=5)
warnings       Warning conditions                (severity=4)
errors         Error conditions                  (severity=3)
critical       Critical conditions               (severity=2)
alerts         Immediate action needed           (severity=1)
emergencies    System is unusable                (severity=0)

The severity will always be mentioned in the log message.  For example, in the “SYS-5-CONFIG” message above, the severity was 5.  This makes sense, because a configuration change is a “normal but significant” condition – it’s worth logging, but doesn’t necessarily indicate a problem.  Something more critical, such as a module or fan failure, would be severity level 2, and so on.

 Step 3: Set the VTY logging

Now that we understand the severity levels, we can make intelligent decisions on what to log where.  If I’m logged in to a device via telnet, SSH, or rlogin, I can set the max severity displayed using the “logging monitor” command.  By default, the device will show all messages when I use the “terminal monitor” and “no terminal monitor” commands.  There’s probably no need to change this, since it’s useful to see debug output in real-time on your screen.  Some will make, but since Telnet, SSH, and Rloging are all TCP-based, there should theoretically never be any messages lost or performance overload.  However, you can avoid showing debug output in your sessions with this command:

(config)#logging monitor 6

Step 4: Set the internal memory logging

Seeing logs in real-time is important, but what about past events?  The most basic form of past event logging is called internal buffer logging, or logging to memory.  The system allocates an amount of RAM (by default, 4 KB) and will log severity 0-6 messages there.  You can view the content with the “show logging” command.

4KB is a ridiculously small amount given the fact that modern network equipment has several hundred, sometimes gigabytes of memory installed.  It’s good to set the amount big but not too big – after all, you’re probably only interested in logs from the last few hours or days, not weeks or months ago.  I like to use 256KB.  Remember that 1 KB = 1024 Bytes, so 256 x 1024 = 262,144

(config)#logging buffered 262144

Like terminal logging, by default, all messages will be sent to internal memory, including debug.  Some may prefer to not send debug output to memory, since it can overwrite more important events:

(config)#logging buffered 262144 6

Note – be sure to re-state the buffer size when changing severity.  If the size is not stated, it will return to the default size of 4KB!

The “logging history” command works the same way as internal buffer logging, but saves the output in a different format and has a limit of 500 events.  I find it’s useful to use “logging history” for events of a greater severity than set in the “logging buffered” command:

(config)#logging history 5
(config)#logging history size 500

Unfortunately, both the internal buffered logging and logging history are erased when the device reboots, or when someone changes their logging settings.  For that reason, it’s a good idea to configure remote logging to a syslog and/or logging to flash memory, if supported.

Step 5: Log to a syslog server

Unix, Linux, and BSD systems ship with a syslog server pre-installed, however you’ll need to verify it’s been enabled and is configured to accept remote connections over the network.  In FreeBSD for example,  /etc/rc.conf will need this line:

syslogd_flags=”-n”

And syslog will need a restart with the “/etc/rc.d/syslogd restart” command.

When dealing with syslog servers, it’s useful to know the default syslog facilities:

Routers and switches – local7
Firewalls – local4

You can override this with the “logging facility” command on the devices.   On the syslog host, you’ll to take a look at the syslog configuration and perhaps modify it to put network devices in a separate logfile.

local6.* /var/log/switches.log
local5.* /var/log/routers.log
local4.* /var/log/firewalls.log

Windows users may want to look at the Kiwi Syslog Server by SolarWinds as a paid option.

Step 6: Downgrade console logging

By default, Cisco devices log everything (including debug!) to the console port – even when there’s no active session.  On older platforms especially, this is a problem as serial console output messages are CPU intensive.  You really want to reserve console messages to things that indicate a major event or problem.  Besides, once logging to internal memory and syslog server(s) has been configured, the only time consoles are useful is when doing maintenance or “Network Down” emergencies.

On routers and switches, I’d recommend only logging warnings and

(config)#logging console 4

On the PIX/ASA platform, it should be set even lower, as severity 3 & 4 messages are quite common:

(config)#logging console 2

The final thing to consider when logging to console is exception output.  This sets a limit (in bytes) to the amount of output to the console in a a software crash.  By default, it is 4 KB.  If you’re fortunate enough to have console servers that record everything, it may be useful to bump it up so you can send as much information to Cisco as possible to determine the cause of the crash.  Like this:

(config)#logging exception 32768

Step 7: Logging to Flash?

A new but very useful feature is the ability to log messages to flash memory.  Flash memory offers a “best of both worlds” option – it does does not lose its contents during a reboot, but can still capture logs if the network is inaccessible.  In IOS 15, this is configured by the “logging persistent” command:

(conf)#logging persistent url flash:/logs size 262144 filesize 65536

I now see a “logs” directory has been created, with a file in it:

#dir flash:logs
Directory of flash:/logs/
13  -rw-         282   Apr 6 2013 14:24:02 -07:00  log_20130406-142403

Even after rebooting a device, I can still see exactly what occurred

#show logging persistent  | inc RELOAD
Apr  6 14:27:07 PDT: %SYS-5-RELOAD: Reload requested by j5. Reload Reason: Reload Command.