Early UNIX systems had no centralized logging system, leaving each application to decide on its own log policies. Log-related decisions included where to place logfiles, how much information to store, how long to store it, and whether to warn any or all logged in users of any particular events/conditions. Of course most applications simply logged to a text file and didn't do any alerting or log file rotation, but that's unimportant. ;)

One of the early UNIX hackers at UC Berkeley took note of the situation, and added a system logging facility to BSD UNIX. It should be noted that if he had forseen the impact of this impromptu system, he certainly would have designed it differently (at a minimum he would surely have included a year field in the log file format).

The way this centralized logging worked was that a program used the system API (a UNIX system call) to log information. The application using the system call (syscall from here on) included information on what subsystem the information was coming from (i.e. ftp, mail, kernel, etc) and also information on the importance or severity of the information (i.e. informational, critical failure, debugging information, etc). Armed with this information, the UNIX system manager could configure a centralized logging mechanism to log according to a single set of guidelines (where to place logfiles, how much information to store, etc).

The original use of a syslog daemon (syslogd) on UNIX was to collect log sent via the syslog syscall(s) (there are usually two similar syslog syscalls, but that's not important here). A single configuration file, usually stored at /etc/syslog.conf was used to configure it. The information from the syscall(s) arrived via a UNIX filesystem socket, usually at /dev/log. There was no network transmission originally involved, simply some local collection and storage of log information.

The traditional use of a syslog daemon is to store events in log files, in a format like this:

It is made up of two main parts:

1. The "header" contains the date/time information and the originating system's hostname (or IP)
2. A two-part message. The two parts of the message are: the program or subsystem name and the message itself. The program/subsystem part (when present) is followed by a colon and often includes the UNIX process ID (or PID) in square brackets to differentiate between multiple instances of the same program. In the message above the program or subsystem name is "foo". Other values commonly seen are long-running UNIX maintenance programs such as cron, network server programs such as sshd, and UNIX kernel modules/subsystems such as network interface information and packet filtering logs.

Sometimes the message has no program/subsystem information - it might contain a message from the syslog program itself (though this is certainly not the only instance when the information might be missing):

This can cause problems with identifying fields in a syslog message when transmitted over the network. The program/subsystem field (called the "TAG" field in RFC 3164) is not required.

Syslog transport format, e.g. when arriving via a UNIX socket or from the network (TCP or UDP) is (in theory) the same as the on-disk format, except with the information about subsystem and importance included. This is called the "pri" field, and is made up of an integer surrounded in angle brackets. The "pri" field comes at the beginning of the message. The subsystem information is called a facility, of which there are 15, and the importance information is called a severity, of which there are 8.

Here is an example syslog message sent over the network:

Note that when the day of the month is a single digit, then it is formatted as a space then the day as shown above.

Remember that a syslog daemon receives locally generated messages over a UNIX socket, which isn't much different than receiving messages over the network.

1. Since messages are single line only, there's no easy way to correlate related messages that span multiple lines (java stack traces for instance).

   A log parsing application can be programmed to analyze it properly, but this really has nothing to do with syslog. It is unfortunate that events have to be limited to a single line and (normally) fewer than 1024 characters.

2. Many syslog programs, when configured to relay messages on to another syslog program on another host, will leave out certain parts of the syslog message - complicating proper identification of certain fields.

   Imagine this common scenario:

   The syslogd program on a Solaris machine is configured to send all logs to a central syslog server. When Solaris syslogd sends logs over the network, it doesn't include the hostname field at all. This means that a "last message repeated" message looks more like this:

   <13>Jan 18 12:34:56 last message repeated fifteen times.

   How do you ("you" being syslog server software) know whether or not the originating hostname is "last"? Well if you only have Solaris servers sending syslog messages, then you know that you simply need to use the IP/hostname of the sending system in the hostname field. But what if you have other systems sending logs? When a message lacks a program/subsystem name followed by a colon it can be hard for a syslog server to know which field is which. This is a strong reason to use syslog-ng on clients and servers, as syslog-ng always relays a complete message across the network to a remote syslog server. Other modern syslog servers might do the same.

   The sysklogd program used as a syslog server for many Linux distributions also leaves out fields. It leaves out the time/date information and the hostname information (the entire "header").

   <13>last message repeated four times

   This can cause problems as well, but is much less error-prone than the way Solaris (and OSF1/Digital UNIX, maybe others I don't know about) leaves out just the hostname.

3. Many syslog programs, when configured to relay messages on to another syslog program on another host, will format the message in a non-standard format.

   When a syslog message arrives that uses a non-standard format, it is normal (and expected) for the receiving syslog server to prepend the date/time and hostname information in the correct/standard format.

   If a Cisco switch sends a message like this:

   2005 Aug 23 03:04:05 UTC +00:00 %PAGP-5-PORTFROMSTP:Port 4/16 left bridge port 4/16

   ...it'll be written to disk like this:

   Aug 23 03:04:05 switch.company.com 2005 Aug 23 03:04:05 UTC +00:00 %PAGP-5-PORTFROMSTP:Port 4/16 left bridge port 4/16

   Cisco's log format is well documented (not that their devices always send messages conforming to the documentation), but it is not consistent with traditional syslog format. For this reason messages from Cisco devices are rewritten by syslog servers when received over the network.

We've already discussed the format of syslog messages, but not the network transport protocols or the roles played by syslog client and server software in typical syslog architectures. Much of this information is also contained in the informational BSD syslog RFC, RFC 3164 (not to mention some of the rewriting behavior described in the format section). I've purposely reused the same terminology. Please see the RFC for a more complete discussion of possible architectures.

1. syslog device

   The system where an event is generated. This might be a general purpose OS such as Windows or UNIX, a printer, a network device, etc. This device sends a text string conforming to syslog format across the network to a listening syslog process on another host.

   Note that this can be as simple as using the popular netcat program from the UNIX or Windows command line (NT/XP/Win2k[3] usage would be slightly different at the CMD prompt, but similar):

   $ echo "<13>Jan 4 12:12:12 host foo[345]: a syslog message" >/tmp/foo
$ nc -v -u loghost.example.dom 514 < /tmp/foo

   The above is a valid syslog device/client. Of course there is no long running process producing the message (as there would be with most other messages), no syslog server software package involved, but the remote syslog server/relay receiving the message has no way of knowing this. In fact some simple syslog mechanisms that by design log to the local system, rely on the local syslog daemon listening on a network socket for incoming messages. They send events to localhost port 514/UDP.

   Part of the reason for the success of syslog as a network event logging mechanism surely comes from its simplicity. This is similar to the success of other simple network communication mechanisms such as SNMP, HTTP or even TCP/IP itself.

2. syslog relay

   A system running syslog server software which receives events via syslog over the network, then acts as a syslog client by relaying it onto another system is considered a relay. Not all relays fit such an inflexible definition, to quote RFC 3164:

   Relays may send all or some of the messages that they receive to a subsequent relay or collector. In the case where they do not forward all of their messages, they are acting as both a collector and a relay. ...

   Relays may also generate their own messages and send them on to subsequent relays or collectors. In that case it is acting as a device. These devices will also be designated as a relay in the following diagram.

3. syslog server

   For this discussion a syslog server is the point in the chain where syslog transmission stops, i.e. no syslog client behavior is used on the host. At this point the logs might be stored to disk, stored to disk and transmitted to a security management application on another host (via TCP or UDP) or on the same host, directly piped to another application on the same host, injected into a database on a remote server via a database client, or perhaps even discarded. The point here is that the message is not relayed on via syslog.