Home |  Syslog-ng FAQ

Jun 26 10:12:20 PDT 2017
Your IP: 54.144.6.40

campin.net tent logo

Campin dot Net

Syslog is essentially three things:

A method of general system logging on UNIX

Early UNIX systems had no centralized logging system, leaving each application to decide on its own log policies. Log-related decisions included where to place logfiles, how much information to store, how long to store it, and whether to warn any or all logged in users of any particular events/conditions. Of course most applications simply logged to a text file and didn't do any alerting or log file rotation, but that's unimportant. ;)

One of the early UNIX hackers at UC Berkeley took note of the situation, and added a system logging facility to BSD UNIX. It should be noted that if he had forseen the impact of this impromptu system, he certainly would have designed it differently (at a minimum he would surely have included a year field in the log file format).

The way this centralized logging worked was that a program used the system API (a UNIX system call) to log information. The application using the system call (syscall from here on) included information on what subsystem the information was coming from (i.e. ftp, mail, kernel, etc) and also information on the importance or severity of the information (i.e. informational, critical failure, debugging information, etc). Armed with this information, the UNIX system manager could configure a centralized logging mechanism to log according to a single set of guidelines (where to place logfiles, how much information to store, etc).

The original use of a syslog daemon (syslogd) on UNIX was to collect log sent via the syslog syscall(s) (there are usually two similar syslog syscalls, but that's not important here). A single configuration file, usually stored at /etc/syslog.conf was used to configure it. The information from the syscall(s) arrived via a UNIX filesystem socket, usually at /dev/log. There was no network transmission originally involved, simply some local collection and storage of log information.

A log format

The traditional use of a syslog daemon is to store events in log files, in a format like this:

Jan 1 12:12:12 10.245.3.99 foo[421]: this is a message from foo

It is made up of two main parts:

  1. The "header" contains the date/time information and the originating system's hostname (or IP)
  2. A two-part message. The two parts of the message are: the program or subsystem name and the message itself. The program/subsystem part (when present) is followed by a colon and often includes the UNIX process ID (or PID) in square brackets to differentiate between multiple instances of the same program. In the message above the program or subsystem name is "foo". Other values commonly seen are long-running UNIX maintenance programs such as cron, network server programs such as sshd, and UNIX kernel modules/subsystems such as network interface information and packet filtering logs.

Sometimes the message has no program/subsystem information - it might contain a message from the syslog program itself (though this is certainly not the only instance when the information might be missing):

Jan 18 12:34:56 hostname last message repeated fifteen times.

This can cause problems with identifying fields in a syslog message when transmitted over the network. The program/subsystem field (called the "TAG" field in RFC 3164) is not required.

Syslog transport format, e.g. when arriving via a UNIX socket or from the network (TCP or UDP) is (in theory) the same as the on-disk format, except with the information about subsystem and importance included. This is called the "pri" field, and is made up of an integer surrounded in angle brackets. The "pri" field comes at the beginning of the message. The subsystem information is called a facility, of which there are 15, and the importance information is called a severity, of which there are 8.

Here is an example syslog message sent over the network:

<13>Jan 1 12:12:12 10.245.3.999 foo: this is a message from foo

Note that when the day of the month is a single digit, then it is formatted as a space then the day as shown above.

Remember that a syslog daemon receives locally generated messages over a UNIX socket, which isn't much different than receiving messages over the network.

Problems

There are two major problems with sending syslog messages over the network to a remote system:

  1. Since messages are single line only, there's no easy way to correlate related messages that span multiple lines (java stack traces for instance).

    A log parsing application can be programmed to analyze it properly, but this really has nothing to do with syslog. It is unfortunate that events have to be limited to a single line and (normally) fewer than 1024 characters.

  2. Many syslog programs, when configured to relay messages on to another syslog program on another host, will leave out certain parts of the syslog message - complicating proper identification of certain fields.

    Imagine this common scenario:

    The syslogd program on a Solaris machine is configured to send all logs to a central syslog server. When Solaris syslogd sends logs over the network, it doesn't include the hostname field at all. This means that a "last message repeated" message looks more like this:

    <13>Jan 18 12:34:56 last message repeated fifteen times.

    How do you ("you" being syslog server software) know whether or not the originating hostname is "last"? Well if you only have Solaris servers sending syslog messages, then you know that you simply need to use the IP/hostname of the sending system in the hostname field. But what if you have other systems sending logs? When a message lacks a program/subsystem name followed by a colon it can be hard for a syslog server to know which field is which. This is a strong reason to use syslog-ng on clients and servers, as syslog-ng always relays a complete message across the network to a remote syslog server. Other modern syslog servers might do the same.

    The sysklogd program used as a syslog server for many Linux distributions also leaves out fields. It leaves out the time/date information and the hostname information (the entire "header").

    <13>last message repeated four times

    This can cause problems as well, but is much less error-prone than the way Solaris (and OSF1/Digital UNIX, maybe others I don't know about) leaves out just the hostname.

  3. Many syslog programs, when configured to relay messages on to another syslog program on another host, will format the message in a non-standard format.

    When a syslog message arrives that uses a non-standard format, it is normal (and expected) for the receiving syslog server to prepend the date/time and hostname information in the correct/standard format.

    If a Cisco switch sends a message like this:

    2005 Aug 23 03:04:05 UTC +00:00 %PAGP-5-PORTFROMSTP:Port 4/16 left bridge port 4/16

    ...it'll be written to disk like this:

    Aug 23 03:04:05 switch.company.com 2005 Aug 23 03:04:05 UTC +00:00 %PAGP-5-PORTFROMSTP:Port 4/16 left bridge port 4/16

    Cisco's log format is well documented (not that their devices always send messages conforming to the documentation), but it is not consistent with traditional syslog format. For this reason messages from Cisco devices are rewritten by syslog servers when received over the network.

There is one basic problem with the way most syslog daemons record events to log files:
  1. With traditional syslog daemons, once logs were committed to disk the PRI information was lost.

    The syslog syscall included with Solaris (starting with Solaris 8) includes this information in the syslog message itself (configurable via a file). This way the information is not lost, no matter what syslog daemon is used and no matter where it is sent across the network.

    Aug 25 01:00:31 hostname sendmail[12100]: [ID 801593 mail.warning] j7P10VO12100: forward /tmp/.forward.hostname+: World writable directory

    Some syslog server software packages feature configurable log formats and macro expansion based on characteristics of the message. This permits recording of the facility/severity information in log files as desired.

Now that you understand the format of syslog, and the variations that are seen on the wire, you can better troubleshoot syslog-related issues. Say for example, that you have a syslog server that can commit events to log files named according to certain fields in syslog messages. If you choose to log according to the hostnames reported in messages, you might end up with a host directory named "last" when you have no such host.

Using the information just learned, you would deduce that the syslog daemon incorrectly interpreted the word last from a "last message repeated X times" message as a hostname. You would also know that a message that appeared to have two headers on it is actually a case of a syslog server prepending a header of its own when it judged that the header of an incoming message didn't conform to the proper format. That is the reasoning behind such a verbose description of syslog log formats in this document.

The syslog-ng software package has a feature to deal with strings which are misinterpreted as hostnames, and avoid this issue (the config file directive is called "bad_hostnames").

A network log tranmission mechanism.

We've already discussed the format of syslog messages, but not the network transport protocols or the roles played by syslog client and server software in typical syslog architectures. Much of this information is also contained in the informational BSD syslog RFC, RFC 3164 (not to mention some of the rewriting behavior described in the format section). I've purposely reused the same terminology. Please see the RFC for a more complete discussion of possible architectures.

There are essentially three broad roles to play: device, relay and collector. Traditional UNIX syslog uses UDP as the transport protocol, destination port 514 on the receiving host. Modern syslog software can also use TCP for transport and often has configurable port parameters. Note that the use of TCP allows for transparent tunneling with SSL/TLS, even though most syslog software packages that support TCP don't directly support encryption.
  1. syslog device

    The system where an event is generated. This might be a general purpose OS such as Windows or UNIX, a printer, a network device, etc. This device sends a text string conforming to syslog format across the network to a listening syslog process on another host.

    Note that this can be as simple as using the popular netcat program from the UNIX or Windows command line (NT/XP/Win2k[3] usage would be slightly different at the CMD prompt, but similar):

    $ echo "<13>Jan 4 12:12:12 host foo[345]: a syslog message" >/tmp/foo
    $ nc -v -u loghost.example.dom 514 < /tmp/foo

    The above is a valid syslog device/client. Of course there is no long running process producing the message (as there would be with most other messages), no syslog server software package involved, but the remote syslog server/relay receiving the message has no way of knowing this. In fact some simple syslog mechanisms that by design log to the local system, rely on the local syslog daemon listening on a network socket for incoming messages. They send events to localhost port 514/UDP.

    Part of the reason for the success of syslog as a network event logging mechanism surely comes from its simplicity. This is similar to the success of other simple network communication mechanisms such as SNMP, HTTP or even TCP/IP itself.

  2. syslog relay

    A system running syslog server software which receives events via syslog over the network, then acts as a syslog client by relaying it onto another system is considered a relay. Not all relays fit such an inflexible definition, to quote RFC 3164:

    Relays may send all or some of the messages that they receive to a subsequent relay or collector. In the case where they do not forward all of their messages, they are acting as both a collector and a relay. ...

    Relays may also generate their own messages and send them on to subsequent relays or collectors. In that case it is acting as a device. These devices will also be designated as a relay in the following diagram.

  3. syslog server

    For this discussion a syslog server is the point in the chain where syslog transmission stops, i.e. no syslog client behavior is used on the host. At this point the logs might be stored to disk, stored to disk and transmitted to a security management application on another host (via TCP or UDP) or on the same host, directly piped to another application on the same host, injected into a database on a remote server via a database client, or perhaps even discarded. The point here is that the message is not relayed on via syslog.

  Home |  Syslog-ng FAQ