Home Page

Configure Syslogd2

Sample Files

Deployment

New Concepts

The Config File

Compile & Install

Misc Topics

Capabilities

BASE FEATURE SET
SYSLOG_EXTRAS

CAP_CACHE
CAP_COMMAND
CAP_FILEROTATE
CAP_FILTERSIN
CAP_FILTERSOUT
CAP_HOUSEKEEPING
CAP_JOURNALTHREADS
CAP_KERNELTHREADS
CAP_OUTPUTTHREADS
CAP_PIPESIN
CAP_REBUILD
CAP_RECONNECT
CAP_SINGLEPOOL
CAP_SINGLEPORT
CAP_SINGLETHREAD
CAP_SPOOLFILES
CAP_STATS
CAP_STREAMIN
CAP_STREAMOUT
CAP_TAILFILES
CAP_THROTTLE
CAP_TRACEFILTER
CAP_USERTHREADS
CAP_VARBUFFERS
CAP_WHATIF
CAP_WORKERTHREADS

Demonstrations

Reference

Glossary of Terms

External Links

Syslogd2 Project Site
DBD2 Home Page
DBD2 Project Site

Other References

RFC 3164 (The BSD Syslog Protocol)
RFC 3339 (Internet Time Format)
RFC 5424 (Syslog Version 1)

Focus on Network

Syslogd2 Input Options
Syslogd2 Output Options
Queueing and Data Loss

BASE FEATURE SET

[Top of page]

---

For purposes of this web-site a 'base' version of Syslogd2 is defined to be a version of Syslogd2 compiled without any 'additional' compiler-symbols whatsoever.

If you have done a 'default' compilation of Syslogd2, then the 'syslogd2b' compiled (binary) image is a 'base' variant.

This list of 'base' features is in no particular order and may be incomplete.
I will add to it as additional base-features are added or come to mind.

• 32 'extra' faciltiies (extra0 - extra31). Access to facilities 'Reserved0'' through 'Reserved3'.
• Delayed Resolution
• Network Awareness
• Programmable Interrupts
• Support for multiple threadpools. The administrator may individually specified the thread-count of each threadpool.
• The ability to move command-line parameters into the confiuration file. Each such line must start with a tilde ('~') and all quotes and escape characters must be removed.
• The use of Soft Comments and '--skip <n> (lines)' to control parsing of configuration files.
• Support for both IPv4 and IPv6.
• The ability to use the '--remote' ('-r') and '--ApplicationMode' macros as well as the tradtional command-line options '-i [p]' and '-f [orwarding]'.
• Support for user-defined IP and Linux sockets (In both input and output directions).
• The ability to disable the automatic creation of the default IP input socket. This allows you to either redefine it with different options or as multiple addresses or to avoid conflict with another (concurrently-running) syslog processor.
• The ability to disable the automatic creation of the default Linux input socket. This allows you to either redefine it with different options or to avoid conflict with another (concurrently-running) syslog processor.
• Automatic detection and parsing of time formats with automatic conversion to the time-format of your choice as selected by the output-line option: 'format'.
• The ability to administratively disable all user-logging (ignores any configured user-list output-destinations).
• The ability to send output to user-defined character devices (such as virtual terminals or different command-line consoles).
• The ability to disable output to /dev/console and /dev/tty to avoid conflict with any other syslog daemons.

Without additional faeture-sets, you still have access to the 'base' command-sets, variables and option-lists of Syslogd2. These lists are extensive, so I'll just point you to the locations in the Reference Page. The 'base' version of Syslogd2 (without optional features) includes:

• All command-line options in this table except where the comment specifically states that a CAP_* feature-set is required.
• All global configuration variables in this table except the ones with dependencies (dependencies are in all-caps, start with 'CAP_' and are identified in the first column).
• All global boolean configuration flags in this table except where the comment specifically states that a CAP_* feature-set is required.
• All input options listed in this table except the ones with dependencies (dependencies are in all-caps, start with 'CAP_' and are identified in the first column).
• All output options listed in light or dark blue in this table except FilterFile (requires CAP_FILTERSOUT) and Stream (requires CAP_STREAMOUT) options.

CAP_CACHE

[Top of page]

---

Benefits of the name-cache

Benefits of the name-cache:

• Whether enabled or not, if the name-cache is declared, it will be used as one of two 'authoritive references' (DNS is the other) to 'resolve' input sources and output destinations.
• If enabled, the name-cache will also be used to resolve incoming host-IP addresses to hostnames and as a 'normalization' authority to convert hostname aliases and nicknames (or secondary addresses) to canonical names for downstream processors.
• The name-cache can also act to 'combine' multiple names or addresses for a single host that /etc/hosts or DNS may designated as distinct hosts
• The cache file can be used to anonymously access DHCP-assigned
• If a complete cache-file is provided to Syslogd2, Syslogd2 does not need to contact the DNS server at all.

---

CAP_COMMAND

[Top of page]

---

Command-Tool Prerequsites

Prerequisites: The Command-Tool requires CAP_STREAMIN to create the streaming input Linux socket and CAP_HOUSEKEEPING to provide an available thread in the running instance of Syslogd2 to act as the command-server.

Running the command-tool ('tsucX') with no parameters will give usage. The default size of the buffer reserved for configuration display is 1500 characters by 1500 lines which should be enough to display most configurations. If your configuration display gets truncated, try increasing the line-count.

The ncurses library is required to be installed on any system on which the command-tool (tsucX) will be run, but is usually already installed since it would have been used by the text-menus used to install the Linux distribution. (The ncurses header files may need to be installed in order to compile the tsucX code however.)

Command-Tool Usage

The configuration for command-tool on the server-side is straightforward: Add the 'command option to any streaming-Linux input socket to designate it as a 'command-tool-input' socket.

From a command-line terminal, execute the correct tsucX binary that matches the suffix of the binary you are running:

When compiled, Syslogd2 creates several variants - each named 'syslogd2'+'X' where 'X' is a suffix string. This same suffix string identifies which instance of tsucX will correctly match the feature-set of the running instance of Syslogd2.

Recommended installation of Syslogd2 results in a symlink at /usr/sbin/syslogd2 which points to the currenly-executing version of Syslogd2.

The command-tool is menu-driven. To start it either run 'tsucX' witout parameters to get syntax or use 'tsucX full-path-to-command-tool-input-socket'.

---

CAP_FILEROTATE

[Top of page]

---

The CAP_FILEROTATE compiler declaration adds the 'RotateFiles' routine into the Syslogd2 binary. This routine runs at fixed intervals. Each time it executes, it 'walks' the list of destinations checking the size of each local output file against the configured MaxFileSize and ignoring all non-file-type destinations.

When if finds an output-file that exceeds the configured MaxFileSize value, it first locks the destination to temporarily block the primary processing threads from writing to it, then creates a new filename by incrementing an integer value (starting at '1' (one) to be added at the end of the file (creating 'logfile.log.1' ... 'logfile.log.<n>').

It then properly closes the file and moves the existing file to this new name before creating a new file for output and releasing its lock to allow the primary processing threads to write to the new file.

---

CAP_FILTERSIN and CAP_FILTERSOUT

[Top of page]

---

• Any filter file can be used on multiple inputs and outputs, but each input or output specification may only specify one filter-file.
• Any filters can be declared without creating the actual filter-file. The file can then be created later and a signal sent to Syslogd2 to activate the filter.
• All filters use string comparisons to select events for action. Filters may be set to either discard or keep messages that do not meet their selection criteria.
• All filters can be used to modify event content when search criteria are met. Modifications can be to facility, priority, hostname or string content.

Pros and cons of declaring CAP_FILTERSIN and / or CAP_FILTERSOUT

Pros:

• Input filters can be applied to any type of input source and apply to all traffic entering Syslogd2 via that input connection.


• Output filters can be applied to any type of output source and apply to all traffic exiting Syslogd2 via that output connection.

Cons:

• The 'cost' (in cpu-cycles) of applying a filter is very small (but not zero so even though filters are a relatively efficient operation, they will consume a variable number of additional cpu cycles per message processed. The overall impact of this extra processing can usually be offset in very high-traffic scenarios by adding a few more threads additional.

---

CAP_HOUSEKEEPING

[Top of page]

---

Housekeeping threads offload the processing load of background maintenance and clean-up activities in Syslogd2 from the parent thread. All but the smallest configurations should enable some number of HouseKeeping threads.

Housekeeping threads (when enabled) will execute the following functions on behalf of the parent thread (whenever the task is enabled).

• CheckSources: This routine:
  • Checks for a network-state-change since last execution.
  • Re-evaluates all inputs when a network state change occurs.
  • Attempts to resolve any previously unresolved inputs. (Delayed Resolution feature.)
  • Re-opens all input connections that are marked as 'invalid' for failing to open on the first attempt.
  • Checks for and re-creates Linix socket and fifo (name-pipe) filesystem entries if they get deleted.
• CheckDestinations:
  • Checks for a network-state-change since last execution.
  • Re-evaluates all outputs when a network state change occurs.
  • Attempts to resolve any previously unresolved outputs. (Delayed Resolution feature.)
  • Re-opens all output connections that are marked as 'invalid' for failing to open on the first attempt.
  • Checks for spool-files attached to any newly-reopened outbound connection and (if one exists), initiates a follow-up process to unspool all data over open connections.
• CheckFilters: [Requires either CAP_FILTERSIN or CAP_FILTERSOUT]: Checks for new filters. If it finds a new or updated filter, it parses, compiles and activates it.
• RotateFiles: [Requires CAP_FILEROTATE] Rotates output files when they reach a specified size.
• Mark: Writes a "-- Mark --" message to local files with a timestamp. (This is a traditional syslog processor action to indicate a low-activity file is still active.)
• Command: [Requires CAP_COMMAND] Acts as the host-server for an interactive command-session. Because an interactive session can only run at the speed of humans, Syslogd2 will not initiate a command-session unless a housekeeping thread is available to host it. The parent thread simply cannot run an interactive session and hope to keep up with all other tasks it would have to run in the absense of housekeeping threads.
• ResetCache: Resets the name-cache to initial values. Executed only on-demand.

---

CAP_JOURNALTHREADS

[Top of page]

---

Journal threads use the Linux systemd-journal API to read selected data from the system's journal. Once enabled, two (2) commands are used to configure this feature:

• --Journal: This is the 'top-level' configuration for the one-off journal-input threadpool. Its settings apply to all instances of all fields read by the journalthreads subsystem.
• This command has no 'primary parameter' -- just an optional comma-separated list of options.
• It accepts most input arguments (exceptions include 'polltime', 'port', 'stream', 'poll', ...).
• It accepts specific options: 'Source' (Src), 'TagName', 'TagPid', 'TimeStamp', 'IncludeAllFields', 'MaxLength', 'NoSDLength' and 'Append'.
  Source (src): A semi-colon-separated list of keywords. These are the values of the '_TRANSPORT' parameter from 'man systemd.journal-fields' that specify how the entry was received by systemd:
  • audit: from the kernel's audit subsystem.
  • driver: from internally driven sources.
  • syslog: via the local syslog socket with the syslog protocol
  • journal: recived via the native journal protocol
  • stdout: read from a system services stdout or stderr output streams
  • kernel: read from the kernel
  The default is to display no additional fields other than those that would comprise a standard syslog message (where fieldnames beginning with 'SYSLOG_' are used if available).
  • NoSDLength
  • IncludeAllFields: A boolean value that reverses the default action of 'source' and includes all fields unles they are explicitly disabled with the --journalfields command (below).
  • DefaultMaxLength (maxLength, length): The maximum number of 'value-characters' to display. Default: 80
  • Append: Start at the current last record of the journal (default is to start at the beginning of the journal).
  • A 'tag' field is create from journal information unless the 'notagfield' is specified:
    • TagName: A single-character binary value: [s]yslog or [j]ournal. 'S' will create a tag field using the syslog value 'SYSLOG_IDENTIFIER'. 'j' will use the short form of the command '_COMM'. The default is 's'.
    • TagPid: Same as for TagName using either the 'SYSLOG_PID' or '_PID' field.
  • TimeStamp: same as for TagName above using 'SYSLOG_TIMESTAMP' vs '_SOURCE_REALTIME_TIMESTAMP'.



• --JournalField: This is the field-specific confiuration command. It customizes any deviations for the selection or display of each field from the parameters set by '--journal'.
  • This command DOES take a 'primary parameter': That parameter is the field-name known by the systemd-journal. For the purpoase of easier configuration in Syslogd2, this field-name is case-non-sensitive and the inclusion of the leading underscore is optional.
  • Multiple instances of this command are allowed. The sequence of parsing is the sequence commands will be displayed in the output list.
  • The general format for this command is --journalfield JournalFieldName [[ = ] new-name] [, option], where option is zero or more of:
    • disable: A boolean value that causes this field to NOT be displayed.
    • maxlength (length): The maximum number of characters in the value component to keep. All other characters will be truncated.



• As an aid to users, Syslogd2 adds an extra field at the end of the journal field-list: SDStringLength=<length>. This value is the actual length of the entire 'sd-string' (including the 'SDStringLength' field itself). This information is often useful when estimating buffer lengths. This field can be suppressed with the --journal option 'NoSDLength'.

---

CAP_KERNELTHREADS

[Top of page]

---

Syslogd2's implementation of kernel input contains two 'off switches':

• KernelThreads: This global boolean value (if disabled), will disable the use of kernelthreads. It will be as if the kernel-threads had not been declared.
• KernelLogging: This global boolean value (if disabled), will disable all kernel logging by any mehtod.

The CAP_KERNELTHREADS compiler symbol causes the compiler to include the code to create and use the 'kernel threadpool'. This threadpool is dedicated to reading kernel input using the '--kernel command-line option.

~ --kernel = readers = 3, filter=kernelfilter, ...

There is never more than one instance of the --kernel keyword required (since there is never more than one instance of the kernel that generates the traffic). Therefore, multiple instances of this keyword are not allowed. Additionally, the 'id' value is neither required nor supported for this threadpool-type.

The boolean value 'kernelthreads' can disable the use of this threadpool when the threadpool is declared at compile-time. This 'off-switch' is intended to facilitate depoloyment for distribution managers so they can freely compile this feature into Syslogd2 and the user can then disable kernel input if desired. When enabled, the kerel-threadpool has two (2) modes of operation:

• When the 'Poll' variable is disabled (default), this routine will directly access the kernel's memory buffer bypassing any need to mount the /proc filesystem for access to syslog messages.
• When the 'Poll' variable is enabled, Syslogd2 will poll the /proc/kmsg pseudo-file for input.

The klogfile parameter is automatically initalized from system header files so should work for virtually all distributions and will likely never need to be modified. Should the proc pseudo-filesystem ever be mounted a a location other than '/proc', this variable can be modified to accomodate the change.

Syslogd2 provides two (2) ways to read kernel data: Kernel-Threads and Tail-Files. Kernel-threads (using the '--kernel keyword) are preferred because the algorithm used by this threadpool is much more efficient. If both are declared and enabled, Syslogd2 starup code will force the use of KernelThreads.

If you choose to disable the kernel-threadpool in favor of tail-files, don't forget to set the 'kernel' opion in the input-specification option-list. Failure to do so will result in kernel messages that have no embedded facpri value being logged as 'user.notice' instead of 'kern.notice'.

---

CAP_OUTPUTTHREADS

[Top of page]

---

Pros and cons of declaring CAP_OUTPUTTHREADS

Pros:

• Frees up worker-threads (or reader-threads if CAP_WORKERTHREAD is not declared) by off-loading the output filtering and output processing chores.
• Further reduces load on processing threads by consolidating output selector-strings so one-test-per-output-threadpool instead of one-test-per-destination-location needs to be made to determine disposition of processed entries. Queueing to memory is much faster than writing to disk or to a network socket / terminal device.
• A 'must-have' if processing large numbers of output-files since output locations can bve distributed among the output threadpools.
• Allows effective use of many more threads (meaning much higher degrees of perallel processing). Allows CPU resources (threads) to be allocated to the portions of SYslogd2 based on where the 'heavy-lifting' will occur (input, processing or output).
• For dedicated syslog servers (such as firewall collectors or central-host-syslog-concentrators), the use of both CAP_WORKERTHREADS and CAP_OUTPUTTHREADS is highly recommended to maximize throughput and prevent data-loss due to congestion.

Cons:

• Causes Syslogd2 to use more system resources than when CAP_OUTPUTTHREADS is not declared on low-volume systems. Each buffer 'message-slot' is approximately 1,500 - 2,000 bytes. Each idle thread will consume a small (but not zero) number of cycles while passive as the OS still has to track them.
• Probably overkill for most syslog host requirements unless a lot of output-files are declared, a LOT of filtering is going on or particularly fragile output connections are involved where there may be a desire to 'isolate' bad actors. The system resources consumed (especially memory) may be better used for other applications unless heavy (server-level) processing loads are anticipated.

---

CAP_PIPESIN

[Top of page]

---

I don't know how useful the CAP_PIPESIN optional ability will turn out to be. I fear that the industry has not used named-pipes in so long (if ever), that most syslog daemons on the market today no longer take this feature seriously and think through what the output format should be. Since pipes always 'talk' to applications, one woult think that the output format would be a complete syslog-formatted string, but testing with rsyslog tells a different story as traffic from an rsyslog named-pipe contains no facility/priority information. I cannot speak to other syslog daemons on other systems due to lack of information and access.

I therefore offer the named-pipe input implementation and a testing tool with the hope that at least some people find it useful.

NOTE: I was surprised to discover just how fast a named-pipe connection can be when the listening end is connected to a multi-threaded application like Syslogd2.

---

CAP_REBUILD

[Top of page]

---

The CAP_REBUILD compiler symbol enables the code that implements the 'delayed resoluion' feature of Syslogd2. It does this by causing the compiler to include (and compile into the binary image) the code necessary to periodically execute code (based on the network state and the number of remaining unresolved sources and destinations) to rebuild all sources or destinations at the start of the CheckSources and CheckDestnation functions.

Without this compiler-flag, CheckSources and CheckDestinations will still run as expected and discovered error-conditions (broken connections, deleted output files, etc) will be corrected, but no new sources or destinations will be dynamically resolved or added to the existing configuration. No network-checks will be enabled and no adjustments for any 'change-of-netowrk' situations will be made.

When enabled the CAP_REBUILD does not support any new settings as both CheckSources and CheckDestinations are assumed to both be active. Since the CAP_REBUILD code is implemented as part of the CheckSOurces and CheckDestinations routines, if these routines are disabled, Syslogd2's ability to re-resolve hostnames or to detect and adjust to changing network states will also be disabled.

---

CAP_RECONNECT

[Top of page]

---

About CAP_RECONNECT

CAP_RECONNECT is a prerequsite for CAP_REBUILD as it sets the schedules for and forms a 'wrapper' around the code of CAP_REBUILD.

CAP_RECONNECT modifies the actions taken by Syslogd2 when network connection, pipe, or Linux socket connection fails. Instead of trying to re-open the connection for writing each time a message is received for that connection, a flag is set to indicate the connection is (temporarily) disabled. Based on the intervals specified by the SourceCheckIntervals or DestinationCheckIntervals schedules, Syslogd2 will periodically walk the list of input or output specifications and attempt to reopen any failed connections. It will also re-create any output files, Linux sockets or pipe devices that were deleted and will attempt to reopen any resolved connections that failed to open on previous attempts (either input or output connections).

Both SourceCheckIntervals (sci) and DestinationCheckIntervals (dci) are global schedules defined by the --defaults command-line parameter. They determine the intervals at which Syslogd2 will walk the list of inputs (SourceCheckIntervals) or destinations (including stderr/stdout files) (DestinationCheckIntervals) to re-open any failed connections and to re-create any missing filesystem entries (file entries, pipe devices, Linux sockets, etc).

---

CAP_SPOOLFILES

[Top of page]

---

About Spooling

Spooling in Syslogd2 is still under development. Both the 'classical' connection spooling and the 'new' network-state-spooling. (It's mostly the de-spooling code that needs the most work...

Only recently have the major components of Syslogd2 (that are required to support spooling) finally stabalized. Once my design and implementation issues with spooling are finally resolved, I'll update this page again.

---

CAP_STATS

[Top of page]

---

As currently implemented, CAP_STATS is limited to monitoring the various FIFO queues in Syslogd2 and reporting their 'high-water-marks' via the command-tool. I am wide open to thoughts about what other statistics may be gathered and reported by Syslogd2 that might be useful to network- or sysmte-administrators.

CAP_STATS (like the Command-Tool) is a very embryonic feature with lots of potential. The single queue-statistic of 'high-water-mark' has already proven will prove invaluable to me in getting a 'feel' for the performance characteristics of Syslogd2. It is certain to help in tuning Syslogd2 for various processing loads.

To date the high-water-mark for FIFO queues is the only statistic implemented, though I'm actively looking for more ways to report the health and performance of Sylsogd2.

CAP_STATS is a feature I would like to see grow with Syslogd2's future capabilities, but as of yet I don't have a feel for what kinds of statistics would be truly useful to network administrators -- there's just too many (as yet untested new features in Syslogd2 to have any feel for what works well now and what statistics would help in the future.

---

CAP_STREAMIN and CAP_STREAMOUT

[Top of page]

---

CAP_STREAMIN and CAP_STREAMOUT do not have 'off' switches. The implementation is 'in addition' to the default datagram code, so Syslogd2 will not perform any differently as a result of these declarations until a streaming connection is actually configured.

TCP support is a separate (and optional) feature for several reasons:

• Not every host of Syslogd2 needs to make use of TCP or streaming Linux sockets to transmit log files. Most systems are not under any form of network-management and do not forward syslog events to remote systems. Consequently, the majority of systems should leave IP services (and TCP support) disabled. The demand for streaming Linux sockets (as of March 2023) is non-existent as no syslog processor other than Syslogd2 (which is just being released) supports this option yet.
• Streaming input and output connections require more 'link maintenance' than datagram protocols which translates to higher processing overhead. There are more ways for TCP communications to cause systemic problems when the receiver is unavailable or overwhelmed with connections or when the network link fails. The need to detect and restart the handshake protocol when streaming connections are lost and the 'blurring' of syslog-event message boundaries in an input stream vs one-message-per-packet datagram operation adds to the complexity of programming for TCP support.
• Though streaming traffic can be more reliable (for example, the handshake protocol of a streaming connection is critical to knowing a connection has gone down and Syslogd2 should start spooling traffic), not every log-file data-transfer needs the guaranteed delivery of a streaming connection.
• Host operating systems (and applications) have limits as to how many streaming connections they may maintain at one time. Though these limits are high enough to generally not be an issue, larger networks can easily exceed them through the sheer number of syslog generators that may be reporting to a single (centralized) host.
• The vast majority of Linux deployements do not need the overhead or increased memory footprint of the extra TCP code or data-structures. Designating these as optional features allows the production of (slightly) smaller binaries.

On the other hand there are some situations where TCP support (or streaming Linux sockets) provide benefits that are well worth the costs.

• When traversing firewalls the use of TCP causes the firewall to rule-check only the handshake negotiation (one check per connection) vs rule-checking every packet in a UDP data stream.
• Stream connections allow for 'authentication' of a connection that can last for all data transmitted over the same stream (because of the handshake).
• The handshake of a streaming connection allows detection of a broken link which is a basic requirement for any implementation of spooling.
• Stream connections are essential when sequential or 'guaranteed' delivery of data is important as they insure that data arrives in the same order it was sent.
• Stream connections (and the spooling they enable) are important in order to capture syslog events logged at the remote end of a broken link for post-outage analysis.

Since cleraly not all hosts fit into one or the other 'camp' described above, Syslogd2 implements streaming input and output not only as options, but as seperate (uni-directional) options.

---

CAP_TAILFILES

[Top of page]

---

Pros and cons of declaring CAP_TAILFILES

Pros:

• Provides an integrated and flexible way to read external log files into a syslog-based management-and-reporting system. Allows all Syslogd2's features (spooling, filters, input control, etc) to be used in managing non-syslog log files. Syslogd2 can read most text-based log files, filling in missing syslog-fields as needed.
• Allows monitoring of application log files in those instances where the application does not support the syslog protocol.
• Can be declared so it is present in the binary image when needed. Will do nothing unless an input text-file is actually configured.

Cons:

• Not every system needs to read text-files since most logging is usually done through the syslog daemon which is already being processed and filtered if Syslogd2 was installed as the host's syslog service.
• For those systems that don't need it, not declaring it makes a smaller binary image that contains neither the code nor data structures used to read and process the text files.

---

CAP_THROTTLE

[Top of page]

---

About CAP_THROTTLE

CAP_THROTTLE retquires either CAP_FILTERSIN or CAP_FILTERSOUT as a prerequisite. It activates the 't' option in filters to 'throttle' high-volume traffic -- converting it from frequent,r multiple repititions of same or similar messages to single messages issued periodically with message-counts added to the 'sdstring' field for the messages that have been suppressed during the user-specified time-period.

Throttling works in two modes -- a 'global' mode and a 'per-host' mode. In both modes, a copy of the specified 'throttle-string and a host name (a NULL ptr for 'global' mode) are stored globally for comparison purposes, thereby slightly increasing the memory footprint.

Throttling should usually be implemented in input-filters (the ideal choke-point for throttling). Implementation in output-filters results is also supported but may be sub-optimal. While the input-filter will throttle an incoming message one time, use of output-filters may result in multiple throttle instances throttling that same message multiple times.

The throtte parameter should always be positioned at the start of the filter-line's key-string for clarity and readability.
The 'throttle' filter-key requires one (match) field to hold its parameters and is not counted as either a 'match' field OR a 'transform' field.

During the parsing of each filter, the 't[hrottle]' data (both key of 't' and data-field) are extracted and stored in a separate sub-structure. Isolating this key from the rest of the filter-line by putting it at the front of the input-line simplifies readability for rest of the filter input-line.

Because the 'throttle' filter-option is NOT considered a 'match-field', it requires additional 'match' criteria to select applicable syslog events. This allows 'throttling' of strings that may not be completely identical or throttling of a selected set of almost-identical strings by setting match criteria that do not exactly match the throttling criteria. Failing to provide additional 'match' criteria on a filter-line that contains a 'throttle' option results in an error and the line being ignored.

Reminder: Use the 'testFilter' tool in the './tools' subdirectory to test and validate your filter-files.

The 't'[hrottle] option currently supports two options:

• 1: 'h'[ost-mode] is a boolean value (present or omitted) that defaults to 'off' when omitted. When active, 'host-mode' causes Syslogd2 to throttle events on a per-source-host basis where both the throttle-string and the host-string are considered as 'throttle-identifiers'.
• 2: 'i'[nterval] takes a parameter (a time-string). The goal of throttling is that during this interval, only one message will be passed by the filter with a repeat-count inserted into the 'sdstring' parameter-list.

Format: <comma-separated-options> + ':' + <throttle-string>

tc /i2h,h:This is a throttle-string for a host-throttle/match-string to identify applicable syslog events/
tc /i1d:This is a 'global' throttle-string/match-string to identify applicable syslog events/

---

CAP_TRACEFILTER

[Top of page]

---

About CAP_TRACEFILTER

The filepath may be absolute or relative. If relative, it is relative to ConfigDir. Multiple input sources may specify the same tracefilter file. Only one tracefilter instance is supported per input specification.

The format of a tracefilter file is a highly-simplified variant of a filter-file. There are only two line formats supported by a tracefilter file (in addition to comments that start with '#' and blank lines).

The first format is 's' + whitespace + <delimited string>.
The second format is 'e' + whitespace + <delimited string>.
The delimiter is auto-recognized (on a per-line basis) as the 1st non-whitespace character following the 's' or 'e' field-type-identifier. For example:

s /------------[ cut here ]------------/
e /---[ end trace 0000000000000000 ]---/

When testing a new tracefilter, the testFilter tool in the ./tools directory may be used to verify both the syntax and the operation of your tracefilter-file prior to committing it to a live configuration.

---

CAP_USERTHREADS

[Top of page]

---

About CAP_USERTHREADS

Syslogd2 supports no more than a single user-threadpool at any one time. This threadpool acts as a pseudo-destination, ready to process any and all traffic going to user terminals.

The user-threadpool may only be configured via the --defaults and --enable/--disable keywords (for global settings) or via --threadmaps for individualized threadpool settings.

The 'lines=' and 'workers=' parameters on user-list and Write-ALL (WALL) destination output-lines apply to the output-threadpool the line is associated with (the 'id' option) -- not the user-threadpool.

User-list and WALL destination types are never assigned to the user-threadpool itself. The user-threadpool exists for the sole purpose of offloading the (time-and-process-intensive) task of looking up and writing to lists of user terminals so the 'regular' output-threads can concentrate on processing non-user traffic.

The user-threadpool functions either as a subroutine that can be called by the regular output threads (in which case the output threads spend the cpu cycles to process user traffic) or as a stand-by threadpool 'destination' with its own FIFO queue that output threads can queue messages into before moving on to process the next destination in their list.

---

CAP_VARBUFFERS

[Top of page]

---

About CAP_VARBUFFERS

CAP_VARBUFFERS requires no activation or configuration. This feature provides the ABILITY (not a requirement) for users to extend buffer sizes to accomodate long syslogg strings or to shorten buffer lengths if concerned about memory usage. Minimal additional memory is required for support of this feature (only the memory required for parsing of the parameters. If not modified, these buffers will be used anyway (just with default values).

If at any time, the length of a data string exceeds the value specified for a 'max' buffer size, the data will be truncated to fit the buffer. For puropases of data truncation the SD-StringLength and MsgLength maximum values are treated as if they applied to separate and independent buffers.

---

CAP_WHATIF

[Top of page]

---

Associated Links:
New Concepts: Configuration Display
New Concepts: Command-Tool

About CAP_WHATIF

'CAP_WHATIF' gets its name from the question: What would the configuration look like if...

'TestConfig' (or '-T') gets its name from the primary usage of this feature: to 'test-run' a (potentially-modified or new) configuration file to see what the resulting configuration would look like: What structures would be built, what settings and resources would be used, etc.

The --TestConfig command-line option is not eligible to be moved to the configuration file because it exits immediately upon finishing it's configuration display. When the '--TestConfig' option is used, Syslogd2 makes the following (temporary) changes to the (in-memory) configuration:

• The --stderr (error) file (if declared) has the suffix '.TestConfig' appended to the filename.
• The --stdout (log) file (if declared) has the suffix '.TestConfig' appended to the filename.
• The pid-file has the suffix '.TestConfig' appended to the filename.

These filename modifications make it safe to use the --TestConfig while another copy of Syslogd2 is running on the same host.

The --TestConfig command-line option accepts up to 4 parameters:

1. A keyword to indicate after which phase of parsing the configuration display should occur:
   • Parsed (AsParsed): This aborts the startup process immediately after all configuration-files have been parsed. Primarily useful for debuggging.
   • Built (AsBuilt): This aborts the startup process immediately after all primary structures have been built. Primarily useful for debuggging.
   • Staged: This aborts the startup process immediately after all primary structures have been staged into production structures but just before system resources (such as mutext, semaphores, buffers and connections are initialized. This is the default.
   • Note: There is also an ability to display the confiuration of an executing Syslogd2 binary, but that ability requires the Command-Tool feature.
2. A keyword to indicate the desired output location:
   • <no keyword>: The users terminal screen. This is the default.
   • stderr (err, E): This sends the output to '<stderr-file>.TestConfig'.
   • stdout (out, O, LogFile, Log): This sends the output to '<stdout-file>.TestConfig'.
3. A keyword to indicate the network-state to emulate:
   • Any: This (option) assumes that all network-states are valid at once. It is the default. By not choosing a specific state it most closely emulates 'conventional' (network-state-less) thinking.
   • Down: The network is assumed to be down (as in not running at all, network drivers not loaded, etc). No IP ports or configuration will be allowed.
   • Local: The network is assumed to be in a 'loopback-only' operational state. This could simulate a laptop in airplane mode or a networked-device during a network outage. Only loop-back connections will be considered 'valid' and no hostname resolution will occur because the DNS server is assumed to be 'unreachable'.
   • Other: The host's network-service is assumed to be fully up and functional on all interfaces (and for future purposes not attached to an identifiable network). However, all only IP connections that have 'other' specified in their 'network' option-list or that have no network-optiona specificied or that include the 'Any' keyword will be considered 'valid' for configuration display purposes. (This is a part of the on-going developmental exploration of network awareness.)
4. An integer (or hex) value indicating which sections of the configuraiton to display. The default is '0' [zero] means 'all sections'. The overall display is divided into sections. At the top of each section is a line containing content similar to: '[Display value 0x4000, (16384 decimal)]'. These values (hex or decimal) from multiple sections are added together to get the ovarll display value specified by this parameter.

For the Testconfig command-line option, it does not matter what order the values are found in or how many values are provided, since defaults will be used where values are not provided.

./syslogd2 --systemd -T 0x7000,s

---

CAP_WORKERTHREADS

[Top of page]

---

Pros and cons of declaring CAP_WORKERTHREADS

Pros:

• Allows reader-threadpools to be MUCH more responsive to high input volumes of syslog traffic. (orders of magnitude higher). This means a potential for much more data to be processed per unit time. (assuming the input is available to process.)
• Allows effective use of many more threads (meaning much higher degrees of perallel processing). Allows CPU resources (threads) to be allocated to the portions of SYslogd2 based on where the 'heavy-lifting' will occur (input, processing or output).
• For dedicated syslog servers (such as firewall collectors or central-host-syslog-concentrators), the use of both CAP_WORKERTHREADS and CAP_OUTPUTTHREADS is highly recommended to maximize throughput and prevent data-loss due to congestion.

Cons:

• Causes Syslogd2 to use more system resources than when CAP_WORKERTHREADS is not declared on low-volume systems. Each buffer 'message-slot' is approximately 1,500 bytes. Each idle thread will consume a small (but not zero) number of cycles while passive as the OS still has to track them.
• Probably overkill for most syslog host requirements. The system resources consumed (especially memory) may be better used for other applications unless heavy (server-level) processing loads are anticipated.

---

SYSLOG_EXTRAS

[Top of page]

---

About SYSLOG_EXTRAS

Syslogd2 provides a number of additional facilities not normally accessible by syslog processors or defined by the syslog specificaitons. these facilities are in two sets: 'Reserved0' - 'Reserved3' and 'extra0' - 'extra<n>'.

The access to facilities 'Reserved0'' through 'Reserved3' is part of the basic Syslogd2 package and is not configurable.

The number of 'extra' facilities defaults to 32 (as of this writing) but can be configured at compile-time between 0 and 1000. The range of 'extra' facilities is also part of the basic Syslogd2 package.

These additional facilities are intended primarily for use by Syslogd2 filters.

---

[Top of page]