About the Syslogd2 JournalThreads Input module
[Top of page]
The Syslogd2 implementation of systemd journal input is a dedicated, single-instance optional input-type threadpool.
Before it can be used (or configured), it must be declared using the CAP_JOURNALTHREADS option at compile-time.
At run-time, it must be enabled usiung --enable journalthreads.
There is never more than one JournalThreads threadpool for any given Syslogd2 binary. This is because there is never more than a single
instance of a systemd journal on any given host and therefore no need for more than one threadpool to monitor this journal.
Configuration of the JournalThreads threadpool consists of a single --journal command followed by zero or
more --journalfield commands.
The JournalThreads implementation in Syslogd2 achieves multiple objectives.
- It provides basic syslog-formatted data allowing selection of either syslog-originated fields or journal-originated
fields in many cases (syslog-fields are preferred if avaialable).
- It illustrages (and demonstrates) the use of the Syslogd2-specific sd-string field for parameter-passing.
The syslog format produced by this module can be fed directly into the DBD2 project where each field can be parsed into named database fields.
As a side note, the implementation of this feature stressed the overall Syslogd2 system sufficiently that it highlighted some additional (structural)
changes that needed to be made due to unexpected issues surrounding buffer lengths (CAP_VARBUFFERS) and
open-file limits
- It provides access to additional "syslog-formatted" input from Linux platforms for network-management purposes.
Each input line (and each additional data-field) from the systemd journal is fully supported by Syslogd2 filters for both routing,
data-reduction and data modification. Thus the non-syslog inputs from the journal can serve as additional sources of network-management alerts being
monitored by Syslogd2.
CAUTION: The systemd journal API used by Syslogd2 opens a large number of files per instance.
I don't know the exact number, but suspect it varies from host-to-host and numbers in the hundreds if not muliple hundreds of files.
This suspicion was borne out when I opened 8 concurrent thread instances of this API and promptly exceeded the process-limit of 1024 files-per-process.
If you use the CAP_JOURNALTHREADS interface, I STRONGLY recommend that you increase this process-limit. The performance and stability of Syslogd2
will otherwise be compromised due to its inability to open additional files (or sockets, pipes, or user-terminals) for either input or output.
I HAVE successfully used the joural interface with low levels of concurrancy (4 instances or less) of CAP_JOURNALTHREADS without exceeding this limit, but cannot
recommend doing so as a routine matter.
root# ulimit -a
root# ulimit -n 2048
root# ulimit -a
The --Journal command
[Top of page]
The '--journal' command is used to configure systemd-journal input to Syslogd2. The journal input-mechanism uses
various journal fields to 'create' a syslog-formatted string that is then submitted to Syslogd2 processing steps as if it had been read
from an external source. The process of creating this string can be affected by various --journal-specific options enumerated below.
If CAP_JOURNALTHREADS is defined and enabled, Syslogd2 will automatically create a default 'journalthreads threadpool as if a '--journal'
command had been entered using an option string of "append".
This will result in 'standard' syslog events being produced from all active journal sources (and likely a large amount of overlap if
combined with the events received from the systemd-forwarding-socket at /run/systemd/journal/syslog).
The overlap would be caused by both Syslogd2 input methods forwarding the same information from the same originating source (the journal).
Syslogd2's default --journal input-specification can be over-ridden by specifying a --journal command anywhere within the configuration file in
accordance with (IAW) Syslogd2's first-come-first-serve parsing policy.
There is no primary parameter for the --journal command since an Application Programming Interface (API) is used to read data.
There is only a list of options. As is the case throughout Syslogd2, all keywords are non-case-sensitive.
'--journal' is considered to be an input-command. As such, it accepts generic Syslogd2 input options in addition to the specific '--journal'
options enumerated below. These generic input options include (but are not limited to) threadpool selection and configuraiton options, 'facility', 'priority', 'ignore',
'filter', 'network', and 'hostname'.
- AllFields (IncludeAllFields): [disabled] A boolean value that specifies the default 'inclusion state' for all journal fields.
When enabled, all journal fields are added into the sd-string of each syslog message by default.
When disabled, no fields are added by default.
- Append: Causes Syslogd2 to start reading at the end of the journal instead of at the beginning. Since the journal makes
extensive use of disk storage, the journal may contain data from a month or more ago if started from the beginning.
- DefaultMaxLength (MaxLength, Length): [80 char]
This non-negative integer specifies the maximum length of any journal-field that is added to the sd-string. It applies to the sd-string only.
Any data field that exceeds this length is truncated.
- NoSDLength: [disabled] A boolean value that enables/disables the inclusion of an additional field at the end of each SD-String: [SDStringLength=nnn].
This additional string is provided by Syslogd2 as an estimate (with a couple of bytes) of the total length of that SD-String (including
the SDStringLength field itself). Enabling this boolean value disables the SDStringLength field.
- NoTagField: [disabled] A boolean value that controls whether a 'tag' field is created from journal fields and prepended to the msg string.
The 'tag' field (if present) consists of an application-identifier ('SYSLOG_IDENTIFIER' if present or '_COMM' if not) and a pid-field ('SYSLOG_PID' if
present or '_PID' if not). Enabling this boolean value disables the creation of the 'tag' field. Example 'tag' field: "kernel [82534]: "
- TagName (AppName): This boolean value (using 's[yslog]' to disable or 'j[ournal]' to enable and defaulting
to 's' [disabled]) specifies whether to use the SYSLOG_IDENTIFIER field or the _COMM field as the tag-field's application-identifier.
The SYSLOG_IDENTIFIER field is taken (by systemd) from the incoming syslog message and is preferred by Syslogd2. The _COMM field is the basename
of the command that the event came from. This may produce slight differences. For example, an event logged by 'CRON' might be issued from a command called 'sh'.
- TagPid: This booelan value works like TagName but uses the fields SYSLOG_PID and _PID. Again, this may differ since the
SYSLOG_PID would represent the pid that generated the syslog message and the _PID could be a child process that produced the actual message.
Again, the Syslogd2 preferred value is SYSLOG_PID if present.
- The systemd journal uses an 8-level priority system that is compatible with that of syslog. All entries are assigned a priority whether or not the originating
source is 'syslog'. When creating a syslog string from journal datafields, the value SYSLOG_FACILITY will be used as the facility component if present.
If this field is not present in the data, Syslogd2 will use the default facility assigned on the --journal command-line (if any) or the default
facility component from UserFacility or KernelFacility as appropriate.
- TimeStamp: This is a boolean value that works the same way as 'TagName' and 'TagPid'.
It affects the value of the timestamp that Sysloggd2 uses for the 'time' field of the syslog msg.
The systemd journal records the time of each event in the field '_SOURCE_REALTIME_TIMESTAMP' (in micro-seconds).
It also extracts the timestamp (if it can) from incoming syslog data events and stores it in the field 'SYSLOG_TIMESTAMP'.
Sysloggd2 uses the SYSLOG_TIMESTAMP field unless the 'TimeStamp' option has been enabled or set to 'j'. The two values should normally differ by very little
if coming from the same application, but may be significantly different if the syslog data is being forwarded from messages originating in a different timezone
or different host with a 'drifted' system-clock.
- Transport (Source, Src): The parameter to this keyword is a semi-colon-separated list of keywords. Each keyword is the name of a
journal-transport mechanism. Each keyword may be preceded by a tilde ('~') or an exclamation point ('!') to indicate 'not'.
Aadditionally, the keywords 'All' or 'None' may be used. The default is 'all'.
The Transport option specifies which sources to accept data from. The individual sources (keywords) are:
- audit: from the kernel's audit subsystem.
- driver: from internally driven sources.
- syslog: via the local syslog socket with the syslog protocol
- journal: recived via the native journal protocol
- stdout: read from a system services stdout or stderr output streams
- kernel: read from the kernel
Example usage (Note: Only one '--journal' line is accepted per instance of Syslogd2 2nd and subsequent entries are errors and will be ignored.
The entries below are intended to show different possibilities for using the command and may NOT be used at the same time.
~ --journal src=!syslog;!kernel, timestamp=s, allfields, tagname=s, tagpid=s, nosdlength, append, id=1, facility=extra23, filter=journal.input.filter
~ --journal src = all, timestamp = j, allfields = no, notagfield, append, id = 0, r = 4, ignore = hostname, network = local;other, dns = no
The --JournalField command
[Top of page]
The
--JournalField command is used to specify field-level exceptions to the general settings provided by the
--journal command (whether the default
instance of the command or one specified by the local administrator.
Multiple instances of
--journalFields command are supported.
The
--journalfield command takes a systemd journal fieldname as a primary argument. Though all fieldnames used by the journal are in uppercase,
Syslogd2
allows these naems to be upper- or lower-case and will ignore leading underscores ('_') when matching fieldnames.
Since there is no definitive list of journal fieldnames, the best way to find out what names are available is to run
Syslogd2 with
allfields = yes for a while
and to then review the available data to see which fields can be ignored completely and which fields you may want to keep.
--journalfields is NOT considered an 'input-command', so does NOT support fields other than those listed in this section.
If AllFields is enabled in the --journal command, --journalfields can use its disable option to suppress individual fields.
If AllFields is disabled in the --journal command, --journalfields will define the fields that are to be displayed.
These fields will be sequenced into the overall SD-String field in the order in which they are declared by successive --journalfield commands.
The list of options supported by --journalfields is currently quite short:
- Journal fields can be renamed by Syslogd2 (whether 'Allfields' is enabled or disabled) by following the journal-field-name in a '--journalfield
command with " = newname".
- Disable (Ignore): [default: False] This boolean value will suppress the applicable field from being included into the msg SD-String if set.
- MaxLength (Length): This setting over-rides the maxlength setting set by the '--journal' command for this field only.
--JournalFields usage examples:
~ --journalfield = transport = src
~ --journalfield = machine_id, disable
~ --journalfield invocation_id, disable = yes
~ --journalfield message = msg, maxlength = 40
[Top of page]