CAP_VARBUFFERS
(User-Defined Buffer-Lengths)
The Reason for VarBuffers
[Top of page]
Traditional syslog logging applications have no need to change the size of their working buffers as they expect (more or less) fixed-length messages with
predictable maximum-lengths. This is not true of Syslogd2.
Part of Syslogd2's designed operation is and operation is to gather and forward data from a maximum number and type of inputs. Prior to (but especially since) the
implementation of CAP_JOURNALTHREADS, Syslogd2 had the ability to access or create certain pieces of meta-data.
One of the original goals for Syslogd2 was to select specific messaes (based on content rather than facility/priority (facpri) and to forward any user-provided
information (static-data or otherwise) to downstream processors. As the design and implementation of DBD2 matured, it became clear taht DBD2 could well act
as one of those primary 'downstream processes', putting named-data fields (in name=value format) into one or more databases for user reporting.
What was somewhat less obvious (until actual testing) is that the number of fields and length of the overall data strings from CAP_JOURNALTHREADS)
(let alone any user-application) was sufficient to cause routine crashes of both Syslogd2 and DBD2 due to buffer overflow conditions.
The 'obvious' solution would have been to increase the buffer size, but how big is 'reasonable' ?
CAP_JOURNALTHREADS routinely produces sd-string lengths of over 700 bytes (assuming all fields are length-limited to 80 characters).
At least 3 fields ('MESSAGE', 'SYSLOG_RAW' and '_CMDLINE') can each easily reach a thousand or more bytes if unconstrained.
Fortunately (for me), such long strings are not that useful for adminstration, so truncation is certainly one option that has been implemented to cope with demands
for oversize data buffers.
However, there are times when admins may have legitimate desires to transmit (or log to disk) the entirety of long data strings (or a high number of data fields).
This leads to the other option I chose to implement - which is to allow the user to modify the lengths of internal buffers at run-time, giving the user
control over the tradeoff between long data-strings and memory conservation.
Syslogd2 compiles with default values for all variable buffers that should be adequate for the vast majority of logging situations.
There are several possible scenarios however, where the default buffer values have proven (or are likely to prove) woefully inadequate. One of these occurred during the
implementation of CAP_JOURNALTHREADS. Due to the over 700 byte length of the aggregate SD-String generated by (not quite) all available journal fields, Syslogd2
routinely 'blew up' while processing input filters (where the SD-String is manipulated independent of the overall input-string or message-string).
To address this issue, I now create an estimated SDStringLength based on the number of journal fields being displayed, their average length, plus the global and individual lengths
specified by the user. I use this estimate as a 'floor' for the MaxSDStringLength variable whenever CAP_JOURNALTHREADS is declared. This estimation process should work
to resolve the issue of long SDStringLength values from CAP_JOURNALTHREADS, but will do nothing to resolve the same issue when triggered by user-defined static-data-fields
provided to Syslogd2 as input. This is the primary reason for CAP_VARBUFFERS. Other reasons may surface as Syslogd2 gains usage and experiences.
[Top of page]
Like all global variables in Syslogd2, the bufferlengths exposed to the user by CAP_VARBUFFERS are configured using the --defaults command.
Multiple instances of the --defaults command may exist in a given configuration file. No configuration is ncessary if the default values are satisfactory.
- MaxMsgLength: Default: 1024 (bytes). This is the maximum length of the message component of the syslog string. Any message coming in with a long msg-string component
will be truncated the length of this buffer.a This value may be set to to any length (the larger it is, the longer the acceptable syslog msg, but the more memory it consumes).
This value is used as an upper bound: There is a 'shadow-copy' of this value that is actually used (adjusted for a 'floor' of 15 characters).
- MaxFQDNLength: Default HOST_NAME_MAX defined by the system.
(64 for Linux, 255 for posix).
This is the maximum length of the host-field when used to hold an IP host name or address.
(Linux pathnames use a different, non-modifiable buffer of 255 bytes and Linux socket paths are limited to approx 107 bytes).
This value has a 'floor' of 40 and a 'ceiling' set by the system of HOST_NAME_MAX). The 'floor' is necessary to allow minimal operation with IPv6 addresses.
- MaxMsgSaveLength: This is the maximum length of the buffer used to hold 'previous messaes' so duplicates can be counted and logged as 2 lines (the msg + 'This line repeated ... times').
- MaxMsgLength: Default: 1024 (bytes). This is the maximum length of the message component of the syslog string. Any message coming in with a long msg-string component
will be truncated the length of this buffer.a This value may be set to to any length (the larger it is, the longer the acceptable syslog msg, but the more memory it consumes).
This value is used as an upper bound: There is a 'shadow-copy' of this value that is actually used (adjusted for a 'floor' of 15 characters).
- MaxFQDNLength: Default HOST_NAME_MAX defined by the system.
(64 for Linux, 255 for posix).
This is the maximum length of the host-field when used to hold an IP host name or address.
(Linux pathnames use a different, non-modifiable buffer of 255 bytes and Linux socket paths are limited to approx 107 bytes).
This value has a 'floor' of 40 and a 'ceiling' set by the system of HOST_NAME_MAX). The 'floor' is necessary to allow minimal operation with IPv6 addresses.
- MaxMsgSaveLength (MaxMsgSavedLength): This is the maximum length of the buffer used to hold 'previous messaes' so duplicates can be counted and logged as 2 lines (the msg + 'This line repeated ... times').
If the 'AllMsgs' flag is enabled at the global or at all local file outputs, this buffer is neither allocated nor used.
This buffer is allocated for local disk files only (one buffer for each disk-file destination that does not hahve 'allmsgs' enabled).
- MaxSDStringLength: Default: 0 bytes.
This buffer length (if set) is added to the defined value of MaxMsgLength.
It designates the estimated length of the SD-String that holds static data parameters.
If CAP_JOURNALTHREADS is defined and configured to define static journal data-fields, Syslogd2 will calculate a minimal estimate for this value to act as a 'floor' to avoid application crashes due to buffer overflow. Otherwise, the user may set a length for this buffer based on the lenght of any parameter string passed into Syslogd2.
There is no upper bound.
- MaxInputLineLength: Default 1500 (bytes). This is the maximum length of a raw input string. An internal calculation is made to act as a 'floor':
50 bytes for fixed data (facpri field, mandatory whitespace as field separators, max length of a (RFC3393) time field) plus
the final calculated length of the FQDN buffer (host field) + the combined final SDString and MaxMsgLength values.
The user may designate additional space if desired.
- MaxOutputLineLength: default 1500 (bytes).
This is the maximum length of the output line.
Any output line that would exceed this length is truncated.
There is no calculated value or 'floor' for this buffer-length. Due to the impact of filters that can add or remove overall msg content (not to mention the length differences between IP addresses and host-names), syslog output strings are rarely the same length as syslog input strings.
If the 'AllMsgs' flag is enabled at the global or at all local file outputs, this buffer is never allocated. This buffer is allocated for local files only (one buffer for each destination that does not hahve 'allmsgs' enabled.).
As Syslogd2 continues to mature, additional internal buffer-lengths may be added to the above list.
[Top of page]