Fundamentals
[Top of page]
Download and Compile Syslogd2
Basic Familiarization Demo
Queuing Demo
These demonstrations assume that you have root access on your Linux test-bed.
They assume that you have just compiled Syslogd2 and have access to both the resultiung binaries and the 'tools' directory.
Each demonstration will (usually) provide all required sample configuration files and (to the extent possible) will run non-destructively alongside any existing syslog processor.
If it is desired to stop the current syslog processor, the demonstration instructions will assume you have a systemd-based Linux distribution (such as Ubuntu or Fedora/Red Hat) and will
provide instructiones accordingly.
All demonstrations of Syslogd2 will be pre-configured for an output directory at /opt2/log, a primary configuration file of /etc/syslog.conf and a
Configuration Directory (ConfigDir) at /etc/syslog.d. Syslogd2 does not create either the /opt2/log directory nor the /etc/syslog.d directory.
- If you use a different configuration file, you will need to add a --configfile=<filename> when starting Syslogd2.
- If you use a different configuration directory, you will need to uncomment and modify the --defaults configuration-file line that specifies <configdir>
- If you use a different output location, you will need to edit all configuration files (search and replace) for your new directory path.
Downloading and Compiling Syslogd2
[Top of page]
Syslogd2 Familiarization Demonstration
[Top of page]
[Goal] [Requirements] [Prep work] [Procedure]
This demonstration can be done from any text-based environment (terminal window, console, telnet session, etc....
This demonstration is non-destructive and can be done while an existing syslog daemon is executing.
Goal:
To gain a basic level of comfort with (and a basic understanding of) Syslogd2 by exploring various features of Syslogd2
Requirements:
- A set of compiled Syslogd2 variants.
- The programs and tools in the ./tools subdirectory directory of the build directory.
- Superuser (root) access on your test-bed Linux host.
Prep:
- Save the file Demo1File1.conf as /etc/syslog.conf.
- Download the include-files Demo1Include1.inc and Demo1Include2.inc and save them to
the /etc/syslog.d directory.
(If you chose to use a non-default location for these auxilliary files, uncomment the applicable --defaults line in the configuration file and specify
the new location for ConfigDir.)
Demonstration Procedure:
- Review the configuration files to see the syntax. Note that all configuration commands to Syslogd2 are in the form of command-line commands.
- Also notice the (commented out) --threadmaps statements that we will get to later and the (disabled and commented out) --journal and --journalfield
commands (that we will also come back to).
- Run a variety of Syslogd2 variants using this configuration file (one at a time). Kill each variant before starting the next using either the 'kill' or 'killall' command.
(If you did not use the default location for the configuration file, add "-c <configuration-file-pathname>" (Syslogd2 will [(normally] immediately
background with no output). For each variant:
- Start the syslogd2h Syslogd2 variant using the additional command-line parameter --TestConfig ('-T').
This will cause Syslogd2 to send configuration output to the local terminal.
Review this information in whatever detail you wish. The TestConfig option is covered in greater detail here.
This command-line option (and corresponding output) is available because the CAP_WHATIF
compiler-symbol was included at compile-time. (The status of all compiler-symbols can be found at the top of the
--TestConfic output.)
- Review the error file. When using the -T option, Syslogd2 modifies the stderr specification by adding ".TestConfig" to the filename, so the errors
and notifications from using the -T option will not show up in the active error file.
The -T option also causes Syslogd2 to quit after it displays the output but before initializing any system resources (ports, files, threads, etc).
The '-T' (--TestConfig) option is safe to run whether or not another syslog daemon is currently running.
- Repeat the above sequence for syslogd2d, syslogd2o, and syslogd2g.
specifically note the display differences in the secions labeled "InputThreadpools", "WorkerThreadpools" and "OutputThreadpools".
- Now start the syslogd2h Syslogd2 variant without using the additional --TestConfig ('-T') command-line parameter.
Syslogd2 should background itself with any output. You will need to run 'ps -axu | grep syslogd2" as the 'root' user to detect whether it is running.
(Run 'netstat -anp | grep syslogd2' to see what Linux and IP sockets Syslogd2 has opened.)
Running two concurrent syslog daemons would normally not be a safe operation to do on any Linux platform, except that our demo configuration has
disabled all conflicting interfaces (syslog socket, IP sockets, kernel input, etc) and is not conflicting with output files.
The sockets opened in /dev/ are unique to our demo configuraiton and are not used by any other syslog daemon process.
- Note that no output files have been created yet. Syslogd2 does not open output files until they are actually written to.
- Start the commad-tool (we will use this tool to 'tune' performance with some of the variants):
./tools/tsuch /dev/logStream
- Now we need to generate some data. You have your choice of data input methods: Linux socket, IP socket, or named-pipe. Choose the appropriate test-stub
from the ./tools directory and the matching input interface from the configuration file. Example:
./tools/ts4 127.0.0.1 1028 "This is a test messagge to a TCP port" 4 "s" extra0.warn,extra1.notice,extra2.info,kern.debug
./tools/tdu /dev/logStream "This is a test messagge to a Linux streaming socket" 4 "s" extra3.warn,extra32.notice,extra31.info,daemon.debug
./tools/tsp /dev/pipeDemo "This is a test messagge to a named-pipe" 4 "s" local1.warn,mail.notice,local2.info,reserved2.debug
In the 3 commands above, there are several references to the 'extra' facilities and one (1) reference to an invalid facility (extra32).
As is the custom for syslog processors, invalid facilities are treated as if the facility/priority had not been provided and converted to the default value of 'user.notice'.
You may also notice that several files are configured with facility ranges.
Send some data to additional facilities assigned to different files to see the effects of the output 'format' option.
- After sending some data, review the files.
Now delete all the files and resend some data.
Note that (as with most syslog processors), no data is output to deleted files.
Send Syslogd2 a 'SIGUSR2' signal ('killall -USR2 syslogd2h').
This will cause Syslogd2 to re-validate and re-initialize all outputs as necessary (re-creating deleted output files, pipes & sockets).
Re-check the files and you may see some 'leftover' data held by threads that could not previouosly write their data.
Fresh data will be written normally.
The fuction that re-initializes output is referred to as 'CheckDestinations' and can be put on a variable-interval schedule.
- Delete the input sockets and pipe devices (/dev/logDemo, /dev/logStream, /dev/pipeDemo).
Attempts to send data to deleted devices will fail until the device is closed and re-opened usually requiring a re-start.
Send Syslogd2 a 'SIGUSR1' signal ('killall -USR1 syslogd2h').
This will cause Syslogd2 to re-validate and re-initialize all inputs as necessary (re-creating deleted input pipes & sockets and resetting any IP ports with
input-error conditions such as being unresolved).
Recheck the /dev/ directory and you should see the named-pipe and socket devices restored.
The fuction that re-initializes input is referred to as 'CheckSources' and can also be put on a variable-interval schedule.
- Note the output-file ownership and permissions. (And the per-file exception for file Demolog2.demo). These values can be configured as either numeric values or string values.
- Start the command-tool that was built for this variant (./tools/tsuch). Explore the options as you wish, but note specifically that there is no queue data under option 'c'.
This is because in the 'h' variant, neither CAP_WORKERTHREADS nor CAP_OUTPUTTHREADS is defined and CAP_USERTHREADS has not been enabled. (Housekeeping threadpools
do not have input queues.)
- Experiment with changing the queue sizes and thread-counts for different threadpools. You can also change which worker-threadpool a given input-threadpool will use to process
its data.
- Again, repeat the above sequence (or just those parts concerning the command-tool: ('tsucX') for syslogd2d, syslogd2o, and syslogd2g.
specifically note the display differences in the secions labeled "InputThreadpools", "WorkerThreadpools" and "OutputThreadpools".
- Observe that when using the 'd', 'o' or 'g' variants, queue information is displayed in option 'c' of the command-tool. This command-tool option also allows you to change the
thread-count / queueid-pointer of any threadpool (input-type threadpools use this colume to display/manipulate their queueid-pointers-to-worker-threads since they don't have actual
input-queues).
Whenever the queue-size is changed via the command-tool, the queue counters (highwater & discards) are reset.
Using the command-tool to detect buffer overflows and to dynamically adjust queue sizes is an easy way to 'tune' Syslogd2 performance for optimum results.
Queues (if of sufficient-size) act to 'smooth out' the 'burstiness' of incoming traffic, while adjusting threadcounts allows you to 'size' the processing capacity of each threadpool
to efficiently handle the averaged (smoothed-out) rate of incoming traffic.
By default, Syslogd2 uses a 'lossy' queuing system for both worker-threadpool and output-threadpool input-queues. This is so that if events come into the application faster than they
can be handled, the processing threads won't 'back up' waiting to queue their 'finished product' and be unable to handle incoming traffic (which would in turn 'back' up,
causing cascading problems.
As with any 'rule' there are exceptions: When Syslogd2 itself controls the rate at which data is acquired (most notably by the --tailfile and --journal input threadpools,
but in the --kernel threadpool to some extent, it may often be preferable to set the 'queuing mode' to a 'lossless' mode of queueing where the speed of the processing and output threads
will intentionally cause this 'back-up' phenomenon, allowing the down-stream processors (if tcp-connections are also used) to limit the speed at which the input-threads can generate
new input.
To set any worker- or output-threadpool queue to use 'lossless' queueing, add the keyword 'lossless' to that threadpool's definition in a --threadmaps command.
The --threadmaps keyword is the only place where 'lossless' queueing can be configured.
To see this 'queue overflow issue' at work:
- Shut down the active command-tool (press 'q' until it exits) and shut down the Syslogd2 variant ('killall syslogd2X').
Delete the output files (to start clean), uncomment the 'append' option in the --journal command and re-start the Syslogd2 variant and command-tool.
(The 'g' variant will give the most extensive results, but may be harder to interpret.)
- Again, shut down the Syslogd2 variant and add the keyword 'lossless' to either or both of the --threadmaps definitions for worker threadpool 3 and/or output threadpool 4.
Restart the Syslogd2 variant and re-check the queue statistics.
- (Hint: the --journal command has been specifically configured to use input threadpool 'journal' (a one-off), worker threadpool 3, and will send traffic to extra31
- the only file in output threadpool 4.
The output files 'demolog1.demo' and 'journal.demo' will be the output files for journal data (duplicate data but different output formats) and the file 'extrafacilities.demo'
will be affected because it contains the facility 'extra31.*' in its range of facilities.
This concludes this familiarization demonstration.
There is much more to Syslogd2 than we have introduced here. I plan to set up more demonstrations to showcase more elements as time and energy permit.
The Syslogd2 Input Options (at the bottom of this site's menu) provides more in-depth coverage of all input options.
The Syslogd2 Output Options page covers output options and formats and the Syslogd2 Processing Options
page covers all other configuration elements.
Queueing Quandry
[Top of page]
[Goal] [Requirements] [Prep work] [Procedure]
Goals
To demonstrate and explore the characteristics of different data-communication protocols and queueing methods and their effects on data-loss.
Background
All queueing can be compared to one of two analogies - both involving the idea of 'handing-off' a flow of water from one piping system to another.
- 'Lossless' queueing (and connection-oriented data-transfer protocols such as TCP) can be compared to simply connecting the flows together with a length of pipe.
The good news is that every event that is entered (every drop of water) is (ultimately) received at the destination.
The bad news is that any event that causes a blockage at any point in the flow affects the ENTIRE 'upstream' flow as well - possibly to the point of preventing receipt
of more data.
- 'Lossy' queueing (and connection-less data-transfer protocols such as UDP) can be compared to a large open-topped bucket into which the source data flows. The data-processors
of a 'lossy' queue are represented by a variable-sized hole in the bottom of the bucket through which the water exits.
The faster (or more) processors there are, the larger the hole.
As long as the bucket is large enough to handle any 'spurts' (data-bursts) and as long as the hole in the bottom is large enough to handle the average flow of water,
all is good.
Problems arise when the hole in the bottom of the bucket is too small and the bucket over-flows (the processors are too slow resulting in data-overflow of the incoming
hand-off buffers).
In most cases, there is no way to detect that this is even occuring.
The good news with 'lossy' queueing is that the data-generators can never be affected by the speed (or lack thereof) of the processing elements and do not have to worry
about the overhead of maintaining a connection handshake.
The bad news is that there is no guaranteed delivery of the data with connection-less protocols.
There is no re-transmission of data should any data fail to be accepted (or processed) at the destination.
Processing of connection-less data streams (UDP and Linux datagram) literally devolve into a 'race' between incoming data and the speed and quantity of processing threads.
Data loss can occur with the UDP protocol at any point in the data path.
There is nothing applications can do about transmission losses when using UDP.
Applications CAN (however) be designed to minimize the losses that occur between the host's network-buffers and the application.
Syslog applications can also be designed (as with Syslogd2) to minimize the data placed onto the network in the first place.
Syslogd2's original focus was on the hand-off from the operating system buffers to the application. The emphasis was on improving the speed with which the reader-threads read
data from the system buffers into (larger) application buffers where (more) threads can then process the syslog events in parallel.
Syslogd2 has also attempted to remove as many chokepoints between threads and delays due to operating system operations as possible.
In other words, Syslogd2's original (and still default) approach is to be faster than the incoming data stream.
In the full-up architecture (represented in default code by the 'me[g]a' model), emphasis is placed on the reader threads doing as little as possible in order to be as fast
(as responsive to incoming data) as possible.
There are several techniques that Syslogd2d employs to improve throughput (both of the reader-threads and of the processing and output threads which are covered in the
Syslogd2 Input Processing Options page, the
Syslogd2 Processing Options page and the Syslogd2 Output Options page).
This demonstration will focus on a discussion of queueing and the indicators of 'good' and 'bad' performance.
Note: The ideal situation for a syslog system receiving external data is to use 'lossy' queueing but with sufficient processing capacity (a sufficiently large hole) such that
'the bucket' is nearly always empty and is just used to hold the 'bursts' of data that frequently occur in both syslog and networking.
Sustained, high-speed input is a challenge for any syslog processor (Syslogd2 included).
Note 2: There are some (limited) circumstances when is actually advantageous to use 'lossless' queueing to act as a 'governor' on the (otherwise unchecked) speed of the input
threads.
One of those situation (systemd journal input) is highlighted in this demonstration.
Requirements:
- A set of compiled Syslogd2 variants.
- The programs and tools in the ./tools subdirectory directory of the build directory.
- Superuser (root) access on your test-bed Linux host.
Prep:
- Save the file Demo2File1.conf as /etc/syslog.conf.
- Download the include-files Demo2Include1.inc and Demo2Include2.inc and save them to
the /etc/syslog.d directory.
(If you chose to use a non-default location for these auxilliary files, uncomment the applicable --defaults line in the configuration file and specify
the new location for ConfigDir.)
Demonstration Pocedure (Setting the stage):
As mentioned earlier, much of Syslogd2's design effort has been to 'be faster than the input'.
Faster reading of OS-hand-off buffers is the only practical way to handle UDP high-volume input.
But as also mentioned there are times when 'lossless' queueing is advantageous because it backs up and degrades the ability to read new-input traffic.
The conditions in which 'lossless' queueing should probably be used will exist when Syslogd2 itself is the generator of the input traffic.
Currently, the only clear-cut situations to use lossless queueing are for --journal and --tailfile input.
In both cases, the data has been generated (and is going nowhere).
In both cases, it is Syslogd2 that determines how fast to read the incoming data. In both cases, a slow reader can never result in lost data. But in both cases,
if either CAP_WORKERTHREADS or CAP_OUTPUTTHREADS are defined, reading data too quickly can 'flood' 'downstream' (from the reader threadpools) processing queues,
resulting lost data when using 'lossy' queueing.
This is when 'lossless' queueing becomes a preferred setting.
To a lesser extent the kernel threadpool would also quaiify under this criteria, but kernel syslog volume is usually so low in overall volume that it's easily handled
by 'lossy' queues.
To configure 'lossless queueing, add the keyword 'lossless' to an applicable section of a --threadmaps command.
(The --threadmaps command is the only place lossless queueing may be configured.)
NOTE: Mixing 'lossy' and 'lossless' queueing in the same threadpool is obviously not pssible, so this is one situation in which a second (or third) worker-threadpool may need to be
declared (providing both a 'lossy' and 'lossless' path through Syslogd2 for data form different sources.
The 'lossless' worker-threadpools (and output-threadpools if they exist) should not be used by other inputs due to the risk of backing up the 'lossless' threads, making them unavailable to
process new input and causing reception problems for non-Syslogd2-controlled input methods.
In other words, it's better for Syslogd2 to receive data and have to throw it away due to internal queue overflow than to not receive the data at all due to system-buffer overflow.
At least when Syslogd2 discards the messae, a queue-overflow is recorded and action can be taken. There is no way to detect when operating system buffers overflow, since the data
has not yet entered Syslogd2's control.
So what are the causes of syslog processing delays ? Most of them have to do with system-calls and each system call blocks the calling thread until the system-call completes:
- Dns response time:
This is one of three primary reasons why many people disable DNS lookups.
The other two are (1) the amount of network traffic generated and (2) the additional load placed on corporate DNS servers.
Syslogd2's response to this is toe create and populate an optional internal name-cache that can be 'fed' from a user-generated
cache-file in addition to the /etc/hosts file. When using a fully-populated cache-file, the DNS service (if enabled) will never be called.
- Parsing the incoming header fields: (not a system-call) All processors do this, but due to Syslogd2's focus on network management, Syslogd2 will attempt to not only identify
a traditional syslog time field, but will also attempt to recognize variations of both traditional and rfc3339 time-strings (as well as its own 'clock' format).
Syslogd2 will also attempt to parse the string and (using the system timezone files), 'revert' the string to a system-clock representaion that can then be used to produce
different output formats.
Parsing the time-stringgs is the most practical way to handle text-based log files that are not formatted as syslog log files.
In this situation, kerma is with Syslogd2: Most high-volume traffic comes from network-devices (particularly firewalls).
Hardware devices almost never specify a time-field or (because then the syslog specifications state they would have to specify a source-hostname field).
Because of this, most high-volume situations can use the 'noheader' input option to tell Syslogd2 not to even look for the time & host fields (the msge 'header').
Most syslog processors have no equivlent to Syslogd2' NoHeader option, so will attempt to find a 'time + hostname header' string even when such a string will nover be
found in the data. (For Syslogd2 hosts that receive input from both hardware devices (where 'NoHeader' applies and other hosts (where it does not), two input sockets
(one on a secondary address) are suggested with one interaface configured with 'NoHeader'.
- Parsing the incoming messae string: Traditional syslog parsers will usually parse the entire command-string of incoming messagges, converting embedded control characters to printable
strings (^c, ^j, etc) and 8-bit characters to octal represendations (the delete key at 127 converts to \177).
Syslogd2 provides the NoForcePrintable option to allow tell Syslogd2 not to do this step.
Syslogd2 will still trim the strin of any trailing white-space or control characters to facility filter matching.
- Writing to invalid stream connections (sockets or pipes): Many syslog processors do not support TCP sockets.
For those that do, attempting to open a TCP connection to a host that is not there results in a network timeout during which time the calling thread is blocked by the OS.
These timeouts can be many seconds in length.
What makes this so bad is that there is usually no 'memory' of the status of the connection, so the next thread that has data for that host will try to open the socket again
(and time out again - and again -- and again...).
Syslogd2's approach is to check its recorded connection-status before attempting to write. If the connection is closed, it will make one attempt to open the connection.
If that attempt fails, the connection is marked 'invalid' and the message is either discarded or spooled for later. If the socket is recorded as 'avalable' but the write attempt
fails, one attempt is made to re-open the connection. The result of this attempt is recorded as either 'available' or 'invalid' and the message disposition is one of transmitted,
discarded or spooled. If the connection status is 'invalid' when first checked, the message is immediately discarded or spooled, but no attempt is made to re-open the socket.
Syslogd2 uses a second (housekeeping) thread to periodically walk the list of outputs attempting to re-open all 'invalid' sockets, pipes, files, etc.
If successful, the 'invalid' flag is cleared and the (now open) socket is returned to operation. The net result of this approach is that only one attempt is made to open streaming
connections to non-responsive ports or applications with far less opportunities for threads to 'back-up' while waiting their turn to time out.
- Reader-threads are distracted: There are two aspects to this point:
- For syslog processors that do not break out the reader/processing/output 'processing phases' into separate threadpools (which is all of them) the reader threads cannot be
responding to network buffers and reading data if they are waiting on a DNS response or waiting on their turn to time-out while attempting to open a 'socket-to-nowhere' or while
they are looking for binary characters to convert to printable form, etc.
- When using the 'generic' reader threadpool with a mix of stream and datagram ports, Syslogd2 will read up to 8K of data from each stream port at once if available.
It will then work on that data until it is exhausted before going back to the list of sockets for the next 'round' of input. This means that though UDP sockets will get read
as often as stream sockets in that threadpool, the stream sockets may get 4 or more messages for every one messagge from UDP due to the block-read and buffering from the stream.
(Unlike datagrams that are 1-msg-per-datagram, stream sockets have no easy way to read just one msg. A 'block' of data needs to be read, then split into individual messages at
each instance of a '<nl>' character.
Demonstration Pocedure:
In this demonstration, we will use the syslogd2d variantto simulate the hand-off buffers between the operating system and the syslog application. Since the read-threads are designed to
do virtually no processing (their role is to read data and queue it as quickly as possible), the worker-threadpool queue will act as the simulated network-buffers and the
worker-threadpools and output-threadpools (that do the vast bulk of the msg processing) will (simulate) a 'typical' single- or multi-threaded syslog daemon.
We will then explore the performace differences when the output-threadpools are also enabled.
[Top of page]