CAP_COMMAND
(Interactive command-interface to a currently-running Syslogd2 instance)
The intent of CAP_COMMAND is to open a two-way communication 'dialog' into a running copy of Syslogd2 in order to query performance and
configuration status and to send selected configuration commands in the hope that constant restarts of the entire system can be avoided.
Constantly re-starting the daemon has a detrimental effect on the integrity of all incoming data-streams as well as a tendency to annoy the applicable
administrator of the application and/or host.
CAP_COMMAND enables a new application (the 'command-tool' which is one of the 'tools' built as part of the Syslogd2 project package).
This 'tool' is referred to as the 'command-tool' for reasons that should become obvious.
Like the Syslogd2 binaries themselves, each compiled variant of this this tool 'takes on' the same suffix as the binary version it is 'mated' to.
The base-name of this tool is tsuc to which is appended one of the suffixes assigned to each variant of Syslogd2 that was compiled.
To be as effective as possible, the command-tool menus use the ncurses library to manage text-menus.
This allows the command-tool to be used (and menu-driven) in the absence of a GUI-interface.
To provide a layer of security, the command-tool only functions over Linux Streaming sockets.
The 'Linux' qualifier insures that all users of the command-tool have authorized access to the host and guards against remote (IP-based) attacks.
The 'streaming' qualifier allows for an authenticated two-way connection with handshake (error detection).
To provide the host-services component, while still allowing the parent-thread to continue to manage the background services of Syslogd2,
the command-tool requires an available housekeeping thread.
The command-tool is still very early in development, and has not had an excessive amount of time put into it, so it is still somewhat primative. Currently it can:
The threadcounts setting is particularly useful to network-management and system administrators as this option provides the necessary detail that allows for 'tuning' of Syslogd2 buffer sizes and thread-counts.
The '[d]emo' variant from the default compilation set provides a method to simulate (and estimate) the data loss experienced by legacy syslog processors when presented with
loads or tasks that are well within network-management expectations. My laptop does not generate excessive syslog traffic (its a fairly routine installation of Ubuntu), but due to
Syslogd2 development, it DOES produce a lot of output-files (~40 if streaming sockets [1 active one to DBD2] are included) because I have configured filters to split the 'daemon.* data-stream into its component parts for reporting and development purposes (network-management activity).
In the real environment (real world), datagram input is placed into a (small) number of input buffers for the syslog application. If the syslog application is fast enough to read the data before additional incoming data over-writes it (the buffer is circular), all is good. If not, data is lost and the application has no way to know how much (if any) data has been lost.
The '[d]emo' variant can simulate this scenario. In the [d]emo variant, the input threadpools play the role of the syslog-generators,
the worker-threadpool FIFO buffer plays the role of the system's hand-off-buffers and the combined worker-and-output-threads play the role of rsyslog threads.
This simulation is far from perfect, but it's close enough to simulate the traffic and illustrate the point.
The 'game plan' is to configure Syslogd2 to be as close as possible to a duplicate configuration of the legacy system (rsyslog in the case of Linux).
Disable internal name-cache and insure the NoForcePrintable boolean flag is also disabled.
Disable all filters and insure that all output-specifications match those of the legacy system (you might choose to '--include' the legacy system's configuration files).
Leave Houskeeping threads and the streaming input socket with the command-option enabled. Set 'queueid=1' and 'id=1' in the input-option-list for the command-input socket.
This will move it out of the 'processing path' of the main test.
Shut down the legacy process so it can be replaced by Syslogd2. ('systemctl stop syslog.service', 'systemctl disable rsyslog.service' for systemd).
Start Syslogd2d manually (with the --systemd command-line option if using systemd).
Let run for several seconds (or minutes), then run the command-tool: ./tools/tsucd <Linux socket path>. Observe the highwater and discard counters for worker-queue 0. This
represents the approximate situation with the system hand-off buffers.
You can adjust the threadcounts and queue-size for the executing Syslogd2d daemon from this screen as desired.
You can also run Syslogd2d with different parameters (number of output-files, DNS on/off, name-cache on/off, etc) to see the impact of different settings
on performance and overall capacity.
Don't forget to re-enable and restart rsyslog when done testing: (killall -9 syslogd2d; systemctl enable rsyslog; systemctl start syslog.service).
If the combination of:
is present, then this option will produce a display similar to:
Pool-Type Pool-Id Thread-Count Worker-Q Q-High-Water
Active / Max Q-Size / Q-Discards
a) Input -2 8/8 0
b) Input 0 6/6 0
c) Input 1 6/6 1
d) Input 2 6/6 2
e) Kernel 0 6/6 0
f) Worker 0 10/10 50 50/29
g) Worker 1 5/5 30 1/0
h) Worker 2 5/5 30 1/0
i) User 0 1/1 5 1/0
j) HouseKeeping 0 4/4
... where every threadpool in the configuration is displayed along with current statistics.
You can (from this screen) also 'adjust' thread-counts and (for non-input thread-pool-types) queue-sizes or (for input-thread-pool-types) queue-id-values.
This configuration (on my laptop) sorts my log data into over 40 output files. As you can see, 10 processing threads in worker-threadpool 0 is not enough to prevent loss of data.
In my configuration, The default Linux input socket is in threadpool 0, the default IP socket is in facility 2 and streaming input (including command=tool) is in threadpool 1.
The display above shows that input-threadpool 0 sends its traffic to worker-queue (worker-threadpool) 0 which is the one that is over-loaded.
Analysis: Since 10 threads are not enough to handle the load of processing and output to over 40 files, I should consider changing my syslog binary and using a model that allows me
to isolate the output processing from the worker-threadpool. This output came from the '[d]emo' model (CAP_WORKERTHREADS is defined but not CAP_OUTPUTTHREADS).
Here is the output from the 'me[g]' model on the same host for the same configuration but with the output separated into 3 separate threadpools:
Pool-Type Pool-Id Thread-Count Worker-Q Q-High-Water
Active / Max Q-Size / Q-Discards
a) Input -2 8/8 0
b) Input 0 6/6 0
c) Input 1 6/6 1
d) Input 2 6/6 2
e) Kernel 0 6/6 0
f) Worker 0 5/5 50 13/0
g) Worker 1 5/5 30 1/0
h) Worker 2 5/5 30 1/0
i) Output 0 5/5 30 12/0
j) Output 1 5/5 30 9/0
k) Output 2 5/5 30 8/0
l) User 0 1/1 5 1/0
m) HouseKeeping 0 4/4
Please notice that 5 processing threads in worker-threadpool-0 are now quite sufficient to handle the presented load with output threadpool 0 showing the most activity.
Destinations in this configuration are split into 3 groups:
Output 0: The set of files inherited from rsyslog. (Including *.* to /var/log/syslog.log [minus the data split out by Syslogd2filters])
--> Because of the amount of data written to a single file, I suspect the output threads in threadpool 0 are 'stacking up' waiting to write to
'/var/log/syslog.log', resulting in delays and backups in the input queue.
Output 1: 35 output files split out from the default Linux socket and 3 pseudo-terminal devices (GUI terminal windows for /dev/pts/0 and /dev/pts/1)
Output 2: Three output streaming sockets (one to DBD2, 2 unused) and 3 output files.